Connect public, paid and private patent data with Google Patents Public Datasets

Hash file system and method for use in a commonality factoring system

Download PDF

Info

Publication number
US20040148306A1
US20040148306A1 US10757753 US75775304A US2004148306A1 US 20040148306 A1 US20040148306 A1 US 20040148306A1 US 10757753 US10757753 US 10757753 US 75775304 A US75775304 A US 75775304A US 2004148306 A1 US2004148306 A1 US 2004148306A1
Authority
US
Grant status
Application
Patent type
Prior art keywords
data
hash
system
file
computer
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US10757753
Inventor
Gregory Moulton
Stephen Whitehill
Original Assignee
Moulton Gregory Hagan
Whitehill Stephen B.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F17/00Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/30Information retrieval; Database structures therefor ; File system structures therefor
    • G06F17/30067File systems; File servers
    • G06F17/30091File storage and access structures
    • G06F17/30097Hash-based
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRICAL DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14Error detection or correction of the data by redundancy in operation
    • G06F11/1402Saving, restoring, recovering or retrying
    • G06F11/1446Point-in-time backing up or restoration of persistent data
    • G06F11/1448Management of the data involved in backup or backup restore
    • G06F11/1453Management of the data involved in backup or backup restore using de-duplication of the data
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • Y10S707/99933Query processing, i.e. searching
    • Y10S707/99936Pattern matching access
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99931Database or file accessing
    • Y10S707/99937Sorting
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10TECHNICAL SUBJECTS COVERED BY FORMER USPC
    • Y10STECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y10S707/00Data processing: database and file management or data structures
    • Y10S707/99951File or database maintenance
    • Y10S707/99952Coherency, e.g. same view to multiple users
    • Y10S707/99953Recoverability

Abstract

A system and method for a computer file system that is based and organized upon hashes and/or strings of digits of certain, different, or changing lengths and which is capable of eliminating or screening redundant copies of aggregate blocks of data (or parts of data blocks) from the system. The hash file system of the present invention utilizes hash values for computer files or file pieces which may be produced by a checksum generating program, engine or algorithm such as industry standard MD4, MD5, SHA or SHA-1 algorithms. Alternatively, the hash values may be generated by a checksum program, engine, algorithm or other means that produces an effectively unique hash value for a block of data of indeterminate size based upon a non-linear probablistic mathematical algorithm.

Description

    CROSS REFERENCE TO RELATED PATENT APPLICATIONS
  • [0001]
    The present invention claims priority from U.S. Provisional Patent Application Serial No. 60/183,762 for: “System and Method for Decentralized Data Storage” filed Feb. 18, 2000, the disclosure of which is herein specifically incorporated by this reference.
  • COPYRIGHT NOTICE/PERMISSION
  • [0002]
    A portion of the disclosure of this patent document may contain material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document of the patent disclosure as it appears in the United States Patent and Trademark Office patent file or records, but otherwise, reserves all copyright rights whatsoever. The following notice applies to the software and data and described below, inclusive of the drawing figures where applicable: Copyright© 2000, Undoo Technologies.
  • BACKGROUND OF THE INVENTION
  • [0003]
    The present invention relates, in general, to the field of hash file systems and commonality factoring systems. More particularly, the present invention relates to a system and method for determining a correspondence between electronic files in a distributed computer data environment and particular applications therefor.
  • [0004]
    Economic, political, and social power are increasingly managed by data. Transactions and wealth are represented by data. Political power is analyzed and modified based on data. Human interactions and relationships are defined by data exchanges. Hence, the efficient distribution, storage, and management of data is expected to play an increasingly vital role in human society.
  • [0005]
    The quantity of data that must be managed, in the form of computer programs, databases, files, and the like, increases exponentially. As computer processing power increases, operating system and application software becomes larger. Moreover, the desire to access larger data sets such as multimedia files and large databases further increases the quantity of data that is managed. This increasingly large data load must be transported between computing devices and stored in an accessible fashion. The exponential growth rate of data is expected to outpace the improvements in communication bandwidth and storage capacity, making data management using conventional methods even more urgent.
  • [0006]
    Many factors must be balanced and often compromised in conventional data storage systems. Because the quantity of data is extremely large, there is continuing pressure to reduce the cost per bit of storage. Also, data management systems should be scaleable to contemplate not only current needs, but future needs as well. Preferably, storage systems are incrementally scaleable so that a user can purchase only the capacity needed at any particular time. High reliability and high availability are also considered as data users are increasingly intolerant of lost, damaged, and unavailable data. Unfortunately, conventional data management architectures must compromise these factors so that no one architecture provides a cost-effective, reliable, high availability, scaleable solution.
  • [0007]
    Conventional RAID (Redundant Array of Independent Disks) systems are a way of storing the same data in different places (thus, redundantly) on multiple storage devices such as hard disks. By placing data on multiple disks, input/output (“I/O”) operations can overlap in a balanced way, improving performance. Since the use of multiple disks increases the mean time between failure (“MTBF”), storing data redundantly also increases fault-tolerance. A RAID system relies on a hardware or software controller to hide the complexities of the actual data management so that a RAID system appears to an operating system as a single logical hard disk. However, RAID systems are difficult to scale because of physical limitations in the cabling and controllers. Also, the availability of RAID systems is highly dependent on the functionality of the controllers themselves so that when a controller fails, the data stored behind the controller becomes unavailable. Moreover, RAID systems require specialized, rather than commodity hardware, and so tend to be expensive solutions.
  • [0008]
    NAS (network-attached storage) refers to hard disk storage that is set up with its own network address rather than being attached to an application server. File requests are mapped to the NAS file server. NAS may provide transparent I/O operations using either hardware or software based RAID. NAS may also automate mirroring of data to one or more other NAS devices to further improve fault tolerance. Because NAS devices can be added to a network, they enable scaling of the total capacity of the storage available to a network. However, NAS devices are constrained in RAID applications to the abilities of the conventional RAID controllers. Also, NAS systems do not enable mirroring and parity across nodes, and so are a limited solution.
  • [0009]
    In addition to data storage issues, data transport is rapidly evolving with improvements in wide area network (“WAN”) and internetworking technology. The Internet, for example, has created a globally networked environment with almost ubiquitous access. Despite rapid network infrastructure improvements, the rate of increase in the quantity of data that requires transport is expected to outpace improvements in available bandwidth.
  • [0010]
    Philosophically, the way data is conventionally managed is inconsistent with the hardware devices and infrastructures that have been developed to manipulate and transport data. For example, computers are characteristically general-purpose machines that are readily programmed to perform a virtually unlimited variety of functions. In large part, however, computers are loaded with a fixed, slowly changing set of data that limit their general-purpose nature to make the machines special-purpose. Advances in processing speed, peripheral performance and data storage capacity are most dramatic in commodity computers. Yet many data storage solutions cannot take advantage of these advances because they are constrained rather than extended by the storage controllers upon which they are based. Similarly, the Internet was developed as a fault tolerant, multi-path interconnected network. However, network resources are conventionally implemented in specific network nodes such that failure of the node makes the resource unavailable despite the fault-tolerance of the network to which the node is connected. Continuing needs exist for high availability, high reliability, highly scaleable data storage solutions.
  • SUMMARY OF THE INVENTION
  • [0011]
    Disclosed herein is a system and method for a computer file system that is based and organized upon hashes and/or strings of digits of certain, different, or changing lengths and which is capable of eliminating or screening redundant copies of the blocks of data (or parts of data blocks) from the system. Also disclosed herein is a system and method for a computer file system wherein hashes may be produced by a checksum generating program, engine or algorithm such as industry standard Message Digest 4 (“MD4”), MD5, Secure Hash Algorithm (“SHA”) or SHA-1 algorithms. Further disclosed herein is a system and method for a computer file system wherein hashes may be generated by a checksum program, engine, algorithm or other means that generates a probabilistically unique hash value for a block of data of indeterminate size based upon a non-linear probablistic mathematical algorithm or any industry standard technique for generating pseudo-random values from an input text of other data/numeric sequence.
  • [0012]
    The system and method of the present invention may be utilized, in a particular application disclosed herein, to automatically factor out redundancies in data allowing potentially very large quantities of unfactored storage to be often reduced in size by several orders of magnitude. In this regard, the system and method of the present invention would allow all computers, regardless of their particular hardware or software characteristics, to share data simply, efficiently and securely and to provide a uniquely advantageous means for effectuating the reading, writing or referencing of data. The system and method of the present invention is especially efficacious with respect to networked computers or computer systems but may also be applied to isolated data storage with comparable results.
  • [0013]
    The hash file system of the present invention advantageously solves a number of problems that plague conventional storage architectures. For example, the system and method of the present invention eliminates the need for managing a huge collection of directories and files, together with all the wasted system resources that inevitably occur with duplicates, and slightly different copies. The maintenance and storage of duplicate files plagues traditional corporate and private computer systems and generally requires painstaking human involvement to “cleanup disk space”. The hash file system of the present invention effectively eliminates this problem by eliminating the disk space used for copies and nearly entirely eliminating the disk space used in partial copies. For example, in a traditional computer system copying a gigabyte directory structure to a new location would require another gigabyte of storage. In particular applications, the hash file system of the present invention reduces the disk space used in this operation by up to a hundred thousand times or more.
  • [0014]
    Currently, some file systems have mechanisms to eliminate copies, but none can accomplish this operation in a short amount of time which, in technical terms, means the system factors copies in O(l) (“on the order of constant time”) time, even as the system scales. This means a unit of time that is constant as opposed to other systems that would require O(N**2), O(N) or O(log(N)) time, meaning time is related to the amount of storage being factored. Factoring storage in non-constant time may be marginally satisfactory for systems where the amount of storage is small, but as a system grows to large sizes, even the most efficient non-constant factoring systems become untenable. The hash file system of the present invention is designed to factor storage on a scale never previously attempted and in a first implementation, is capable of factoring 2 million petabytes of storage, with the ability to expand to much larger sizes. Existing file systems are incapable of managing data on such scales.
  • [0015]
    Moreover, the hash file system of the present invention may be utilized to provide inexpensive, global computer system data protection and backup. Its factoring function operates very efficiently on typical backup data sets because computer file systems rarely change more than a few percent of their overall storage between each backup operation. Further, the hash file system of the present invention can serve as the basis for an efficient messaging (e-mail) system. E-mail systems are fundamentally data copying mechanisms wherein an author writes a message and sends it to a list of recipients. An e-mail system implements this “sending” operation effectively by copying the data from one place to another. The author generally keeps copies of the messages he sends and the recipients each keep their own copies. These copies are often, in turn, attached in replies that are also kept (i.e. copies of copies). The commonality factoring feature of the present invention can eliminate this gross inefficiency while transparently allowing e-mail users to retain this familiar copy-oriented paradigm.
  • [0016]
    Because, as previously noted, most data in computer systems rarely change, the hash file system of the present invention allows for the reconstruction of complete snapshots of entire systems which can be kept, for example, for every hour of every day they exist or even continuously, with snapshots taken at even minute (or less) intervals depending on the system needs. Further, since conventional computer systems often provide limited versioning of files (i.e. Digital Equipment Corporation's VAX® VMS® file system), the hash file system of the present invention also provides significant advantages in this regard. Versioning in conventional systems presents both good and bad aspects. In the former instance, it helps prevent accidents, but, in the latter, it requires regular purging to reduce the disk space it consumes. The hash file system of the present invention provides versioning of files with little overhead through the factoring of identical copies or edited copies with little extra space. For example, saving one hundred revisions of a typical document typically requires about one hundred times the space of the original file. Using the hash file system disclosed herein, those revisions might require only three times the space of the original (depending on the document's size, the degree and type of editing, and external factors).
  • [0017]
    Still other potential applications of the hash file system of the present invention include web-serving. In this regard, the hash file system can be used to efficiently distribute web content because the method of factoring commonality (hashing) also produces uniform distribution over all hash file system servers. This even distribution permits a large array of servers to function as a gigantic web server farm with an evenly distributed load. In other applications, the hash file system of the present invention can be used as a network accelerator inasmuch as it can be used to reduce network traffic by sending proxies (hashes) for data instead of the data itself. A large percentage of current network traffic is redundant data moving between locations. Sending proxies for the data would allow effective local caching mechanisms to operate, possibly reducing the traffic on the Internet by several orders of magnitude.
  • [0018]
    As particularly disclosed herein, the hash file system and method of the present invention may be implemented using 160 bit hashsums as universal pointers. This differs from conventional file systems which use pointers assigned from a central authority (i.e. in Unix a 32 bit “inode” is assigned by the kernel's file systems in a lock-step operation to assure uniqueness). In the hash file system of the present invention, these 160 bit hashsums are assigned without a central authority (i.e. without locking, without synchronization) by a hashing algorithm.
  • [0019]
    Known hashing algorithms produce probabilistically unique numbers that uniformly span a range of values. In the case of the hash function SHA-1, that range is between 0 and 10e48. This hashing operation is done by examining only the contents of the data being stored and, therefore, can be done in complete isolation, asynchronously, and without interlocking.
  • [0020]
    Hashing is an operation that can be verified by any component of the system, eliminating the need for trusted operations across those components. The hash file system and method of the present invention disclosed herein is, therefore, functional to eliminate the critical bottleneck of conventional large scale distributed file systems, that is, a trusted encompassing central authority. It permits the construction of a large scale distributed file system with no limits on simultaneous read/write operations, that can operate without risk of incoherence and without the limitation of certain conventional bottlenecks.
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • [0021]
    The aforementioned and other features and objects of the present invention and the manner of attaining them will become more apparent, and the invention itself will be best understood by reference to the following description of a preferred embodiment taken in conjunction with the accompanying drawings, wherein:
  • [0022]
    [0022]FIG. 1 is a high level illustration of a representative networked computer environment in which the system and method of the present invention may be implemented;
  • [0023]
    [0023]FIG. 2 is a more detailed conceptual representation of a possible operating environment for utilization of the system and method of the present invention wherein files maintained on any number of computers or data centers may be stored in a decentralized computer system through an Internet connection to a number of Redundant Arrays of Independent Nodes (“RAIN”) racks located, for example, at geographically diverse locations;
  • [0024]
    [0024]FIG. 3 is a logic flow chart depicting the steps in the entry of a computer file into the hash file system of the present invention wherein the hash value for the file is checked against hash values for files previously maintained in a set, or database;
  • [0025]
    [0025]FIG. 4 is a further logic flow chart depicting the steps in the breakup of a file or other data sequence into hashed pieces resulting in the production of a number of data pieces as well as corresponding probabilistically unique hash values for each piece;
  • [0026]
    [0026]FIG. 5 is another logic flow chart depicting the comparison of the hash values for each piece of a file to existing hash values in the set (or database), the production of records showing the equivalence of a single hash value for all file pieces with the hash values of the various pieces and whereupon new data pieces and corresponding new hash values are added to the set;
  • [0027]
    [0027]FIG. 6 is yet another logic flow chart illustrating the steps in the comparison of file hash or directory list hash values to existing directory list hash values and the addition of new file or directory list hash values to the set directory list;
  • [0028]
    [0028]FIG. 7 is a comparison of the pieces of a representative computer file with their corresponding hash values both before and after editing of a particular piece of the exemplary file;
  • [0029]
    [0029]FIG. 8 is a conceptual representation of the fact that composite data which may be derived by means of the system and method of the present invention is effectively the same as the data represented explicitly but may instead be created by a “recipe” such as the concatenation of data represented by its corresponding hashes or the result of a function using the data represented by the hashes;
  • [0030]
    [0030]FIG. 9 is another conceptual representation of how the hash file system and method of the present invention my be utilized to organize data to optimize the reutilization of redundant sequences through the use of hash values as pointers to the data they represent and wherein data may be represented either as explicit byte sequences (atomic data) or as groups of sequences (composites);
  • [0031]
    [0031]FIG. 10 is a simplified diagram illustrative of a hash file system address translation function for an exemplary 160 bit hash value;
  • [0032]
    [0032]FIG. 11 is a simplified exemplary illustration of an index stripe splitting function for use with the system and method of the present invention;
  • [0033]
    [0033]FIG. 12 is a simplified illustration of the overall functionality of the system and method of the present invention for use in the backup of data for a representative home computer having a number of program and document files on Day 1 and wherein one of the document files is edited on Day 2 together with the addition of a third document file; and
  • [0034]
    [0034]FIG. 13 illustrates the comparison of various pieces of a particular document file marked by a number of “sticky bytes” both before and following editing wherein one of the pieces is thereby changed while other pieces remain the same.
  • DESCRIPTION OF A REPRESENTATIVE EMBODIMENT
  • [0035]
    In a particular implementation of the hash file system and method of the present invention as disclosed herein, its application is directed toward a high availability, high reliability data storage system that leverages rapid advances in commodity computing devices and the robust nature of internetwork technology such as the Internet. Particularly disclosed herein is a hash file system that manages the correspondence of one or more block(s) of data (including but not limited to files, directories, drive images, software applications, digitized voice, and rich media content) together with one or more symbol(s) for that block of data, wherein the symbol may be a number, hash, checksum, binary sequence, or other identifier that is derived from the block of data itself and is statistically, probabilistically, or otherwise effectively unique to that block of data. The system itself works on any computer system including, without limitation: personal computers; supercomputers; distributed or non-distributed networks; storage area networks (“SAN”) using IDE, SCSI or other disk buses; network attached storage (“NAS”) or other systems capable of storing and/or processing data.
  • [0036]
    In a particular implementation of the hash file system disclosed herein, the symbol(s) may be derived using one or more hash or checksum generating engines, programs, or algorithms, including but not limited to MD4, MD5, SHA, SHA-1, or their derivatives. Further, the symbol(s) may comprise parts of variable or invariable length symbols derived using a hash or checksum generating engine, program, or algorithm, including but not limited to MD4, MD5, SHA, SHA-1, or other methods of generating probabilistically unique identifiers based on data content. In a particular implementation disclosed herein, file seeks, or lookups for retrieving data or checking on the existence/availability of data, may be accelerated by looking at all or a smaller portion of the symbol, with the symbol portion indicating or otherwise providing the routing information for finding, retrieving, or checking on the existence/availability of the data.
  • [0037]
    Further disclosed herein is a system and method for a hash file system wherein the symbols allow for the identification of redundant copies within the system and/or allow for the identification of copies within the system redundant with data presented to the system for filing and storage. The symbols allow for the elimination of, or allow for the screening of, redundant copies of the data and/or parts of the data in the system or in data and/or parts of data presented to the system, without loss of data integrity and can provide for the even distribution of data over available storage for the system. The system and method of the present invention as disclosed herein requires no central operating point and balances processing and/or input/output (“I/O”) load across all computers, supercomputers, or other devices capable of storing and/or processing data attached to the system. The screening of redundant copies of the data and/or parts of the data provided herein allows for the creation, repetitive creation, or retention of intelligent boundaries for screening other data in the system, future data presented to the system, or future data stored by the system.
  • [0038]
    The present invention is illustrated and described in terms of a distributed computing environment such as an enterprise computing system using public communication channels such as the Internet. However, an important feature of the present invention is that it is readily scaled upwardly and downwardly to meet the needs of a particular application. Accordingly, unless specified to the contrary the present invention is applicable to significantly larger, more complex network environments as well as small network environments such as conventional LAN systems.
  • [0039]
    With reference now to FIG. 1, the present invention may be utilized in conjunction with a novel data storage system on a network 10. In this figure, an exemplary internetwork environment 10 may include the Internet which comprises a global internetwork formed by logical and physical connection between multiple wide area networks (“WANs”) 14 and local area networks (“LANs”) 16. An Internet backbone 12 represents the main lines and routers that carry the bulk of the data traffic. The backbone 12 is formed by the largest networks in the system that are operated by major Internet service providers (“ISPs”) such as GTE, MCI, Sprint, UUNet, and America Online, for example. While single connection lines are used to conveniently illustrate WANs 14 and LANs 16 connections to the Internet backbone 12, it should be understood that in reality, multi-path, routable physical connections exist between multiple WANs 14 and LANs 16. This makes internetwork 10 robust when faced with single or multiple failure points.
  • [0040]
    It is important to distinguish network connections from internal data pathways implemented between peripheral devices within a computer. A “network” comprises a system of general purpose, usually switched physical connections that enable logical connections between processes operating on nodes 18. The physical connections implemented by a network are typically independent of the logical connections that are established between processes using the network. In this manner, a heterogeneous set of processes ranging from file transfer, mail transfer, and the like can use the same physical network. Conversely, the network can be formed from a heterogeneous set of physical network technologies that are invisible to the logically connected processes using the network. Because the logical connection between processes implemented by a network is independent of the physical connection, internetworks are readily scaled to a virtually unlimited number of nodes over long distances.
  • [0041]
    In contrast, internal data pathways such as a system bus, peripheral component interconnect (“PCI”) bus, Intelligent Drive Electronics (“IDE”) bus, small computer system interface (“SCSI”) bus, and the like define physical connections that implement special-purpose connections within a computer system. These connections implement physical connections between physical devices as opposed to logical connections between processes. These physical connections are characterized by limited distance between components, limited number of devices that can be coupled to the connection, and constrained format of devices that can be connected over the connection.
  • [0042]
    In a particular implementation of the present invention, storage devices may be placed at nodes 18. The storage at any node 18 may comprise a single hard drive, or may comprise a managed storage system such as a conventional RAID device having multiple hard drives configured as a single logical volume. Significantly, the present invention manages redundancy operations across nodes, as opposed to within nodes, so that the specific configuration of the storage within any given node is less relevant.
  • [0043]
    Optionally, one or more of the nodes 18 may implement storage allocation management (“SAM”) processes that manage data storage across nodes 18 in a distributed, collaborative fashion. SAM processes preferably operate with little or no centralized control for the system as whole. SAM processes provide data distribution across nodes 18 and implement recovery in a fault-tolerant fashion across network nodes 18 in a manner similar to paradigms found in RAID storage subsystems.
  • [0044]
    However, because SAM processes operate across nodes rather than within a single node or within a single computer, they allow for greater fault tolerance and greater levels of storage efficiency than conventional RAID systems. For example, SAM processes can recover even where a network node 18, LAN 16, or WAN 14 become unavailable. Moreover, even when a portion of the Internet backbone 12 becomes unavailable through failure or congestion, the SAM processes can recover using data distributed on nodes 18 that remain accessible. In this manner, the present invention leverages the robust nature of internetworks to provide unprecedented availability, reliability, fault tolerance and robustness.
  • [0045]
    With reference additionally now to FIG. 2, a more detailed conceptual view of an exemplary network computing environment in which the present invention is implemented is depicted. The internetwork 10 of the preceding figure (or Internet 118 in this figure) enables an interconnected network 100 of a heterogeneous set of computing devices and mechanisms 102 ranging from a supercomputer or data center 104 to a hand-held or pen-based device 114. While such devices have disparate data storage needs, they share an ability to retrieve data via network 100 and operate on that data within their own resources. Disparate computing devices 102 including mainframe computers (e.g., VAX station 106 and IBM AS/400 station 116) as well as personal computer or workstation class devices such as IBM compatible device 108, Macintosh device 110 and laptop computer 112 are readily interconnected via internetwork 10 and network 100. Although not illustrated, mobile and other wireless devices may be coupled to the internetwork 10.
  • [0046]
    Internet-based network 120 comprises a set of logical connections, some of which are made through Internet 118, between a plurality of internal networks 122. Conceptually, Internet-based network 120 is akin to a WAN 14 (FIG. 1) in that it enables logical connections between geographically distant nodes. Internet-based networks 120 may be implemented using the Internet 118 or other public and private WAN technologies including leased lines, Fibre Channel, and the like.
  • [0047]
    Similarly, internal networks 122 are conceptually akin to LANs 16 (FIG. 1) in that they enable logical connections across a more limited distance than WAN 14. Internal networks 122 may be implemented using various LAN technologies including Ethernet, Fiber Distributed Data Interface (“FDDI”), Token Ring, Appletalk, Fibre Channel, and the like.
  • [0048]
    Each internal network 122 connects one or more redundant arrays of independent nodes (RAIN) elements 124 to implement RAIN nodes 18 (FIG. 1). Each RAIN element 124 comprises a processor, memory, and one or more mass storage devices such as hard disks. RAIN elements 124 also include hard disk controllers that may be conventional IDE or SCSI controllers, or may be managing controllers such as RAID controllers. RAIN elements 124 may be physically dispersed or co-located in one or more racks sharing resources such as cooling and power. Each node 18 (FIG. 1) is independent of other nodes 18 in that failure or unavailability of one node 18 does not affect availability of other nodes 18, and data stored on one node 18 may be reconstructed from data stored on other nodes 18.
  • [0049]
    In a particular exemplary implementation, the RAIN elements 124 may comprise computers using commodity components such as Intel-based microprocessors mounted on a motherboard supporting a PCI bus and 256 megabytes of random access memory (“RAM”) housed in a conventional AT or ATX case. SCSI or IDE controllers may be implemented on the motherboard and/or by expansion cards connected to the PCI bus. Where the controllers are implemented only on the motherboard, a PCI expansion bus may be optionally used. In a particular implementation, the motherboard may implement two mastering EIDE channels and a PCI expansion card which is used to implement two additional mastering EIDE channels so that each RAIN element 124 includes up to four or more EIDE hard disks. In the particular implementation, each hard disk may comprise an 80 gigabyte hard disk for a total storage capacity of 320 gigabytes or more per RAIN element. The hard disk capacity and configuration within RAIN elements 124 can be readily increased or decreased to meet the needs of a particular application. The casing also houses supporting mechanisms such as power supplies and cooling devices (not shown).
  • [0050]
    Each RAIN element 124 executes an operating system. In a particular implementation, the UNIX or UNIX variant operating system such as Linux may be used. It is contemplated, however, that other operating systems including DOS, Microsoft Windows, Apple Macintosh OS, OS/2, Microsoft Windows NT and the like may be equivalently substituted with predictable changes in performance. The operating system chosen forms a platform for executing application software and processes, and implements a file system for accessing mass storage via the hard disk controller(s). Various application software and processes can be implemented on each RAIN element 124 to provide network connectivity via a network interface using appropriate network protocols such as user datagram protocol (“UDP”), transmission control protocol (TCP), Internet protocol (IP) and the like.
  • [0051]
    With reference additionally now to FIG. 3, a logic flow chart is shown depicting the steps in the entry of a computer file into the hash file system of the present invention and wherein the hash value for the file is checked against hash values for files previously maintained in a set, or database.
  • [0052]
    The process 200 begins by entry of a computer file data 202 (e.g. “File A”) into the hash file system (“HFS”) of the present invention upon which a hash function is performed at step 204. The data 206 representing the hash of File A is then compared to the contents of a set containing hash file values at decision step 208. If the data 206 is already in the set, then the file's hash value is added to a directory list at step 210. The contents of the set 212 comprising hash values and corresponding data is provided in the form of existing hash values 214 for the comparison operation of decision step 208. On the other hand, if the hash value for File A is not currently in the set, the file is broken into hashed pieces (as will be more fully described hereinafter) at step 216.
  • [0053]
    With reference additionally now to FIG. 4, a further logic flow chart is provided depicting the steps in the process 300 for breakup of a digital sequence (e.g. a file or other data sequence) into hashed pieces. This process 300 ultimately results in the production of a number of data pieces as well as corresponding probabilistically unique hash values for each piece.
  • [0054]
    The file data 302 is divided into pieces based on commonality with other pieces in the system or the likelihood of pieces being found to be in common in the future at step 304. The results of the operation of step 304 upon the file data 302 is, in the representative example shown, the production of four file pieces 306 denominated A1 through A5 inclusively. Each of the file pieces 306 is then operated on at step 308 by placing it through individual hash function operations to assign a probabilistically unique number to each of the pieces 306 A1 through A5. The results of the operation at step 308 is that each of the pieces 306 (A1 through A5) has an associated, probabilistically unique hash value 310 (shown as A1 Hash through A5 Hash respectively). The file division process of step 304 is described in greater detail hereinafter in conjunction with the unique “sticky byte” operation also disclosed herein.
  • [0055]
    With reference additionally now to FIG. 5, another logic flow chart is shown depicting a comparison process 400 for the hash values 310 of each piece 306 of the file to those of existing hash values 214 maintained in the set 212. Particularly, at step 402, the hash values 310 for each piece 306 of the file are compared to existing hash values 214 and new hash values 408 and corresponding new data pieces 406 are added to the set 212. In this way, hash values 408 not previously present in the database set 212 are added together with their associated data pieces 406. The process 400 also results in the production of records 404 showing the equivalence of a single hash value for all file pieces with the hash values 310 of the various pieces 306.
  • [0056]
    With reference additionally now to FIG. 6, yet another logic flow chart is shown illustrating a process 500 for the comparison of file hash or directory list hash values to existing directory list hash values and the addition of new file or directory list hash values to the database directory list. The process 500 operates on stored data 502 which comprises an accumulated list of file names, file meta-data (e.g. date, time, file length, file type etc.) and the file's hash value for each item in a directory. At step 504, the hash function is run upon the contents of the directory list. Decision step 506 is operative to determine whether or not the hash value for the directory list is in the set 212 of existing hash values 214. If it is, then the process 500 returns to add another file hash or directory list hash to a directory list. Alternatively, if the hash value for the directory list is not already in the database set 212, the hash value and data for the directory list are added to the database 212 set at step 508.
  • [0057]
    With reference additionally now to FIG. 7, a comparison 600 of the pieces 306 of a representative computer file (i.e. “File A”) with their corresponding hash values 310 is shown both before and after editing of a particular piece of the exemplary file. In this example, the record 404 contains the hash value of File A as well as the hash values 310 of each of the pieces of the file A1 through A5. A representative edit of the File A may produce a change in the data for piece A2 (now represented by A2-b) of the file pieces 306A along with a corresponding change in the hash value A2-b of the hash values 310A. The edited file piece now produces an updated record 404A which includes the modified hash value of File A and the modified hash value of piece A2-b.
  • [0058]
    With reference additionally now to FIG. 8, a conceptual representation 700 is shown illustrative of the fact that composite data (such as composite data 702 and 704) derived by means of the system and method of the present invention, is effectively the same as the data 706 represented explicitly but is instead created by a “recipe”, or formula. In the example shown, this recipe includes the concatenation of data represented by its corresponding hashes 708 or the result of a function using the data represented by the hashes. The data blocks 706 may be variable length quantities as shown and the hash values 708 are derived from their associated data blocks. As previously stated, the hash values 708 are a probabilistically unique identification of the corresponding data pieces but truly unique identifications can be used instead or intermixed therewith. It should also be noted that the composite data 702, 704 can also reference other composite data many levels deep while the hash values 708 for the composite data can be derived from the value of the data the recipe creates or the hash value of the recipe itself.
  • [0059]
    With reference additionally now to FIG. 9, another conceptual representation 800 is shown of how the hash file system and method of the present invention may be utilized to organize data 802 to optimize the reutilization of redundant sequences through the use of hash values 806 as pointers to the data they represent and wherein data 802 may be represented either as explicit byte sequences (atomic data) 808 or as groups of sequences (composites) 804.
  • [0060]
    The representation 800 illustrates the tremendous commonality of recipes and data that gets reused at every level. The basic structure of the hash file system of the present invention is essentially that of a “tree” or “bush” wherein the hash values 806 are used instead of conventional pointers. The hash values 806 are used in the recipes to point to the data or another hash value that could also itself be a recipe. In essence then, recipes can point to other recipes that point to still other recipes that ultimately point to some specific data that may, itself, point to other recipes that point to even more data, eventually getting down to nothing but data.
  • [0061]
    With reference additionally now to FIG. 10, a simplified diagram 900 is shown illustrative of a hash file system address translation function for an exemplary 160 bit hash value 902. The hash value 902 includes a data structure comprising a front portion 904 and a back portion 906 as shown and the diagram 900 illustrates a particular “0 of 1” operation that is used for enabling the use of the hash value 902 to go to the location of the particular node in the system that contains the corresponding data.
  • [0062]
    The diagram 900 illustrates how the front portion 904 of the hash value 902 data structure may be used to indicate the hash prefix to stripe identification (“ID”) 908 and how that is, in turn, utilized to map the stripe ID to IP address and the ID class to IP address 910. In this example, the “S2” indicates stripe 2 of index Node 37 912. The index stripe 912 of Node 37 then indicates stripe 88 of data Node 73 indicated by the reference numeral 914. In operation then, a portion of the hash value 902 itself may be used to indicate which node in the system contains the relevant data, another portion of the hash value 902 may be used to indicate which stripe of data at that particular node and yet another portion of the hash value 902 to indicate where within that stripe the data resides. Through this three step process, it can rapidly be determined if the data represented by the hash value 902 is already present in the system.
  • [0063]
    With reference additionally now to FIG. 11, a simplified exemplary illustration of an index stripe splitting function 1000 is shown for use with the system and method of the present invention. In this illustration, an exemplary function 1000 is shown that may be used to effectively split a stripe 1002 (S2) into two stripes 1004 (S2) and 1006 (S7) should one stripe become too full. In this example, the odd entries have been moved to stripe 1006 (S7) while the even ones remain in stripe 1004. This function 1000 is one example of how stripe entries may be handled as the overall system grows in size and complexity.
  • [0064]
    With reference additionally now to FIG. 12, a simplified illustration 1100 of the overall functionality of the system and method of the present invention is shown for use, for example, in the backup of data for a representative home computer having a number of program and document files 1102A and 1104A on Day 1 and wherein the program files 1102B remain the same on Day 2 while one of the document files 1104B is edited on Day 2 (Y.doc) together with the addition of a third document file (Z.doc).
  • [0065]
    The illustration 1100 shows the details of how a computer file system may be broken into pieces and then listed as a series of recipes on a global data protection network (“gDPN”) to reconstruct the original data from the pieces. This very small computer system is shown in the form of a “snapshot” on “Day 1” and then subsequently on “Day 2”. On “Day 1”, the “program files H5” and “my documents H6” are illustrated by numeral 1106, with the former being represented by a recipe 1108 wherein a first executable file is represented by a hash value H1 1114 and a second represented by a hash value H2 1112. The document files are represented by hash value H6 1110 with the first document being represented by hash value H3 1118 and the second by hash value H4 1116. Thereafter on “Day 2”, the “program files H5” and “my documents” H10 indicated by numeral 1120 show that the “program files H5” have not changed, but the “my document H10” have. H10 indicated by numeral 1122 shows the “X.doc” is still represented by hash value H3 1118 while “Y.doc” is now represented by hash value H8 at number 1124. New document file “Z.doc” is now represented by hash value H9 at numeral 1126.
  • [0066]
    In this example, it can be seen that on Day 2, some of the files have changed, while others have not. In the files that have changed, some of the pieces of them have not changed while other pieces have. Through the use of the hash file system of the present invention, a “snap shot” of the computer system can be made on Day 1 (producing the necessary recipes for reconstruction of the computer files as they exist then) and then on Day 2 through the reuse of some of the previous day's recipes together with the reformulation of others and the addition of new ones to describe the system at that time. In this manner, the computer system may be recreated in its entirety at any point in time on both Day 1 or Day 2 as well as on any subsequent day.
  • [0067]
    With reference additionally now to FIG. 13, a comparison 1200 of various pieces of a particular document file marked by a number of “sticky bytes” 1204 is shown both before (Day 1 1202A) and following editing (Day 2 1202B) wherein one of the pieces is thereby changed while other pieces remain the same.
  • [0068]
    For example, on Day 1, file 1202A comprises variable length pieces 1206 (1.1), 1208 (1.2), 1210 (2.1), 1212 (2.), 1214 (2.3) and 1216 (3.1). On Day 2, pieces 1206, 1208, 1210, 1214 and 1216 remain the same (thus having the same hash values) while piece 1212 has now been edited to produce piece 1212A (thus having a differing hash value).
  • [0069]
    Data sticky bytes (or “sticky points”) are a unique, fully automated way to sub-divide computer files such that common elements may be found on multiple related and unrelated computers without the need for communication between the computers. The means in which data sticky points are found is completely mathematical in nature and performs equally well regardless of the data content of the files. In the hash file system of the present invention, all data objects may be indexed, stored and retrieved using, for example (but not limited to), an industry standard checksum such as: MD4, MD5, SHA, or SHA-1. In operation, if two files have the same checksum, it may be considered to be highly likely that they are the same file. Using the system and method disclosed herein, data sticky points may be produced with a standard mathematical distribution and with standard deviations that are a small percentage of the target size.
  • [0070]
    A data sticky point is a statistically infrequent arrangement of n bytes. In this case, an example is given with 32 bytes because of its ease in implementation in current microprocessor technology.
  • [0071]
    A rolling hash of 32 bits could be generated for the file “f”.
    //f[i] = is the ith byte of the file “f”.
    //scramble is a 256 entry array of integers with each
    //being 32 bits wide;
    //these integers are typically chosen to uniformly //span
    the range.
    int t=8 //target number of trailing zeros
    int hash = 0;
    int sticky_bits;
    for(int i=O; i<filesize; i++)
     hash = hash >> 1 |scramble[f[i]];
     //At every byte in the file, hash represents the
    //rolling hash of the file.
     sticky_bits = (hash − 1){circumflex over ( )}hash;
     //sticks_bits is a variable which will have the //number
    of ones in the hash
     //that correspond to the number of trailing zeros in
    //the “hash”.
     number_of_bits = count_ones(stick-bits);
    if(number_of_bits > t)
      output_sticky_point(i);
    }
  • [0072]
    A sticky point is defined to be a rolling hash with at least the number of trailing zeros as the target number with the hash represented in binary. Statistically speaking, this algorithm will find points that are spaced at 2{circumflex over ( )}t, where t is the target number of trailing zeros. For this example where t=8, the algorithm will find, on average, sticky points that are spaced at 2{circumflex over ( )}8=256 bytes apart.
  • [0073]
    A rolling hash of 32 bits may be generated for the f file where:
    f[i] = is the ith byte of the file f.
    scramble is a 256 entry array of random elements with each
    being n bits wide;
    int t=8 // target number of trailing zeros
    int target_distance = 256; // 2 to the power of 8
    int hash = 0;
    int sticky_bits;
    int distance = 0;
    int last_point = 0;
    for(int i=O; i<filesize; i++) {
      hash = hash >> 1| scramble[f[i]];
      //At every byte in the file hash represents the
    //rolling hash of the file.
      sticky_bits = (hash − 1){circumflex over ( )}hash;
      //sticks_bits is a variable which will have the
    //number of ones that correspond to the number of
    //trailing zeros in the “hash”.
      number_of_bits = count_ones(stick_bits);
      distance = i-last_point;
    if(number_of_bits * distance/target_distance >t)
      last_point =i;
    output_sticky_point(i);
      }
    }
  • [0074]
    While the hashing function utilized to implement the hash file system of the present invention requires a moderately complex computation, it is well within the capability of present day computer systems. Hashing functions are inherently probabilistic and any hashing function might possibly produce incorrect results when two different data objects happen to have the same hash value. However, the system and method herein disclosed mitigates this problem by using well known and researched hashing functions that reduce the probability of collision down to levels acceptable for reliable use (i.e. one chance in a trillion trillion), far less than the error rates otherwise tolerated in conventional computer hardware operations.
  • [0075]
    Although as used herein, the term “Internet infrastructure” encompasses a variety of hardware and software mechanisms, the term primarily refers to routers, router software, and physical links between these routers that function to transport data packets from one network node to another. As also used herein, a “digital sequence” may comprise, without limitation, computer program files, computer applications, data files, network packets, streaming data such as multimedia (including audio and video), telemetry data and any other form of data which can be represented by a digital or numeric sequence. The probabilistically unique identifiers produced by means of the hash file system and method of the present invention may also be used as URLs in network applications.
  • [0076]
    While there have been described above the principles of the present invention in conjunction with specific implementations and applications of the system and method of the present invention, it is to be clearly understood that the foregoing description is made only by way of example and not as a limitation to the scope of the invention. Particularly, it is recognized that the teachings of the foregoing disclosure will suggest other modifications to those persons skilled in the relevant art. Such modifications may involve other features which are already known per se and which may be used instead of or in addition to features already described herein. Although claims have been formulated in this application to particular combinations of features, it should be understood that the scope of the disclosure herein also includes any novel feature or any novel combination of features disclosed either explicitly or implicitly or any generalization or modification thereof which would be apparent to persons skilled in the relevant art, whether or not such relates to the same invention as presently claimed in any claim and whether or not it mitigates any or all of the same technical problems as confronted by the present invention. The applicants hereby reserve the right to formulate new claims to such features and/or combinations of such features during the prosecution of the present application or of any further application derived therefrom.

Claims (40)

What is claimed is:
1. A method for managing data comprising:
producing a probabilistically unique identifier for a digital sequence; and
comparing said probabilistically unique identifier to a list of other identifiers with their corresponding digital sequences.
2. The method of claim 1 further comprising:
adding said probabilistically unique identifier to said list if said probabilistically unique identifier is not previously in said list.
3. The method of claim 1 further comprising:
removing said probabilistically unique identifier from said list if said probabilistically unique identifier is previously in said list.
4. The method of claim 2 further comprising:
adding said digital sequence corresponding to said probabilistically unique identifier to said list.
5. The method of claim 3 further comprising:
removing said digital sequence corresponding to said probabilistically unique identifier from said list.
6. The method of claim 4 further comprising:
adding a correspondence between said digital sequence and said probabilistically unique identifier for that sequence.
7. The method of claim 1 wherein said step of producing comprises:
hashing said digital sequence to produce said probabalistically unique identifier.
8. The method of claim 7 wherein said step of hashing is carried out by means of an industry standard digest algorithm.
9. The method of claim 8 wherein said step of hashing is carried out by one of an MD4, MD5, SHA or SHA-1 algorithm.
10. The method of claim 1 wherein said step of producing comprises:
generating a checksum for said digital sequence to produce said probabilistically unique identifier.
11. The method of claim 1 wherein said digital sequence is descriptive meta data of at least one other digital sequence.
12. The method of claim 1 wherein said digital sequence is descriptive meta data of at least one probabilistically unique identifier.
13. The method of claim 1 wherein said digital sequence describes a method that represents at least one digital sequence.
14. A method for managing data comprising:
dividing a digital sequence into a plurality of shorter digital sequences; and
producing probabilistically unique identifiers for each said plurality of shorter digital sequences; and
comparing said probabilistically unique identifiers to a list of other identifiers.
15. The method of claim 14 further comprising the step of:
dividing said digital sequence into a plurality of shorter digital sequences; and
producing a like plurality of probabilistically unique identifiers corresponding to each of said plurality of shorter digital sequences.
16. The method of claim 14 further comprising;
comparing each plurality of identifiers to said list.
17. The method of claim 14 wherein said step of dividing produces said shorter digital sequences having individually variable lengths.
18. The method of claim 14 wherein said step of dividing is based on the content of said digital sequence.
19. The method of claim 14 wherein said step of dividing is based on meta data describing said digital sequence.
20. The method of claim 14 wherein said step of dividing produces said shorter digital sequences having substantially invariable lengths.
21. The method of claim 14 wherein said step of producing said like plurality of probabilistically unique identifiers comprises:
individually hashing said shorter digital sequences to produce said like plurality of probabilistically unique identifiers.
22. The method of claim 14 further comprising the step of:
adding said plurality of shorter digital sequences and said corresponding like plurality of probabilistically unique identifiers to said list.
23. The method of claim 14 further comprising the step of:
removing said plurality of shorter digital sequences and said corresponding like plurality of probabilistically unique identifiers from said list.
24. The method of claim 9 further comprising the step of:
utilizing at least a portion of said probabilistically unique identifier as an indicator to a location in said list for said step of comparing.
25. A computer program product comprising:
a computer usable medium having computer readable code embodied therein for managing data, said computer program product comprising:
computer readable program code devices configured to cause a computer to effect producing a probabilistically unique identifier for a digital sequence; and
computer readable program code devices configured to cause a computer to effect comparing said probabilistically unique identifier to a list of other identifiers corresponding to other digital sequences.
26. The computer program product of claim 25 further comprising:
computer readable program code devices configured to cause a computer to effect adding said probabilistically unique identifier to said list if said probabilistically unique identifier is not previously in said list.
27. The computer program product of claim 26 further comprising:
computer readable program code devices configured to cause a computer to effect adding said corresponding digital sequence to said list.
28. The computer program product of claim 25 wherein said computer readable program code devices configured to cause said computer to effect producing comprises:
computer readable program code devices configured to cause a computer to effect hashing said digital sequence to produce said probabilistically unique identifier.
29. The computer program product of claim 28 wherein said computer readable program code devices configured to cause a computer to effect hashing is carried out by means of an industry standard digest algorithm.
30. The computer program product of claim 29 wherein said computer readable program code devices configured to cause a computer to effect hashing is carried out by one of an MD4, MD5, SHA or SHA-1 algorithm.
31. The computer program product of claim 25 wherein said computer readable program code devices configured to cause a computer to effect producing comprises:
computer readable program code devices configured to cause a computer to effect generating a checksum for said digital sequence to produce said probabilistically unique identifier.
32. The computer program product of claim 25 further comprising:
computer readable program code devices configured to cause a computer to effect creating a directory list containing said probabilistically unique identifier for said digital sequence.
33. The computer program product of claim 25 further comprising:
computer readable program code devices configured to cause a computer to effect dividing said digital sequence into a plurality of shorter digital sequences; and
computer readable program code devices configured to cause a computer to effect producing a like plurality of probabilistically unique identifiers corresponding to each of said plurality of shorter digital sequences.
34. The computer program product of claim 33 wherein said computer readable program code devices configured to cause a computer to effect dividing produces said shorter digital sequences having individually variable length.
35. The computer program product of claim 33 wherein said computer readable program code devices configured to cause a computer to effect dividing produces said shorter digital sequences having substantially invariable length.
36. The computer program product of claim 33 wherein said computer readable program code devices configured to cause a computer to effect producing said like plurality of probabilistically unique identifiers comprises:
computer readable program code devices configured to cause a computer to effect individually hashing said shorter digital sequences to produce said like plurality of probabilistically unique identifiers.
37. The computer program product of claim 33 further comprising:
computer readable program code devices configured to cause a computer to effect adding said plurality of shorter digital sequences and said corresponding like plurality of probabilistically unique identifiers to said list.
38. The computer program product of claim 25 further comprising:
computer readable program code devices configured to cause a computer to effect utilizing at least a portion of said probabilistically unique identifier as an index into a table of locations for said list for said step of comparing.
39. A method for managing data comprising:
producing a probabilistically unique identifier for a digital sequence; and
comparing said probabilistically unique identifier to a list of other identifiers corresponding to other digital sequences.
40. The method of claim 39 further comprising:
adding said probabilistically unique identifier to said list if said probabilistically unique identifier is not previously in said list.
US10757753 2000-02-18 2004-01-14 Hash file system and method for use in a commonality factoring system Abandoned US20040148306A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US18376200 true 2000-02-18 2000-02-18
US09777150 US6704730B2 (en) 2000-02-18 2001-02-05 Hash file system and method for use in a commonality factoring system
US10757753 US20040148306A1 (en) 2000-02-18 2004-01-14 Hash file system and method for use in a commonality factoring system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US10757753 US20040148306A1 (en) 2000-02-18 2004-01-14 Hash file system and method for use in a commonality factoring system

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US09777150 Continuation US6704730B2 (en) 2000-02-18 2001-02-05 Hash file system and method for use in a commonality factoring system

Publications (1)

Publication Number Publication Date
US20040148306A1 true true US20040148306A1 (en) 2004-07-29

Family

ID=26879495

Family Applications (2)

Application Number Title Priority Date Filing Date
US09777150 Active 2022-03-17 US6704730B2 (en) 2000-02-18 2001-02-05 Hash file system and method for use in a commonality factoring system
US10757753 Abandoned US20040148306A1 (en) 2000-02-18 2004-01-14 Hash file system and method for use in a commonality factoring system

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US09777150 Active 2022-03-17 US6704730B2 (en) 2000-02-18 2001-02-05 Hash file system and method for use in a commonality factoring system

Country Status (1)

Country Link
US (2) US6704730B2 (en)

Cited By (66)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20010034795A1 (en) * 2000-02-18 2001-10-25 Moulton Gregory Hagan System and method for intelligent, globally distributed network storage
US20030182291A1 (en) * 2002-03-20 2003-09-25 Sreenath Kurupati Method and data structure for a low memory overhead database
US20040225655A1 (en) * 2000-11-06 2004-11-11 Moulton Gregory Hagan System and method for unorchestrated determination of data sequences using sticky factoring to determine breakpoints in digital sequences
US20060031529A1 (en) * 2004-06-03 2006-02-09 Keith Robert O Jr Virtual application manager
US20060047946A1 (en) * 2004-07-09 2006-03-02 Keith Robert O Jr Distributed operating system management
US20060047716A1 (en) * 2004-06-03 2006-03-02 Keith Robert O Jr Transaction based virtual file system optimized for high-latency network connections
US20060224545A1 (en) * 2005-03-04 2006-10-05 Keith Robert O Jr Computer hardware and software diagnostic and report system
US20060224544A1 (en) * 2005-03-04 2006-10-05 Keith Robert O Jr Pre-install compliance system
US7143251B1 (en) * 2003-06-30 2006-11-28 Data Domain, Inc. Data storage using identifiers
US20070233633A1 (en) * 2005-03-04 2007-10-04 Keith Robert O Jr Computer hardware and software diagnostic and report system
US20070274315A1 (en) * 2006-05-24 2007-11-29 Keith Robert O System for and method of securing a network utilizing credentials
US20080077630A1 (en) * 2006-09-22 2008-03-27 Keith Robert O Accelerated data transfer using common prior data segments
US20080077724A1 (en) * 2006-09-22 2008-03-27 Parthasarathy Sarangam Interrupt coalescing control scheme
US7386616B1 (en) * 2003-05-09 2008-06-10 Google Inc. System and method for providing load balanced processing
US20080229037A1 (en) * 2006-12-04 2008-09-18 Alan Bunte Systems and methods for creating copies of data, such as archive copies
US20080243958A1 (en) * 2006-12-22 2008-10-02 Anand Prahlad System and method for storing redundant information
WO2008147516A1 (en) * 2007-05-22 2008-12-04 Network Appliance, Inc. System and method for on-the-fly elimination of redundant date
US20090007261A1 (en) * 2007-06-29 2009-01-01 Smith Mark A Receiving data in a data store in a server computer system
US20090051705A1 (en) * 2007-08-20 2009-02-26 Shih-Kuang Tsai Method for automatically memorizing picture orientation
US20090234892A1 (en) * 2008-03-14 2009-09-17 International Business Machines Corporation Method and system for assuring integrity of deduplicated data
US20090271454A1 (en) * 2008-04-29 2009-10-29 International Business Machines Corporation Enhanced method and system for assuring integrity of deduplicated data
US20090319585A1 (en) * 2008-06-24 2009-12-24 Parag Gokhale Application-aware and remote single instance data management
US20100005259A1 (en) * 2008-07-03 2010-01-07 Anand Prahlad Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices
US20100058002A1 (en) * 2008-08-27 2010-03-04 Netapp, Inc. System and method for file system level compression using compression group descriptors
US20100082672A1 (en) * 2008-09-26 2010-04-01 Rajiv Kottomtharayil Systems and methods for managing single instancing data
US20100169287A1 (en) * 2008-11-26 2010-07-01 Commvault Systems, Inc. Systems and methods for byte-level or quasi byte-level single instancing
US20100250549A1 (en) * 2009-03-30 2010-09-30 Muller Marcus S Storing a variable number of instances of data objects
US7840514B2 (en) 2006-09-22 2010-11-23 Maxsp Corporation Secure virtual private network utilizing a diagnostics policy and diagnostics engine to establish a secure network connection
US7844686B1 (en) 2006-12-21 2010-11-30 Maxsp Corporation Warm standby appliance
WO2010151813A1 (en) * 2009-06-26 2010-12-29 Simplivt Corporation File system
US20110022566A1 (en) * 2009-06-26 2011-01-27 Simplivt Corporation File system
US8099605B1 (en) 2006-06-05 2012-01-17 InventSec AB Intelligent storage device for backup system
US8131924B1 (en) 2008-03-19 2012-03-06 Netapp, Inc. De-duplication of data stored on tape media
US8175418B1 (en) 2007-10-26 2012-05-08 Maxsp Corporation Method of and system for enhanced data storage
US8307239B1 (en) 2007-10-26 2012-11-06 Maxsp Corporation Disaster recovery appliance
US8364652B2 (en) 2010-09-30 2013-01-29 Commvault Systems, Inc. Content aligned block-based deduplication
CN102915325A (en) * 2012-08-11 2013-02-06 深圳市极限网络科技有限公司 Md5 Hash list-based file decomposing and combining technique
US20130086000A1 (en) * 2011-09-29 2013-04-04 International Business Machines Corporation Managing back up operations for data
US8423821B1 (en) 2006-12-21 2013-04-16 Maxsp Corporation Virtual recovery server
US8484162B2 (en) 2008-06-24 2013-07-09 Commvault Systems, Inc. De-duplication systems and methods for application-specific data
US8572340B2 (en) 2010-09-30 2013-10-29 Commvault Systems, Inc. Systems and methods for retaining and using data block signatures in data protection operations
US8578120B2 (en) 2009-05-22 2013-11-05 Commvault Systems, Inc. Block-level single instancing
US8589323B2 (en) 2005-03-04 2013-11-19 Maxsp Corporation Computer hardware and software diagnostic and report system incorporating an expert system and agents
US8645515B2 (en) 2007-10-26 2014-02-04 Maxsp Corporation Environment manager
US8667273B1 (en) 2006-05-30 2014-03-04 Leif Olov Billstrom Intelligent file encryption and secure backup system
US8898319B2 (en) 2006-05-24 2014-11-25 Maxsp Corporation Applications and services as a bundle
US8930306B1 (en) 2009-07-08 2015-01-06 Commvault Systems, Inc. Synchronized data deduplication
US8935492B2 (en) 2010-09-30 2015-01-13 Commvault Systems, Inc. Archiving data objects using secondary copies
US8954446B2 (en) 2010-12-14 2015-02-10 Comm Vault Systems, Inc. Client-side repository in a networked deduplicated storage system
US9020900B2 (en) 2010-12-14 2015-04-28 Commvault Systems, Inc. Distributed deduplicated storage system
US9020890B2 (en) 2012-03-30 2015-04-28 Commvault Systems, Inc. Smart archiving and data previewing for mobile devices
CN104731792A (en) * 2013-12-19 2015-06-24 中国银联股份有限公司 Method and system for verifying database consistency and method and system for positioning data difference
US9098495B2 (en) 2008-06-24 2015-08-04 Commvault Systems, Inc. Application-aware and remote single instance data management
WO2015153259A1 (en) * 2014-03-31 2015-10-08 Amazon Technologies, Inc. Namespace management in distributed storage systems
US9218375B2 (en) 2012-06-13 2015-12-22 Commvault Systems, Inc. Dedicated client-side signature generator in a networked storage system
US20160042183A1 (en) * 2013-03-28 2016-02-11 Irdeto B.V. Generating identifier
US9357031B2 (en) 2004-06-03 2016-05-31 Microsoft Technology Licensing, Llc Applications as a service
US9575673B2 (en) 2014-10-29 2017-02-21 Commvault Systems, Inc. Accessing a file system using tiered deduplication
US9633033B2 (en) 2013-01-11 2017-04-25 Commvault Systems, Inc. High availability distributed deduplicated storage system
US9633022B2 (en) 2012-12-28 2017-04-25 Commvault Systems, Inc. Backup and restoration for a deduplicated file system
US9633056B2 (en) 2014-03-17 2017-04-25 Commvault Systems, Inc. Maintaining a deduplication database
US9646176B2 (en) 2015-03-24 2017-05-09 TmaxData Co., Ltd. Method for encrypting database
US9772787B2 (en) 2014-03-31 2017-09-26 Amazon Technologies, Inc. File storage using variable stripe sizes
US9779015B1 (en) 2014-03-31 2017-10-03 Amazon Technologies, Inc. Oversubscribed storage extents with on-demand page allocation
US9811522B2 (en) 2013-08-21 2017-11-07 Hewlett Packard Enterprise Development Lp System and method for transforming a source virtual machine without copying of payload data
US9875183B2 (en) 2012-02-24 2018-01-23 Hewlett Packard Enterprise Development Lp Method and apparatus for content derived data placement in memory

Families Citing this family (161)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6704730B2 (en) * 2000-02-18 2004-03-09 Avamar Technologies, Inc. Hash file system and method for use in a commonality factoring system
US7379940B1 (en) * 2000-07-20 2008-05-27 Repliweb, Inc. Focal point compression method and apparatus
US7197513B2 (en) * 2000-12-08 2007-03-27 Aol Llc Distributed image storage architecture
US7779117B2 (en) * 2002-05-31 2010-08-17 Aol Inc. Monitoring digital images
US7047420B2 (en) * 2001-01-17 2006-05-16 Microsoft Corporation Exclusive encryption
WO2002065316A9 (en) * 2001-02-12 2003-09-25 Otg Software Inc System and method of indexing unique electronic mail messages and uses for the same
US7043637B2 (en) * 2001-03-21 2006-05-09 Microsoft Corporation On-disk file format for a serverless distributed file system
US7478243B2 (en) * 2001-03-21 2009-01-13 Microsoft Corporation On-disk file format for serverless distributed file system with signed manifest of file modifications
US7062490B2 (en) * 2001-03-26 2006-06-13 Microsoft Corporation Serverless distributed file system
US6981138B2 (en) 2001-03-26 2005-12-27 Microsoft Corporation Encrypted key cache
US7310644B2 (en) * 2001-06-06 2007-12-18 Microsoft Corporation Locating potentially identical objects across multiple computers
US6988124B2 (en) * 2001-06-06 2006-01-17 Microsoft Corporation Locating potentially identical objects across multiple computers based on stochastic partitioning of workload
US7134041B2 (en) * 2001-09-20 2006-11-07 Evault, Inc. Systems and methods for data backup over a network
US6795903B2 (en) * 2002-01-17 2004-09-21 Thomas Licensing S.A. System and method for searching for duplicate data
JP4154893B2 (en) * 2002-01-23 2008-09-24 株式会社日立製作所 The method for forming the virtual network storage
US7020665B2 (en) * 2002-03-07 2006-03-28 Microsoft Corporation File availability in distributed file storage systems
US20030188275A1 (en) * 2002-03-27 2003-10-02 Meares Lawrence G. System and method of preventing the simulation of a circuit if a change to the circuit topology is detected
US7152243B2 (en) * 2002-06-27 2006-12-19 Microsoft Corporation Providing a secure hardware identifier (HWID) for use in connection with digital rights management (DRM) system
US7120666B2 (en) 2002-10-30 2006-10-10 Riverbed Technology, Inc. Transaction accelerator for client-server communication systems
US8176186B2 (en) 2002-10-30 2012-05-08 Riverbed Technology, Inc. Transaction accelerator for client-server communications systems
US6667700B1 (en) * 2002-10-30 2003-12-23 Nbt Technology, Inc. Content-based segmentation scheme for data compression in storage and transmission including hierarchical segment representation
JP2004171299A (en) * 2002-11-20 2004-06-17 Fujitsu Ltd Network relay storage device and network relay method
US6928526B1 (en) 2002-12-20 2005-08-09 Datadomain, Inc. Efficient data storage system
US7363349B2 (en) 2003-01-31 2008-04-22 Visto Corporation Asynchronous real-time retrieval of data
US20040186859A1 (en) * 2003-03-20 2004-09-23 Sun Microsystems, Inc. File access based on file digests
US8266113B2 (en) * 2003-04-01 2012-09-11 Cybersoft, Inc. Methods, apparatus and articles of manufacture for computer file integrity and baseline maintenance
EP1611527A1 (en) * 2003-04-07 2006-01-04 Synematics Inc. System and method for providing scalable management on commodity routers
US7304994B2 (en) * 2003-04-09 2007-12-04 Nec Laboratories America, Inc. Peer-to-peer system and method with prefix-based distributed hash table
US7451168B1 (en) 2003-06-30 2008-11-11 Data Domain, Inc. Incremental garbage collection of data in a secondary storage
US7424498B1 (en) 2003-06-30 2008-09-09 Data Domain, Inc. Probabilistic summary data structure based encoding for garbage collection
US8280926B2 (en) * 2003-08-05 2012-10-02 Sepaton, Inc. Scalable de-duplication mechanism
US7650416B2 (en) * 2003-08-12 2010-01-19 Riverbed Technology Content delivery for client-server protocols with user affinities using connection end-point proxies
US7577806B2 (en) * 2003-09-23 2009-08-18 Symantec Operating Corporation Systems and methods for time dependent data storage and recovery
US7904428B2 (en) * 2003-09-23 2011-03-08 Symantec Corporation Methods and apparatus for recording write requests directed to a data store
US7725760B2 (en) * 2003-09-23 2010-05-25 Symantec Operating Corporation Data storage system
US7577807B2 (en) * 2003-09-23 2009-08-18 Symantec Operating Corporation Methods and devices for restoring a portion of a data store
US7991748B2 (en) * 2003-09-23 2011-08-02 Symantec Corporation Virtual data store creation and use
US7698345B2 (en) * 2003-10-21 2010-04-13 The Nielsen Company (Us), Llc Methods and apparatus for fusing databases
US7613775B2 (en) * 2003-11-25 2009-11-03 Freescale Semiconductor, Inc. Network message filtering using hashing and pattern matching
US8244677B2 (en) * 2004-01-23 2012-08-14 Elad Baron Focal point compression method and apparatus
US7574418B1 (en) 2004-02-13 2009-08-11 Data Domain, Inc. Method and apparatus for storing composite data streams
JP4521865B2 (en) * 2004-02-27 2010-08-11 株式会社日立製作所 Storage system, attribute setting method of a computer system or storage area
US7272654B1 (en) * 2004-03-04 2007-09-18 Sandbox Networks, Inc. Virtualizing network-attached-storage (NAS) with a compact table that stores lossy hashes of file names and parent handles rather than full names
GB0411053D0 (en) * 2004-05-18 2004-06-23 Ricardo Uk Ltd Data processing
US20060031230A1 (en) * 2004-07-21 2006-02-09 Kumar Sinha M Data storage systems
JP4530763B2 (en) * 2004-08-23 2010-08-25 富士機械製造株式会社 Component data forming method and component data generating device
US7827362B2 (en) * 2004-08-24 2010-11-02 Symantec Corporation Systems, apparatus, and methods for processing I/O requests
US7296008B2 (en) * 2004-08-24 2007-11-13 Symantec Operating Corporation Generation and use of a time map for accessing a prior image of a storage device
US7631120B2 (en) * 2004-08-24 2009-12-08 Symantec Operating Corporation Methods and apparatus for optimally selecting a storage buffer for the storage of data
US7409587B2 (en) * 2004-08-24 2008-08-05 Symantec Operating Corporation Recovering from storage transaction failures using checkpoints
US7287133B2 (en) * 2004-08-24 2007-10-23 Symantec Operating Corporation Systems and methods for providing a modification history for a location within a data store
US7730222B2 (en) * 2004-08-24 2010-06-01 Symantec Operating System Processing storage-related I/O requests using binary tree data structures
US7239581B2 (en) * 2004-08-24 2007-07-03 Symantec Operating Corporation Systems and methods for synchronizing the internal clocks of a plurality of processor modules
US7523098B2 (en) * 2004-09-15 2009-04-21 International Business Machines Corporation Systems and methods for efficient data searching, storage and reduction
US8725705B2 (en) * 2004-09-15 2014-05-13 International Business Machines Corporation Systems and methods for searching of storage data with reduced bandwidth requirements
US7689633B1 (en) * 2004-09-15 2010-03-30 Data Domain, Inc. Network file system-based data storage system
KR100714682B1 (en) * 2004-12-02 2007-05-04 삼성전자주식회사 File system path processing device and method thereof
US7844691B2 (en) * 2004-12-30 2010-11-30 Xstor Systems, Inc. Scalable distributed storage and delivery
US8201021B1 (en) * 2005-01-31 2012-06-12 Symantec Corporation Method and apparatus for efficiently creating backup files with less redundancy
WO2006094365A1 (en) * 2005-03-11 2006-09-14 Rocksoft Limited Method for storing data with reduced redundancy using data clusters
US8356021B2 (en) * 2005-03-11 2013-01-15 Ross Neil Williams Method and apparatus for indexing in a reduced-redundancy storage system
JP5043820B2 (en) * 2005-03-11 2012-10-10 ロックソフト リミテッド Method of performing an index of low-redundancy storage system
JP4975724B2 (en) * 2005-03-11 2012-07-11 ロックソフト リミテッド Method for detecting the presence of a sub-block in the low-redundant memory system
US7814129B2 (en) * 2005-03-11 2010-10-12 Ross Neil Williams Method and apparatus for storing data with reduced redundancy using data clusters
US8051252B2 (en) 2005-03-11 2011-11-01 Ross Neil Williams Method and apparatus for detecting the presence of subblocks in a reduced-redundancy storage system
US20060248194A1 (en) 2005-03-18 2006-11-02 Riverbed Technology, Inc. Connection forwarding
US8364815B2 (en) 2005-03-18 2013-01-29 Riverbed Technology, Inc. Reliability and availability of distributed servers
US7984018B2 (en) * 2005-04-18 2011-07-19 Microsoft Corporation Efficient point-to-multipoint data reconciliation
GB0510878D0 (en) * 2005-05-27 2005-07-06 Qinetiq Ltd Digital evidence bag
US8984636B2 (en) 2005-07-29 2015-03-17 Bit9, Inc. Content extractor and analysis system
US8272058B2 (en) 2005-07-29 2012-09-18 Bit 9, Inc. Centralized timed analysis in a network security system
US7895651B2 (en) 2005-07-29 2011-02-22 Bit 9, Inc. Content tracking in a network security system
CA2618135C (en) 2005-08-09 2014-10-28 Nexsan Technologies Canada Inc. Data archiving system
US7584338B1 (en) * 2005-09-27 2009-09-01 Data Domain, Inc. Replication of deduplicated storage system
US20070083378A1 (en) * 2005-10-11 2007-04-12 Microsoft Corporation Secure application programming interface
EP1952236B1 (en) * 2005-10-14 2017-06-14 Symantec Operating Corporation Technique for timeline compression in a data store
GB2431741B (en) * 2005-10-27 2010-11-03 Hewlett Packard Development Co A method of digitally signing data and a data repository storing digitally signed data
JP2007140887A (en) * 2005-11-18 2007-06-07 Hitachi Ltd Storage system, disk array device, method of presenting volume, and method of verifying data consistency
US20080276171A1 (en) * 2005-11-29 2008-11-06 Itzchak Sabo Filing System
US7457934B2 (en) * 2006-03-22 2008-11-25 Hitachi, Ltd. Method and apparatus for reducing the amount of data in a storage system
US8832045B2 (en) 2006-04-07 2014-09-09 Data Storage Group, Inc. Data compression and storage techniques
EP2013974A4 (en) * 2006-04-07 2009-08-05 Data Storage Group Data compression and storage techniques
US7840540B2 (en) 2006-04-20 2010-11-23 Datascout, Inc. Surrogate hashing
US7814070B1 (en) 2006-04-20 2010-10-12 Datascout, Inc. Surrogate hashing
US20070250810A1 (en) * 2006-04-20 2007-10-25 Tittizer Abigail A Systems and methods for managing data associated with computer code
US9020964B1 (en) 2006-04-20 2015-04-28 Pinehill Technology, Llc Generation of fingerprints for multimedia content based on vectors and histograms
US7801868B1 (en) 2006-04-20 2010-09-21 Datascout, Inc. Surrogate hashing
US8051043B2 (en) 2006-05-05 2011-11-01 Hybir Inc. Group based complete and incremental computer file backup system, process and apparatus
WO2008005629A3 (en) * 2006-05-26 2008-09-25 Nitin Gupta Throttling of predictive acks in an accelerated network communication system
US7769834B2 (en) 2006-05-30 2010-08-03 Riverbed Technology, Inc. System for selecting a proxy pair based on configurations of autodiscovered proxies on a network
US7921077B2 (en) 2006-06-29 2011-04-05 Netapp, Inc. System and method for managing data deduplication of storage systems utilizing persistent consistency point images
US7640262B1 (en) 2006-06-30 2009-12-29 Emc Corporation Positional allocation
US7673099B1 (en) 2006-06-30 2010-03-02 Emc Corporation Affinity caching
US7720892B1 (en) 2006-06-30 2010-05-18 Emc Corporation Bulk updates and tape synchronization
US7930559B1 (en) 2006-06-30 2011-04-19 Emc Corporation Decoupled data stream and access structures
US8082231B1 (en) 2006-09-22 2011-12-20 Emc Corporation Techniques using identifiers and signatures with data operations
US7725704B1 (en) 2006-09-22 2010-05-25 Emc Corporation Techniques for performing a prioritized data restoration operation
US7685171B1 (en) 2006-09-22 2010-03-23 Emc Corporation Techniques for performing a restoration operation using device scanning
US8069321B2 (en) 2006-11-13 2011-11-29 I365 Inc. Secondary pools
DE102006055964A1 (en) * 2006-11-24 2008-05-29 Bdt-Solutions Gmbh Method and apparatus for data security
US8024307B2 (en) * 2006-12-13 2011-09-20 Canon Kabushiki Kaisha Information processing apparatus and information processing method
US7733910B2 (en) * 2006-12-29 2010-06-08 Riverbed Technology, Inc. Data segmentation using shift-varying predicate function fingerprinting
US7921461B1 (en) * 2007-01-16 2011-04-05 Kaspersky Lab, Zao System and method for rootkit detection and cure
US7853750B2 (en) * 2007-01-30 2010-12-14 Netapp, Inc. Method and an apparatus to store data patterns
US8533310B2 (en) * 2007-03-09 2013-09-10 Riverbed Technology, Inc. Method and apparatus for acceleration by prefetching associated objects
US20080281908A1 (en) * 2007-05-08 2008-11-13 Riverbed Technology, Inc. Hybrid segment-oriented file server and wan accelerator
US7949637B1 (en) 2007-06-27 2011-05-24 Emc Corporation Storage management for fine grained tiered storage with thin provisioning
US8775368B1 (en) 2007-06-27 2014-07-08 Emc Corporation Fine grained tiered storage with thin provisioning
US8463000B1 (en) 2007-07-02 2013-06-11 Pinehill Technology, Llc Content identification based on a search of a fingerprint database
US8549022B1 (en) 2007-07-02 2013-10-01 Datascout, Inc. Fingerprint generation of multimedia content based on a trigger point with the multimedia content
US7991206B1 (en) 2007-07-02 2011-08-02 Datascout, Inc. Surrogate heuristic identification
US8156132B1 (en) 2007-07-02 2012-04-10 Pinehill Technology, Llc Systems for comparing image fingerprints
US7774385B1 (en) 2007-07-02 2010-08-10 Datascout, Inc. Techniques for providing a surrogate heuristic identification interface
US8028106B2 (en) * 2007-07-06 2011-09-27 Proster Systems, Inc. Hardware acceleration of commonality factoring with removable media
US8095509B2 (en) * 2007-08-11 2012-01-10 Novell, Inc. Techniques for retaining security restrictions with file versioning
US7822939B1 (en) 2007-09-25 2010-10-26 Emc Corporation Data de-duplication using thin provisioning
US7870105B2 (en) * 2007-11-20 2011-01-11 Hitachi, Ltd. Methods and apparatus for deduplication in storage system
US8209334B1 (en) 2007-12-28 2012-06-26 Don Doerner Method to direct data to a specific one of several repositories
US7949630B1 (en) * 2007-12-31 2011-05-24 Emc Corporation Storage of data addresses with hashes in backup systems
US8825971B1 (en) * 2007-12-31 2014-09-02 Emc Corporation Age-out selection in hash caches
US9766983B2 (en) * 2008-03-05 2017-09-19 Ca, Inc. Proximity and in-memory map based signature searching for duplicate data
US8620877B2 (en) * 2008-04-30 2013-12-31 International Business Machines Corporation Tunable data fingerprinting for optimizing data deduplication
US9063947B2 (en) * 2008-08-18 2015-06-23 Hewlett-Packard Development Company, L.P. Detecting duplicative hierarchical sets of files
US8311985B2 (en) * 2008-09-16 2012-11-13 Quest Software, Inc. Remote backup and restore system and method
WO2010036889A1 (en) 2008-09-25 2010-04-01 Bakbone Software, Inc. Remote backup and restore
WO2010080591A3 (en) * 2008-12-18 2010-09-30 Sumooh Inc. Methods and apparatus for content-aware data partitioning and data de-duplication
US8566549B1 (en) 2008-12-31 2013-10-22 Emc Corporation Synchronizing performance requirements across multiple storage platforms
US8209290B1 (en) * 2009-03-11 2012-06-26 Symantec Corporation Generic granular restore of application data from a volume image backup
US8346736B2 (en) * 2009-03-16 2013-01-01 International Business Machines Corporation Apparatus and method to deduplicate data
US9160611B2 (en) * 2009-04-22 2015-10-13 Webroot Inc. System and method for performing longest common prefix strings searches
US9329791B2 (en) * 2009-06-01 2016-05-03 Mstar Semiconductor, Inc. File system and file system converting method
US20110055471A1 (en) * 2009-08-28 2011-03-03 Jonathan Thatcher Apparatus, system, and method for improved data deduplication
US8924439B2 (en) * 2009-09-01 2014-12-30 Verizon Patent And Licensing Inc. System for and method of partial file hashing
JP5595701B2 (en) * 2009-09-16 2014-09-24 株式会社日立製作所 File management method and storage system
US8620939B2 (en) * 2010-01-25 2013-12-31 Sepaton, Inc. System and method for summarizing data
US20110218973A1 (en) * 2010-03-02 2011-09-08 Renew Data Corp. System and method for creating a de-duplicated data set and preserving metadata for processing the de-duplicated data set
CN102792281B (en) * 2010-03-04 2015-11-25 日本电气株式会社 Storage devices
EP2545472A4 (en) * 2010-03-12 2017-03-15 Copiun Inc Distributed catalog, data store, and indexing
US8452739B2 (en) * 2010-03-16 2013-05-28 Copiun, Inc. Highly scalable and distributed data de-duplication
US8370354B2 (en) * 2010-06-30 2013-02-05 International Business Machines Corporation Acceleration of legacy to service oriented (L2SOA) architecture renovations
US9621405B2 (en) 2010-08-24 2017-04-11 Good Technology Holdings Limited Constant access gateway and de-duplicated data cache server
US9235593B2 (en) * 2010-09-30 2016-01-12 Emc Corporation Transmitting filesystem changes over a network
US8688651B2 (en) 2011-01-25 2014-04-01 Sepaton, Inc. Dynamic deduplication
CN102436402B (en) * 2011-03-29 2014-12-10 奇智软件(北京)有限公司 Module repairing method in software and software equipment
US20120324236A1 (en) * 2011-06-16 2012-12-20 Microsoft Corporation Trusted Snapshot Generation
US20130232124A1 (en) * 2012-03-05 2013-09-05 Blaine D. Gaither Deduplicating a file system
JP2013222230A (en) 2012-04-13 2013-10-28 Hitachi-Lg Data Storage Inc Information processing system
US8984028B2 (en) 2012-05-29 2015-03-17 Recommind, Inc. Systems and methods for storing data and eliminating redundancy
US9405684B1 (en) 2012-09-28 2016-08-02 Emc Corporation System and method for cache management
US9766832B2 (en) 2013-03-15 2017-09-19 Hitachi Data Systems Corporation Systems and methods of locating redundant data using patterns of matching fingerprints
JP2016517243A (en) 2013-04-18 2016-06-09 リソフトデブ,インコーポレイテッド System and method for encrypting data
US9256611B2 (en) 2013-06-06 2016-02-09 Sepaton, Inc. System and method for multi-scale navigation of data
US9405479B1 (en) 2013-08-26 2016-08-02 Western Digital Technologies, Inc. Faster file compression using sliding compression window and backward compound pointers
US9678973B2 (en) 2013-10-15 2017-06-13 Hitachi Data Systems Corporation Multi-node hybrid deduplication
US9639268B2 (en) 2014-08-21 2017-05-02 Datrium, Inc. Distributed data storage system with key-based addressing
US9417955B2 (en) 2014-08-21 2016-08-16 Datrium, Inc. Data reconstruction in distributed data storage system with key-based addressing
US9152500B1 (en) * 2014-09-22 2015-10-06 Storagecraft Technology Corporation Hash collision recovery in a deduplication vault
US9732593B2 (en) 2014-11-05 2017-08-15 Saudi Arabian Oil Company Systems, methods, and computer medium to optimize storage for hydrocarbon reservoir simulation
US9659047B2 (en) * 2014-12-03 2017-05-23 Netapp, Inc. Data deduplication utilizing extent ID database
US9639715B2 (en) 2015-04-27 2017-05-02 Microsoft Technology Licensing, Llc Protecting user identifiable information in the transfer of telemetry data
US9866619B2 (en) 2015-06-12 2018-01-09 International Business Machines Corporation Transmission of hierarchical data files based on content selection

Citations (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6170744B2 (en) *
US5754844A (en) * 1995-12-14 1998-05-19 Sun Microsystems, Inc. Method and system for accessing chunks of data using matching of an access tab and hashing code to generate a suggested storage location
US5915024A (en) * 1996-06-18 1999-06-22 Kabushiki Kaisha Toshiba Electronic signature addition method, electronic signature verification method, and system and computer program product using these methods
US5978791A (en) * 1995-04-11 1999-11-02 Kinetech, Inc. Data processing system using substantially unique identifiers to identify data items, whereby identical data items have the same identifiers
US5990810A (en) * 1995-02-17 1999-11-23 Williams; Ross Neil Method for partitioning a block of data into subblocks and for storing and communcating such subblocks
US6170744B1 (en) * 1998-09-24 2001-01-09 Payformance Corporation Self-authenticating negotiable documents
US6219423B1 (en) * 1995-12-29 2001-04-17 Intel Corporation System and method for digitally signing a digital agreement between remotely located nodes
US6377907B1 (en) * 1999-11-17 2002-04-23 Mci Worldcom, Inc. System and method for collating UNIX performance metrics
US20020194484A1 (en) * 2001-03-21 2002-12-19 Bolosky William J. On-disk file format for serverless distributed file system with signed manifest of file modifications
US20020194209A1 (en) * 2001-03-21 2002-12-19 Bolosky William J. On-disk file format for a serverless distributed file system
US6535894B1 (en) * 2000-06-01 2003-03-18 Sun Microsystems, Inc. Apparatus and method for incremental updating of archive files
US6553494B1 (en) * 1999-07-21 2003-04-22 Sensar, Inc. Method and apparatus for applying and verifying a biometric-based digital signature to an electronic document
US6671358B1 (en) * 2001-04-25 2003-12-30 Universal Identity Technologies, Inc. Method and system for rewarding use of a universal identifier, and/or conducting a financial transaction
US6704730B2 (en) * 2000-02-18 2004-03-09 Avamar Technologies, Inc. Hash file system and method for use in a commonality factoring system
US6826690B1 (en) * 1999-11-08 2004-11-30 International Business Machines Corporation Using device certificates for automated authentication of communicating devices

Family Cites Families (91)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US3668647A (en) 1970-06-12 1972-06-06 Ibm File access system
US4215402A (en) 1978-10-23 1980-07-29 International Business Machines Corporation Hash index table hash generator apparatus
US4404676A (en) * 1981-03-30 1983-09-13 Pioneer Electric Corporation Partitioning method and apparatus using data-dependent boundary-marking code words
US4887235A (en) 1982-12-17 1989-12-12 Symbolics, Inc. Symbolic language data processing system
US4649479A (en) 1985-02-28 1987-03-10 International Business Machines Corp. Device driver and adapter binding technique
US4901223A (en) 1986-04-30 1990-02-13 International Business Machines Corporation Method and apparatus for application software control of echo response
US4761785B1 (en) 1986-06-12 1996-03-12 Ibm Parity spreading to enhance storage access
US4897781A (en) 1987-02-13 1990-01-30 International Business Machines Corporation System and method for using cached data at a local node after re-opening a file at a remote node in a distributed networking environment
US4887204A (en) 1987-02-13 1989-12-12 International Business Machines Corporation System and method for accessing remote files in a distributed networking environment
US5005122A (en) 1987-09-08 1991-04-02 Digital Equipment Corporation Arrangement with cooperating management server node and network service node
US5109515A (en) 1987-09-28 1992-04-28 At&T Bell Laboratories User and application program transparent resource sharing multiple computer interface architecture with kernel process level transfer of user requested services
EP0343820B1 (en) 1988-05-26 1996-12-04 Digital Equipment Corporation Temporary state preservation for a distributed file service
US5146568A (en) 1988-09-06 1992-09-08 Digital Equipment Corporation Remote bootstrapping a node over communication link by initially requesting remote storage access program which emulates local disk to load other programs
US4982324A (en) 1988-12-19 1991-01-01 International Business Machines Corporation Method of and system for using device drivers to couple the communication and data storage of remote computer systems
US5126739A (en) 1989-01-13 1992-06-30 Stac Electronics Data compression apparatus and method
US5016009A (en) 1989-01-13 1991-05-14 Stac, Inc. Data compression apparatus and method
US5089958A (en) 1989-01-23 1992-02-18 Vortex Systems, Inc. Fault tolerant computer backup system
US5018060A (en) 1989-01-26 1991-05-21 Ibm Corporation Allocating data storage space of peripheral data storage devices using implied allocation based on user parameters
US4929946A (en) * 1989-02-09 1990-05-29 Storage Technology Corporation Adaptive data compression apparatus including run length encoding for a tape drive system
JPH0833958B2 (en) 1989-05-30 1996-03-29 沖電気工業株式会社 Customer information processing system
US5239637A (en) 1989-06-30 1993-08-24 Digital Equipment Corporation Digital data management system for maintaining consistency of data in a shadow set
US5454099A (en) 1989-07-25 1995-09-26 International Business Machines Corporation CPU implemented method for backing up modified data sets in non-volatile store for recovery in the event of CPU failure
US5133065A (en) 1989-07-27 1992-07-21 Personal Computer Peripherals Corporation Backup computer program for networks
US5163148A (en) 1989-08-11 1992-11-10 Digital Equipment Corporation File backup system for producing a backup copy of a file which may be updated during backup
US5163131A (en) 1989-09-08 1992-11-10 Auspex Systems, Inc. Parallel i/o network file server architecture
US5276860A (en) 1989-12-19 1994-01-04 Epoch Systems, Inc. Digital data processor with improved backup storage
US5276867A (en) 1989-12-19 1994-01-04 Epoch Systems, Inc. Digital data storage system with improved data migration
US5218695A (en) 1990-02-05 1993-06-08 Epoch Systems, Inc. File server system having high-speed write execution
US5307481A (en) 1990-02-28 1994-04-26 Hitachi, Ltd. Highly reliable online system
US5479654A (en) 1990-04-26 1995-12-26 Squibb Data Systems, Inc. Apparatus and method for reconstructing a file from a difference signature and an original file
DE69126066D1 (en) 1990-06-29 1997-06-19 Oracle Corp Method and apparatus for optimizing the use log lifting
US5317728A (en) 1990-09-07 1994-05-31 International Business Machines Corporation Storage management of a first file system using a second file system containing surrogate files and catalog management information
US5239647A (en) 1990-09-07 1993-08-24 International Business Machines Corporation Data storage hierarchy with shared storage level
US5210866A (en) 1990-09-12 1993-05-11 Storage Technology Corporation Incremental disk backup system for a dynamically mapped data storage subsystem
US5162986A (en) 1990-10-19 1992-11-10 Allen-Bradley Company, Inc. Remote downloading and uploading of motion control program information to and from a motion control I/O module in a programmable controller
US5155835A (en) 1990-11-19 1992-10-13 Storage Technology Corporation Multilevel, hierarchical, dynamically mapped data storage subsystem
US5278838A (en) 1991-06-18 1994-01-11 Ibm Corp. Recovery from errors in a redundant array of disk drives
US5239659A (en) 1991-06-19 1993-08-24 Storage Technology Corporation Phantom duplex copy group apparatus for a disk drive array data storge subsystem
US5347653A (en) 1991-06-28 1994-09-13 Digital Equipment Corporation System for reconstructing prior versions of indexes using records indicating changes between successive versions of the indexes
US5566330A (en) * 1991-08-20 1996-10-15 Powersoft Corporation Method for forming a reusable and modifiable database interface object
US5305389A (en) 1991-08-30 1994-04-19 Digital Equipment Corporation Predictive cache system
US5140321A (en) 1991-09-04 1992-08-18 Prime Computer, Inc. Data compression/decompression method and apparatus
US5325505A (en) 1991-09-04 1994-06-28 Storage Technology Corporation Intelligent storage manager for data storage apparatus having simulation capability
US5367698A (en) 1991-10-31 1994-11-22 Epoch Systems, Inc. Network file migration system
EP0541281B1 (en) 1991-11-04 1998-04-29 Commvault Systems, Inc. Incremental-computer-file backup using signatures
US5802264A (en) 1991-11-15 1998-09-01 Fujitsu Limited Background data reconstruction in a storage device array system
US5369758A (en) 1991-11-15 1994-11-29 Fujitsu Limited Checking for proper locations of storage devices in a storage array
US5452454A (en) 1991-12-10 1995-09-19 Digital Equipment Corporation Generic remote boot for networked workstations by creating local bootable code image
US5367637A (en) 1992-03-26 1994-11-22 International Business Machines Corporation Self-tuning virtual storage management for dedicated real-time computer system
US5263154A (en) 1992-04-20 1993-11-16 International Business Machines Corporation Method and system for incremental time zero backup copying of data
US5448718A (en) 1992-04-20 1995-09-05 International Business Machines Corporation Method and system for time zero backup session security
US5586322A (en) 1992-06-11 1996-12-17 Beck; Robert E. Workgroup organized network manager with workstation comparison system
US5403639A (en) 1992-09-02 1995-04-04 Storage Technology Corporation File server having snapshot application data groups
US5406279A (en) 1992-09-02 1995-04-11 Cirrus Logic, Inc. General purpose, hash-based technique for single-pass lossless data compression
US5497483A (en) 1992-09-23 1996-03-05 International Business Machines Corporation Method and system for track transfer control during concurrent copy operations in a data processing storage subsystem
US5530855A (en) 1992-10-13 1996-06-25 International Business Machines Corporation Replicating a database by the sequential application of hierarchically sorted log records
US5404508A (en) 1992-12-03 1995-04-04 Unisys Corporation Data base backup and recovery system and method
US5487160A (en) 1992-12-04 1996-01-23 At&T Global Information Solutions Company Concurrent image backup for disk storage system
US5404527A (en) 1992-12-31 1995-04-04 Unisys Corporation System and method for remote program load
US5544320A (en) 1993-01-08 1996-08-06 Konrad; Allan M. Remote information service access system based on a client-server-service model
US5659747A (en) 1993-04-22 1997-08-19 Microsoft Corporation Multiple level undo/redo mechanism
DE69413977T2 (en) 1993-07-01 1999-03-18 Legent Corp Arrangement and method for distributed data management in networked computer systems
US5452440A (en) 1993-07-16 1995-09-19 Zitel Corporation Method and structure for evaluating and enhancing the performance of cache memory systems
US5521597A (en) * 1993-08-02 1996-05-28 Mircosoft Corporation Data compression for network transport
US5515502A (en) 1993-09-30 1996-05-07 Sybase, Inc. Data backup system with methods for stripe affinity backup to multiple archive devices
US5771354A (en) 1993-11-04 1998-06-23 Crawford; Christopher M. Internet online backup system provides remote storage for customers using IDs and passwords which were interactively established when signing up for backup services
JPH07234921A (en) * 1993-12-28 1995-09-05 Nikon Corp Loading/unloading mechanism for recording medium
US5835953A (en) 1994-10-13 1998-11-10 Vinca Corporation Backup system that takes a snapshot of the locations in a mass storage device that has been identified for updating prior to updating
US5574906A (en) 1994-10-24 1996-11-12 International Business Machines Corporation System and method for reducing storage requirement in backup subsystems utilizing segmented compression and differencing
US5659743A (en) 1994-12-05 1997-08-19 Legent Corporation Method and apparatus for a pattern based spaced management system
US5513314A (en) 1995-01-27 1996-04-30 Auspex Systems, Inc. Fault tolerant NFS server system and mirroring protocol
US5604862A (en) 1995-03-14 1997-02-18 Network Integrity, Inc. Continuously-snapshotted protection of computer files
US5802297A (en) 1995-07-03 1998-09-01 Sun Microsystems, Inc. Client-server computer system and method utilizing a local client disk drive as a data cache
JPH0937788A (en) * 1995-07-31 1997-02-10 Nitto Chem Ind Co Ltd New nitrilase gane
US5742811A (en) * 1995-10-10 1998-04-21 International Business Machines Corporation Method and system for mining generalized sequential patterns in a large database
US5778395A (en) 1995-10-23 1998-07-07 Stac, Inc. System for backing up files from disk volumes on multiple nodes of a computer network
US5933104A (en) * 1995-11-22 1999-08-03 Microsoft Corporation Method and system for compression and decompression using variable-sized offset and length fields
US5765173A (en) 1996-01-11 1998-06-09 Connected Corporation High performance backup via selective file saving which can perform incremental backups and exclude files and uses a changed block signature list
DE19602848A1 (en) * 1996-01-26 1997-07-31 Basf Ag A process for the preparation of pigments
US5831558A (en) 1996-06-17 1998-11-03 Digital Equipment Corporation Method of compressing and decompressing data in a computer system by encoding data using a data dictionary
US5850565A (en) 1996-08-26 1998-12-15 Novell, Inc. Data compression method and apparatus
US5794254A (en) 1996-12-03 1998-08-11 Fairbanks Systems Group Incremental computer file backup using a two-step comparison of first two characters in the block and a signature with pre-stored character and signature sets
JPH10171350A (en) * 1996-12-10 1998-06-26 Hitachi Ltd Method and device for generating hash value
US6044220A (en) * 1997-02-25 2000-03-28 Motorola, Inc. Method and apparatus for operating a data processor to execute software written using a foreign instruction set
US6016553A (en) 1997-09-05 2000-01-18 Wild File, Inc. Method, software and apparatus for saving, using and recovering data
JP3730385B2 (en) * 1997-12-05 2006-01-05 東芝テック株式会社 De - data compression apparatus
US6029168A (en) 1998-01-23 2000-02-22 Tricord Systems, Inc. Decentralized file mapping in a striped network file system in a distributed computing environment
US6122754A (en) 1998-05-22 2000-09-19 International Business Machines Corporation Method and system for data recovery using a distributed and scalable data structure
US6320520B1 (en) 1998-09-23 2001-11-20 Digital Fountain Information additive group code generator and decoder for communications systems
US6307487B1 (en) 1998-09-23 2001-10-23 Digital Fountain, Inc. Information additive code generator and decoder for communication systems
JP3956638B2 (en) * 2000-11-21 2007-08-08 日本精工株式会社 Rolling bearing grease lubrication

Patent Citations (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6170744B2 (en) *
US5990810A (en) * 1995-02-17 1999-11-23 Williams; Ross Neil Method for partitioning a block of data into subblocks and for storing and communcating such subblocks
US5978791A (en) * 1995-04-11 1999-11-02 Kinetech, Inc. Data processing system using substantially unique identifiers to identify data items, whereby identical data items have the same identifiers
US5754844A (en) * 1995-12-14 1998-05-19 Sun Microsystems, Inc. Method and system for accessing chunks of data using matching of an access tab and hashing code to generate a suggested storage location
US6219423B1 (en) * 1995-12-29 2001-04-17 Intel Corporation System and method for digitally signing a digital agreement between remotely located nodes
US5915024A (en) * 1996-06-18 1999-06-22 Kabushiki Kaisha Toshiba Electronic signature addition method, electronic signature verification method, and system and computer program product using these methods
US6170744B1 (en) * 1998-09-24 2001-01-09 Payformance Corporation Self-authenticating negotiable documents
US6553494B1 (en) * 1999-07-21 2003-04-22 Sensar, Inc. Method and apparatus for applying and verifying a biometric-based digital signature to an electronic document
US6826690B1 (en) * 1999-11-08 2004-11-30 International Business Machines Corporation Using device certificates for automated authentication of communicating devices
US6377907B1 (en) * 1999-11-17 2002-04-23 Mci Worldcom, Inc. System and method for collating UNIX performance metrics
US6704730B2 (en) * 2000-02-18 2004-03-09 Avamar Technologies, Inc. Hash file system and method for use in a commonality factoring system
US6535894B1 (en) * 2000-06-01 2003-03-18 Sun Microsystems, Inc. Apparatus and method for incremental updating of archive files
US20020194209A1 (en) * 2001-03-21 2002-12-19 Bolosky William J. On-disk file format for a serverless distributed file system
US20020194484A1 (en) * 2001-03-21 2002-12-19 Bolosky William J. On-disk file format for serverless distributed file system with signed manifest of file modifications
US6671358B1 (en) * 2001-04-25 2003-12-30 Universal Identity Technologies, Inc. Method and system for rewarding use of a universal identifier, and/or conducting a financial transaction

Cited By (143)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20010034795A1 (en) * 2000-02-18 2001-10-25 Moulton Gregory Hagan System and method for intelligent, globally distributed network storage
US20050120137A1 (en) * 2000-02-18 2005-06-02 Moulton Gregory H. System and method for intelligent, globally distributed network storage
US20040225655A1 (en) * 2000-11-06 2004-11-11 Moulton Gregory Hagan System and method for unorchestrated determination of data sequences using sticky factoring to determine breakpoints in digital sequences
US7272602B2 (en) * 2000-11-06 2007-09-18 Emc Corporation System and method for unorchestrated determination of data sequences using sticky byte factoring to determine breakpoints in digital sequences
US20030182291A1 (en) * 2002-03-20 2003-09-25 Sreenath Kurupati Method and data structure for a low memory overhead database
US7467151B2 (en) 2002-03-20 2008-12-16 Intel Corporation Method and data structure for a low memory overhead database
US20060122989A1 (en) * 2002-03-20 2006-06-08 Sreenath Kurupati Method and data structure for a low memory overhead database
US7058642B2 (en) * 2002-03-20 2006-06-06 Intel Corporation Method and data structure for a low memory overhead database
US7386616B1 (en) * 2003-05-09 2008-06-10 Google Inc. System and method for providing load balanced processing
US7143251B1 (en) * 2003-06-30 2006-11-28 Data Domain, Inc. Data storage using identifiers
US20060047716A1 (en) * 2004-06-03 2006-03-02 Keith Robert O Jr Transaction based virtual file system optimized for high-latency network connections
US8812613B2 (en) 2004-06-03 2014-08-19 Maxsp Corporation Virtual application manager
US20060031529A1 (en) * 2004-06-03 2006-02-09 Keith Robert O Jr Virtual application manager
US9357031B2 (en) 2004-06-03 2016-05-31 Microsoft Technology Licensing, Llc Applications as a service
US9569194B2 (en) 2004-06-03 2017-02-14 Microsoft Technology Licensing, Llc Virtual application manager
US7908339B2 (en) 2004-06-03 2011-03-15 Maxsp Corporation Transaction based virtual file system optimized for high-latency network connections
US7664834B2 (en) 2004-07-09 2010-02-16 Maxsp Corporation Distributed operating system management
US20060047946A1 (en) * 2004-07-09 2006-03-02 Keith Robert O Jr Distributed operating system management
US7624086B2 (en) 2005-03-04 2009-11-24 Maxsp Corporation Pre-install compliance system
US20060224545A1 (en) * 2005-03-04 2006-10-05 Keith Robert O Jr Computer hardware and software diagnostic and report system
US20060224544A1 (en) * 2005-03-04 2006-10-05 Keith Robert O Jr Pre-install compliance system
US7512584B2 (en) 2005-03-04 2009-03-31 Maxsp Corporation Computer hardware and software diagnostic and report system
US20070233633A1 (en) * 2005-03-04 2007-10-04 Keith Robert O Jr Computer hardware and software diagnostic and report system
US8589323B2 (en) 2005-03-04 2013-11-19 Maxsp Corporation Computer hardware and software diagnostic and report system incorporating an expert system and agents
US8234238B2 (en) 2005-03-04 2012-07-31 Maxsp Corporation Computer hardware and software diagnostic and report system
US9160735B2 (en) 2006-05-24 2015-10-13 Microsoft Technology Licensing, Llc System for and method of securing a network utilizing credentials
US20070274315A1 (en) * 2006-05-24 2007-11-29 Keith Robert O System for and method of securing a network utilizing credentials
US9584480B2 (en) 2006-05-24 2017-02-28 Microsoft Technology Licensing, Llc System for and method of securing a network utilizing credentials
US8898319B2 (en) 2006-05-24 2014-11-25 Maxsp Corporation Applications and services as a bundle
US8811396B2 (en) 2006-05-24 2014-08-19 Maxsp Corporation System for and method of securing a network utilizing credentials
US8667273B1 (en) 2006-05-30 2014-03-04 Leif Olov Billstrom Intelligent file encryption and secure backup system
US8099605B1 (en) 2006-06-05 2012-01-17 InventSec AB Intelligent storage device for backup system
US7840514B2 (en) 2006-09-22 2010-11-23 Maxsp Corporation Secure virtual private network utilizing a diagnostics policy and diagnostics engine to establish a secure network connection
US9317506B2 (en) 2006-09-22 2016-04-19 Microsoft Technology Licensing, Llc Accelerated data transfer using common prior data segments
US20110047118A1 (en) * 2006-09-22 2011-02-24 Maxsp Corporation Secure virtual private network utilizing a diagnostics policy and diagnostics engine to establish a secure network connection
US8099378B2 (en) 2006-09-22 2012-01-17 Maxsp Corporation Secure virtual private network utilizing a diagnostics policy and diagnostics engine to establish a secure network connection
US20080077630A1 (en) * 2006-09-22 2008-03-27 Keith Robert O Accelerated data transfer using common prior data segments
US20080077724A1 (en) * 2006-09-22 2008-03-27 Parthasarathy Sarangam Interrupt coalescing control scheme
US8909881B2 (en) 2006-11-28 2014-12-09 Commvault Systems, Inc. Systems and methods for creating copies of data, such as archive copies
US8392677B2 (en) 2006-12-04 2013-03-05 Commvault Systems, Inc. Systems and methods for creating copies of data, such as archive copies
US8140786B2 (en) * 2006-12-04 2012-03-20 Commvault Systems, Inc. Systems and methods for creating copies of data, such as archive copies
US20080229037A1 (en) * 2006-12-04 2008-09-18 Alan Bunte Systems and methods for creating copies of data, such as archive copies
US7844686B1 (en) 2006-12-21 2010-11-30 Maxsp Corporation Warm standby appliance
US9645900B2 (en) 2006-12-21 2017-05-09 Microsoft Technology Licensing, Llc Warm standby appliance
US8745171B1 (en) 2006-12-21 2014-06-03 Maxsp Corporation Warm standby appliance
US8423821B1 (en) 2006-12-21 2013-04-16 Maxsp Corporation Virtual recovery server
US8285683B2 (en) * 2006-12-22 2012-10-09 Commvault Systems, Inc. System and method for storing redundant information
US7953706B2 (en) 2006-12-22 2011-05-31 Commvault Systems, Inc. System and method for storing redundant information
US8037028B2 (en) 2006-12-22 2011-10-11 Commvault Systems, Inc. System and method for storing redundant information
US20080243879A1 (en) * 2006-12-22 2008-10-02 Parag Gokhale System and method for storing redundant information
US20130006946A1 (en) * 2006-12-22 2013-01-03 Commvault Systems, Inc. System and method for storing redundant information
US8712969B2 (en) * 2006-12-22 2014-04-29 Commvault Systems, Inc. System and method for storing redundant information
US20080243957A1 (en) * 2006-12-22 2008-10-02 Anand Prahlad System and method for storing redundant information
US7840537B2 (en) 2006-12-22 2010-11-23 Commvault Systems, Inc. System and method for storing redundant information
US20080243958A1 (en) * 2006-12-22 2008-10-02 Anand Prahlad System and method for storing redundant information
WO2008147516A1 (en) * 2007-05-22 2008-12-04 Network Appliance, Inc. System and method for on-the-fly elimination of redundant date
US8315984B2 (en) 2007-05-22 2012-11-20 Netapp, Inc. System and method for on-the-fly elimination of redundant data
US20090007261A1 (en) * 2007-06-29 2009-01-01 Smith Mark A Receiving data in a data store in a server computer system
US8615798B2 (en) 2007-06-29 2013-12-24 International Business Machines Corporation Optimizing a data deduplication system using client authentication information
US20090051705A1 (en) * 2007-08-20 2009-02-26 Shih-Kuang Tsai Method for automatically memorizing picture orientation
US8175418B1 (en) 2007-10-26 2012-05-08 Maxsp Corporation Method of and system for enhanced data storage
US9448858B2 (en) 2007-10-26 2016-09-20 Microsoft Technology Licensing, Llc Environment manager
US8307239B1 (en) 2007-10-26 2012-11-06 Maxsp Corporation Disaster recovery appliance
US8422833B2 (en) 2007-10-26 2013-04-16 Maxsp Corporation Method of and system for enhanced data storage
US9092374B2 (en) 2007-10-26 2015-07-28 Maxsp Corporation Method of and system for enhanced data storage
US8645515B2 (en) 2007-10-26 2014-02-04 Maxsp Corporation Environment manager
US7814074B2 (en) * 2008-03-14 2010-10-12 International Business Machines Corporation Method and system for assuring integrity of deduplicated data
US8280859B2 (en) 2008-03-14 2012-10-02 International Business Machines Corporation Method and system for assuring integrity of deduplicated data
US20090234892A1 (en) * 2008-03-14 2009-09-17 International Business Machines Corporation Method and system for assuring integrity of deduplicated data
US20100299311A1 (en) * 2008-03-14 2010-11-25 International Business Machines Corporation Method and system for assuring integrity of deduplicated data
US8131924B1 (en) 2008-03-19 2012-03-06 Netapp, Inc. De-duplication of data stored on tape media
US8515909B2 (en) 2008-04-29 2013-08-20 International Business Machines Corporation Enhanced method and system for assuring integrity of deduplicated data
US20090271454A1 (en) * 2008-04-29 2009-10-29 International Business Machines Corporation Enhanced method and system for assuring integrity of deduplicated data
US8219524B2 (en) 2008-06-24 2012-07-10 Commvault Systems, Inc. Application-aware and remote single instance data management
US8484162B2 (en) 2008-06-24 2013-07-09 Commvault Systems, Inc. De-duplication systems and methods for application-specific data
US9405763B2 (en) 2008-06-24 2016-08-02 Commvault Systems, Inc. De-duplication systems and methods for application-specific data
US20090319585A1 (en) * 2008-06-24 2009-12-24 Parag Gokhale Application-aware and remote single instance data management
US9098495B2 (en) 2008-06-24 2015-08-04 Commvault Systems, Inc. Application-aware and remote single instance data management
US20100005259A1 (en) * 2008-07-03 2010-01-07 Anand Prahlad Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices
US8838923B2 (en) 2008-07-03 2014-09-16 Commvault Systems, Inc. Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices
US8380957B2 (en) 2008-07-03 2013-02-19 Commvault Systems, Inc. Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices
US8612707B2 (en) 2008-07-03 2013-12-17 Commvault Systems, Inc. Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices
US8166263B2 (en) 2008-07-03 2012-04-24 Commvault Systems, Inc. Continuous data protection over intermittent connections, such as continuous data backup for laptops or wireless devices
US8285687B2 (en) 2008-08-27 2012-10-09 Netapp, Inc. System and method for file system level compression using compression group descriptors
US20100058002A1 (en) * 2008-08-27 2010-03-04 Netapp, Inc. System and method for file system level compression using compression group descriptors
US8898117B2 (en) 2008-08-27 2014-11-25 Netapp, Inc. System and method for file system level compression using compression group descriptors
US20100082672A1 (en) * 2008-09-26 2010-04-01 Rajiv Kottomtharayil Systems and methods for managing single instancing data
US9015181B2 (en) 2008-09-26 2015-04-21 Commvault Systems, Inc. Systems and methods for managing single instancing data
US20100169287A1 (en) * 2008-11-26 2010-07-01 Commvault Systems, Inc. Systems and methods for byte-level or quasi byte-level single instancing
US8412677B2 (en) 2008-11-26 2013-04-02 Commvault Systems, Inc. Systems and methods for byte-level or quasi byte-level single instancing
US9158787B2 (en) 2008-11-26 2015-10-13 Commvault Systems, Inc Systems and methods for byte-level or quasi byte-level single instancing
US8725687B2 (en) 2008-11-26 2014-05-13 Commvault Systems, Inc. Systems and methods for byte-level or quasi byte-level single instancing
US8401996B2 (en) 2009-03-30 2013-03-19 Commvault Systems, Inc. Storing a variable number of instances of data objects
US9773025B2 (en) 2009-03-30 2017-09-26 Commvault Systems, Inc. Storing a variable number of instances of data objects
US20100250549A1 (en) * 2009-03-30 2010-09-30 Muller Marcus S Storing a variable number of instances of data objects
US9058117B2 (en) 2009-05-22 2015-06-16 Commvault Systems, Inc. Block-level single instancing
US8578120B2 (en) 2009-05-22 2013-11-05 Commvault Systems, Inc. Block-level single instancing
US8478799B2 (en) 2009-06-26 2013-07-02 Simplivity Corporation Namespace file system accessing an object store
WO2010151813A1 (en) * 2009-06-26 2010-12-29 Simplivt Corporation File system
US20100332846A1 (en) * 2009-06-26 2010-12-30 Simplivt Corporation Scalable indexing
US20110022566A1 (en) * 2009-06-26 2011-01-27 Simplivt Corporation File system
CN102483755A (en) * 2009-06-26 2012-05-30 森普利维蒂公司 File system
US9367551B2 (en) 2009-06-26 2016-06-14 Simplivity Corporation File system accessing an object store
US8930306B1 (en) 2009-07-08 2015-01-06 Commvault Systems, Inc. Synchronized data deduplication
US9262275B2 (en) 2010-09-30 2016-02-16 Commvault Systems, Inc. Archiving data objects using secondary copies
US8364652B2 (en) 2010-09-30 2013-01-29 Commvault Systems, Inc. Content aligned block-based deduplication
US9110602B2 (en) 2010-09-30 2015-08-18 Commvault Systems, Inc. Content aligned block-based deduplication
US8935492B2 (en) 2010-09-30 2015-01-13 Commvault Systems, Inc. Archiving data objects using secondary copies
US9639563B2 (en) 2010-09-30 2017-05-02 Commvault Systems, Inc. Archiving data objects using secondary copies
US9639289B2 (en) 2010-09-30 2017-05-02 Commvault Systems, Inc. Systems and methods for retaining and using data block signatures in data protection operations
US9619480B2 (en) 2010-09-30 2017-04-11 Commvault Systems, Inc. Content aligned block-based deduplication
US8572340B2 (en) 2010-09-30 2013-10-29 Commvault Systems, Inc. Systems and methods for retaining and using data block signatures in data protection operations
US8578109B2 (en) 2010-09-30 2013-11-05 Commvault Systems, Inc. Systems and methods for retaining and using data block signatures in data protection operations
US8577851B2 (en) 2010-09-30 2013-11-05 Commvault Systems, Inc. Content aligned block-based deduplication
US9239687B2 (en) 2010-09-30 2016-01-19 Commvault Systems, Inc. Systems and methods for retaining and using data block signatures in data protection operations
US9116850B2 (en) 2010-12-14 2015-08-25 Commvault Systems, Inc. Client-side repository in a networked deduplicated storage system
US9104623B2 (en) 2010-12-14 2015-08-11 Commvault Systems, Inc. Client-side repository in a networked deduplicated storage system
US9020900B2 (en) 2010-12-14 2015-04-28 Commvault Systems, Inc. Distributed deduplicated storage system
US8954446B2 (en) 2010-12-14 2015-02-10 Comm Vault Systems, Inc. Client-side repository in a networked deduplicated storage system
US9092377B2 (en) * 2011-09-29 2015-07-28 International Business Machines Corporation Managing back up operations for data
US20130086000A1 (en) * 2011-09-29 2013-04-04 International Business Machines Corporation Managing back up operations for data
US9811426B2 (en) 2011-09-29 2017-11-07 International Business Machines Corporation Managing back up operations for data
US9875183B2 (en) 2012-02-24 2018-01-23 Hewlett Packard Enterprise Development Lp Method and apparatus for content derived data placement in memory
US9020890B2 (en) 2012-03-30 2015-04-28 Commvault Systems, Inc. Smart archiving and data previewing for mobile devices
US9218376B2 (en) 2012-06-13 2015-12-22 Commvault Systems, Inc. Intelligent data sourcing in a networked storage system
US9218374B2 (en) 2012-06-13 2015-12-22 Commvault Systems, Inc. Collaborative restore in a networked storage system
US9218375B2 (en) 2012-06-13 2015-12-22 Commvault Systems, Inc. Dedicated client-side signature generator in a networked storage system
US9251186B2 (en) 2012-06-13 2016-02-02 Commvault Systems, Inc. Backup using a client-side signature repository in a networked storage system
US9858156B2 (en) 2012-06-13 2018-01-02 Commvault Systems, Inc. Dedicated client-side signature generator in a networked storage system
CN102915325A (en) * 2012-08-11 2013-02-06 深圳市极限网络科技有限公司 Md5 Hash list-based file decomposing and combining technique
US9633022B2 (en) 2012-12-28 2017-04-25 Commvault Systems, Inc. Backup and restoration for a deduplicated file system
US9633033B2 (en) 2013-01-11 2017-04-25 Commvault Systems, Inc. High availability distributed deduplicated storage system
US9665591B2 (en) 2013-01-11 2017-05-30 Commvault Systems, Inc. High availability distributed deduplicated storage system
US20160042183A1 (en) * 2013-03-28 2016-02-11 Irdeto B.V. Generating identifier
US9811522B2 (en) 2013-08-21 2017-11-07 Hewlett Packard Enterprise Development Lp System and method for transforming a source virtual machine without copying of payload data
CN104731792A (en) * 2013-12-19 2015-06-24 中国银联股份有限公司 Method and system for verifying database consistency and method and system for positioning data difference
US9633056B2 (en) 2014-03-17 2017-04-25 Commvault Systems, Inc. Maintaining a deduplication database
US9772787B2 (en) 2014-03-31 2017-09-26 Amazon Technologies, Inc. File storage using variable stripe sizes
US9779015B1 (en) 2014-03-31 2017-10-03 Amazon Technologies, Inc. Oversubscribed storage extents with on-demand page allocation
WO2015153259A1 (en) * 2014-03-31 2015-10-08 Amazon Technologies, Inc. Namespace management in distributed storage systems
US9495478B2 (en) 2014-03-31 2016-11-15 Amazon Technologies, Inc. Namespace management in distributed storage systems
US9575673B2 (en) 2014-10-29 2017-02-21 Commvault Systems, Inc. Accessing a file system using tiered deduplication
US9646176B2 (en) 2015-03-24 2017-05-09 TmaxData Co., Ltd. Method for encrypting database

Also Published As

Publication number Publication date Type
US6704730B2 (en) 2004-03-09 grant
US20010037323A1 (en) 2001-11-01 application

Similar Documents

Publication Publication Date Title
Ghemawat et al. The Google file system
Honicky et al. Replication under scalable hashing: A family of algorithms for scalable decentralized data distribution
US7209973B2 (en) Distributed network data storage system and method
US7797279B1 (en) Merging of incremental data streams with prior backed-up data
US7424514B2 (en) Peer-to-peer method and system for performing and managing backups in a network of nodes
US6615225B1 (en) System and method for relating files in a distributed data storage environment
US6374266B1 (en) Method and apparatus for storing information in a data processing system
US20070100913A1 (en) Method and system for data backup
US20030028587A1 (en) System and method for accessing and storing data in a common network architecture
US8219524B2 (en) Application-aware and remote single instance data management
US20070094312A1 (en) Method for managing real-time data history of a file system
US7600125B1 (en) Hash-based data block processing with intermittently-connected systems
US7475432B2 (en) Content addressable information encapsulation, representation, and transfer
US7257257B2 (en) Method and apparatus for differential, bandwidth-efficient and storage-efficient backups
US8412848B2 (en) Method and apparatus for content-aware and adaptive deduplication
US8074289B1 (en) Access to content addressable data over a network
US20090319534A1 (en) Application-aware and remote single instance data management
You et al. Deep Store: An archival storage system architecture
US20100106691A1 (en) Remote backup and restore
US7814149B1 (en) Client side data deduplication
US20040205152A1 (en) File replication method for distributed file systems
US7447839B2 (en) System for a distributed column chunk data store
US20060064416A1 (en) Method and system for data reduction
US7873599B2 (en) Backup control apparatus and method eliminating duplication of information resources
Hong et al. Duplicate Data Elimination in a SAN File System.