JP4846156B2 - Hash file system and method for use in a commonality factoring system - Google Patents

Description

[0001]
[Copyright notice / permission]
Part of the disclosure of this patent document may include material that is subject to copyright protection. The copyright owner has no objection to anyone who reproduces a patent document with a patent disclosure as described in the US Patent and Trademark Office patent file or record. Also ensure all copyright rights. The following notice applies to the following description, including software, data and, where applicable, drawings. Copyright 2000, Undoo Technologies
[0002]
BACKGROUND OF THE INVENTION
The present invention relates generally to the field of hash file systems and commonality factoring systems. More particularly, the present invention relates to a system and method for determining correspondence between electronic files in a distributed computer data environment and specific applications thereto.
[0003]
Economic, political and social power is increasingly managed by data. Transactions and richness are represented by data. Political power is analyzed and corrected based on data. Human interactions and relationships are defined by data exchange. Therefore, it is expected that efficient distribution, storage and management of data will play an increasingly essential role in human society.
[0004]
The amount of data in the form of computer programs, databases, files, etc. that must be managed is increasing rapidly. As computer processing power increases, operating systems and application software become larger. Furthermore, the desire to access larger data sets such as multimedia files and large databases increases the amount of data managed. It is essential that this growing data load be moved between computing devices and stored in an accessible manner. Data index growth rates are expected to overtake improvements in communication bandwidth and storage capacity, making data management using conventional methods more urgent.
[0005]
Many factors need to be balanced and often need to be compromised in conventional data storage systems. The amount of data is so great that there is no constant pressure on reducing the cost per stored bit. In addition, the data management system needs to be scalable not only for current needs but also foreseeing future needs.
[0006]
The storage system is desirably incrementally scalable so that users can purchase only the capacity they need at any particular time. As data users become increasingly unacceptable for data loss, damage and unavailability, high reliability and high availability are also considered.
[0007]
Unfortunately, traditional data management architectures must compromise on these factors, so no architecture can provide a cost effective, reliable, highly available, scalable solution.
[0008]
A conventional RAID (redundant array consisting of a plurality of independent disks) system is a method of storing the same data (and thus redundantly) in different positions of a multiple storage device such as a hard disk. By putting data on multiple disks, input / output (“I / O”) operations can overlap in a balanced manner, improving performance. Redundant data storage also increases fault tolerance because the use of multiple disks increases the mean time between failures ("MTBF"). RAID systems rely on hardware or software controllers to hide the complexity of actual data management, so the RAID system appears to the operating system as a single logical hard disk. However, RAID systems are difficult to scale due to the physical limitations of cabling and controllers. In addition, the availability of a RAID system is highly dependent on the functionality of the controller itself, so if the controller fails, the data stored behind the controller becomes unavailable. In addition, RAID systems tend to require specialized hardware rather than common hardware, and thus tend to be an expensive solution.
[0009]
NAS (Network Attached Storage) refers to hard disk storage that is not connected to an application server but is set by its own network address. The file request is mapped to the NAS file server. The NAS may provide transparent I / O operations using hardware or software based on RAID. The NAS may also automate data mirroring for one or more other NAS devices to further improve fault tolerance. Since NAS devices can be added to the network, they allow for a standardization of the total capacity of storage available to the network. However, NAS devices are limited by the capabilities of conventional RAID controllers in RAID applications. NAS systems also make mirroring and parity across nodes impossible, and are therefore a limited solution.
[0010]
In addition to data storage issues, data transport is rapidly evolving with improvements in wide area networks (“WANs”) and internetworking technologies. The Internet, for example, has created a globally networked environment with access almost everywhere. Despite rapid network infrastructure improvements, it is expected that the rate of increase in the amount of data that requires transport will overtake improvements in effective bandwidth.
[0011]
Ideally, traditional data management methods are not consistent with hardware devices and infrastructure developed for the manipulation and movement of data. For example, a computer is a general-purpose machine that is easily programmed as a feature and that performs various virtually unlimited functions. However, computers are loaded with a large percentage of fixed and slowly changing data sets that have limited general-purpose properties to meet the special purpose of the machine. Advances in computing speed, peripheral device performance and data storage capacity are most dramatic in commercial computers. However, many data storage solutions are not extended by the underlying storage controller, but rather are limited so that these advances cannot be exploited. Similarly, the Internet has evolved as a fault-tolerant multipath interconnection network. However, network resources are conventionally implemented in certain network nodes where a node failure makes the resource unavailable despite the fault tolerance of the network to which the node is connected. The need continues to exist for high availability, high reliability, and highly scalable data storage solutions.
[0012]
SUMMARY OF THE INVENTION
Disclosed herein is a computer file system system and method organized based on a hash, and / or a string of digits of specific, different, or varying lengths, from which data blocks (or data It is possible to remove or screen redundant copies of (part of the block). Also, a computer file system system and method that may be generated by a checksum generation program, engine or algorithm such as industry standard message digest 4 (“MD4”), MD5, secure hash algorithm (“SHA”) or SHA-1 algorithm Is also disclosed in the present application. In addition, the hash is of indeterminate size based on any industry standard technology that generates pseudo-random values from checksum programs, engines, algorithms, or non-linear stochastic numerical algorithms or other data / numeric sequence input text. Also disclosed herein are computer file system systems and methods that may be generated by other means for generating a stochastic unique hash value of a data block.
[0013]
The system and method of the present invention may be used to automatically exclude data redundancy from consideration in certain applications disclosed herein, potentially potentially very large amounts of unaccounted storage space. Allows the size to be reduced frequently in several orders. In this regard, the system and method of the present invention allows all computers to share, read, write, or reference data simply, efficiently and securely, regardless of specific hardware or software characteristics. Making it possible to provide a unique and advantageous means of achieving this. The systems and methods of the present invention are particularly effective with networked computers or computer systems, but may be applied to isolated data storage with comparable results.
[0014]
The hash file system of the present invention advantageously solves many of the problems that plague conventional storage architectures. For example, the system and method of the present invention eliminates the need to manage a huge collection of directories and files along with all the wasted system resources that are inevitably generated by replication and slightly different copies. The maintenance and storage of replicated files plagues traditional company and personal computer systems and generally requires involvement in laborious “disk space cleanup”. The hash file system of the present invention effectively eliminates this problem by removing the disk space used for copying and removing the disk space used for partial copying almost completely. For example, in a conventional computer system, an additional gigabyte of storage is required to copy a gigabyte directory structure to a new location. In certain applications, the hash file system of the present invention reduces the disk space used in this operation by up to 100,000 times or more.
[0015]
Currently, some file systems have a mechanism to delete a copy, but all achieve this operation in a short time (in technical terms, O (l) ("On order of time") means that the system will factor the copy in time). This is O (N^**2) means a unit of time, as opposed to other systems that require O (N) or O (log (N)) time, which is related to the amount of storage being factored There is. Factoring storage at varying times is barely satisfactory for the system when the amount of storage is small, but as the system grows to larger sizes, it is the most efficient variable factoring system. Even it cannot be maintained. The hash file system of the present invention is designed to factor storage at a scale that has never been attempted before, with the ability to scale to a fairly large size in the first implementation, with 2 million. Petabit storage can be factored. Existing file systems cannot manage data at this scale.
[0016]
Furthermore, the hash file system of the present invention may be utilized to provide inexpensive, global computer system data protection and backup. Since the computer file system rarely changes more than a few percent of the total storage during each backup process, its factoring function operates on a typical backup data set very efficiently. Furthermore, the hash file system of the present invention can be used as a standard in an efficient message communication (e-mail) system. An electronic mail system is basically a data copy mechanism in which an author writes a message and sends it to a recipient list. E-mail systems effectively accomplish this “send” operation by copying data from one location to another. Authors generally keep a copy of the message they send, and the recipient keeps a copy of their own. These copies are often attached to responses that are also retained (ie, copies of copies). The commonality factoring feature of the present invention can remove this total inefficiency while clearly allowing email users to maintain this well-known copy-oriented paradigm.
[0017]
As mentioned above, since most data in the computer system is not changed first, the hash file system of the present invention allows the playback of a complete snapshot of the entire system, for example, any time during the day when the snapshot exists. It can be kept, or it can be kept continuously by snapshots taken at intervals of minutes (or less) depending on system needs. In addition, since conventional computer systems often provide limited versioning of files (ie, VAX® VMS® file system from Digital Equipment Corporation), the hash file system of the present invention also provides for this. Provides important advantages with respect to. Versioning in conventional systems presents both good and bad aspects. In the former, it helps prevent accidents, while in the latter, it requires a regular purge to reduce disk space consumption. The hash file system of the present invention provides file versioning with little overhead by factoring the same or edited copy with little extra space. For example, storing 100 revisions of a typical document usually requires about 100 times as much space as the original file. By using the hash file system disclosed herein, those revisions will require only 3 times the original space (document size, editing level and type, and external factors). Depending on).
[0018]
Yet another potential application of the hash file system of the present invention includes web serving. In this regard, the hash file system can be used to distribute web content efficiently, since the commonality factoring (hash) method also produces a uniform distribution on all hash file system servers. This uniform distribution allows a large array of servers to function as a huge web server farm with uniformly distributed loads. In other applications, the hash file system of the present invention can be used as a network accelerator because it can be used to reduce network traffic by sending a proxy (hash) of data instead of the data itself. A large percentage of current network traffic is duplicate data moving between locations. Proxy transmission of data allows an effective local cache mechanism to work, possibly allowing traffic on the Internet to be reduced by several orders.
[0019]
In particular, as disclosed herein, the hash file system and method of the present invention may be implemented using a 160-bit hash sum as a general pointer. This is different from traditional file systems that use pointers allocated from a central authority (ie, on Unix, 32-bit “i-nodes” are allocated by the kernel file system in lockstep operations and are not unique. Secure). In the hash file system of the present invention, these 160-bit hash sums are assigned by the hash algorithm without central authority (ie, no locking, no synchronization).
[0020]
Known hash algorithms generate a stochastic unique number that evenly spans the range. For hash function SHA-1, the range is 0-10e⁴⁸It is. This hashing operation is performed by examining only the contents of the stored data and can therefore be performed without complete isolation, asynchronousness and interlocking.
[0021]
A hash is an operation that can be verified by any component of the system, eliminating the need for operations that are trusted across those components. Thus, the hash file system and method of the present invention disclosed herein functions to remove an important bottleneck of conventional large distributed file systems (ie, trusted inclusion central authority). It allows large distributed file system structures that are unlimited for simultaneous read-write operations and can function without the risk of non-coherence and the limitations of certain conventional bottlenecks.
[0022]
The foregoing and other features and objects of the invention and the manner in which it will be accomplished will become more apparent from the following description, which is best understood by referring to the accompanying drawings and the following description of the preferred embodiments. Will.
[0023]
[Description of representative examples]
In certain implementations of the hash file system and method of the present invention as disclosed herein, the application affects the rapid advancement of the powerful nature of internetworking technologies such as commodity computing devices and the Internet. Aimed at high availability and high reliability data storage system. Specifically disclosed herein are one or more data blocks (including but not limited to files, directories, drive images, software applications, digital audio and rich media content) and the data blocks A hash file system that manages the correspondence of one or more symbols to a symbol, which can be derived from a number, hash, checksum, binary sequence, or data block itself, statistically, probabilistically, If not, it may be another identifier that is effectively unique to the data block. The system works for any computer system including, but not limited to: personal computers; supercomputers; distributed or non-distributed networks; storage area networks (“SAN” using IDE, SCSI, or other disk buses) "); Network Attached Storage (" NAS ") or other system capable of storing and / or processing data.
[0024]
In a particular implementation of the hash file system disclosed herein, the symbol may be an engine, program, or algorithm that includes, but is not limited to, MD4, MD5, SHA, SHA-1, or their derivatives. It may be derived using more than one hash or checksum. In addition, the symbol generates an engine, program, or a hash or checksum that generates an algorithm including but not limited to MD4, MD5, SHA, SHA-1, or a stochastic unique identifier based on data content. Some of the variable length or invariant length symbols derived using other methods may be provided. In certain implementations disclosed herein, file seeks or lookups that search for data or check for the presence / availability of data can be accelerated by looking at all parts of the symbol or a small portion of the symbol. Yes, some of the symbols indicate or otherwise provide routing information to find, retrieve or check the existence / availability of the data.
[0025]
In addition, hash file system systems and methods are disclosed herein, where the symbols enable identification of redundant copies within the system and / or redundant systems for data presented to filing and storage systems. Allows identification of the copy within. Symbols allow the removal or screening of redundant copies of data and / or portions of data in a system or data and / or portions of data presented to the system without loss of data integrity. The data can be distributed evenly on the effective storage of the system. As disclosed herein, the systems and methods of the present invention can store and / or process central operating points, processing balances, and / or data connected to all computers, supercomputers, or systems. Does not require input / output ("I / O") loads across other devices. Screening of redundant copies of data and / or portions of data provided in this application is based on intelligent boundaries that screen other data in the system, subsequent data presented to the system, or subsequent data stored by the system. Allows creation, iteration creation or saving.
[0026]
The invention is illustrated and described with reference to a distributed computing environment such as an enterprise computing system using a public communications line such as the Internet. However, an important feature of the present invention is that it is easily scaled up and down to meet the needs of a particular application. Therefore, the present invention can be applied to a considerably large and complicated network environment as well as a small network environment such as a conventional LAN system, unless there is an instruction to the contrary.
[0027]
With reference to FIG. 1, the present invention can be used with a novel data storage system on the network 10. In this figure, a typical internetwork environment 10 is an Internet comprising a global internetwork formed by logical and physical connections between a multiple wide area network (“WAN”) 14 and a local area network (“LAN”) 16. May be included. The Internet backbone 12 represents the main line and router that carries the bulk of the data traffic. The backbone 12 is formed by the largest network of systems managed by major Internet service providers (“ISPs”) such as GTE, MCI, Sprint, UUNet and America Online. A single connection line is used to conveniently illustrate the connection of WAN 14 and LAN 16 to the Internet backbone 12, but in reality there are multiple paths that can be routed between multiple WANs 14 and LAN 16. It should be understood that it exists. This makes the internetwork 10 powerful in the face of single or multiple failure points.
[0028]
It is important to distinguish network connections from internal data paths implemented between peripheral devices in the computer. The “network” comprises a general purpose system and typically switches physical connections that allow logical connections between processes running on node 18. The physical connection realized by the network is generally independent of the logical connection established between processes using the network. In this way, different process sets ranging from file transfer and mail transfer can use the same physical network. Conversely, a network can be formed from a heterogeneous set of physical network technologies that are not visible to processes that are logically connected using the network. Because the logical connection between processes implemented by the network is independent of the physical connection, the internetwork is easily scaled to a virtually unlimited number of nodes over long distances.
[0029]
In contrast, internal data paths such as system buses, peripheral component interconnect (“PCI”) buses, intelligent drive electronics (“IDE”) buses, and small computer system interface (“SCSI”) buses are special in the computer system. Define a physical connection to achieve the target connection. These connections realize physical connections between physical devices with respect to logical connections between processes. These physical connections are characterized by a limited distance between components, a limited number of devices that can be coupled to the connection, and devices in a conditional format that can be connected through the connection.
[0030]
In certain implementations of the invention, the storage device may be located at node 18. Every node 18 storage may comprise a single hard disk or may comprise a managed storage system such as a conventional RAID device having multiple hard disks configured as a single logical volume. Importantly, the specific configuration of storage in any node is less important because the present invention manages redundant operations across the node as opposed to doing it in a node.
[0031]
Optionally, one or more of the nodes 18 may implement a storage allocation management (“SAM”) process that manages data storage across the nodes 18 in a distributed and collaborative manner. The SAM process as a whole preferably functions so as to have little or no centralized control over the system. The SAM process provides data distribution across the nodes 18 and achieves recovery in a manner that allows failures across the network nodes 18 in a manner similar to the paradigm found in the RAID storage subsystem.
[0032]
However, because the SAM process works across the entire node rather than within one node or one computer, the process allows greater fault tolerance and storage efficiency levels than conventional RAID systems. For example, the SAM process can be recovered even if the network node 18, LAN 16 or WAN 14 becomes unavailable. Furthermore, even if some Internet backbones 12 become unavailable due to failure or congestion, the SAM process can be recovered using data distributed at nodes 18 that remain accessible. Thus, the present invention affects the powerful nature of internetworks and provides unprecedented availability, reliability, fault tolerance and robustness.
[0033]
With reference to FIG. 2, a more detailed conceptual diagram of a typical network computing environment in which the present invention is implemented is depicted. The internetwork 10 of the preceding figure (or the Internet 118 in this figure) is an interconnected network of heterogeneous computing devices and mechanisms 102 that range from a supercomputer or data center 104 to a handheld or pen-based device 114. Allow 100. Although such devices have different data storage needs, they share the ability to retrieve data over the network 100 and operate on the data in their own resources. Heterogeneous computing devices 102 including mainframe computers (eg VAX station 106 and IBM AS / 400 station 116) as well as personal computer or workstation class devices such as IBM compatible device 108, Macintosh device 110 and laptop computer 112 are Easily interconnect via internetwork 10 and network 100. Although not shown, mobile and other wireless devices may be coupled to internetwork 10.
[0034]
The internet-based network 120 includes a set of logical connections, some of which are created between multiple internal networks 122 via the internet 118. Conceptually, Internet-based network 120 is similar to WAN 14 (Figure 1), which allows logical connections between geographically distant nodes. The Internet-based network 120 may be implemented using the Internet 118 or other public and private WAN technologies including leased lines, Fiber Channel, etc.
[0035]
Similarly, the internal network 122 is conceptually similar to the LAN 16 (FIG. 1), which allows logical connections over a limited distance than the WAN 14. The internal network 122 may be implemented using various LAN technologies including Ethernet, Fiber Distributed Data Interface (“FDDI”), Token Ring, AppleTalk, Fiber Channel, etc.
[0036]
Each internal network 122 connects one or more redundant arrays of independent node (RAIN) elements 124 to implement a RAIN node 18 (FIG. 1). Each RAIN element 124 includes a processor, memory and one or more mass storage devices (eg, hard disks). The RAIN element 124 also includes a hard disk controller that may be a conventional IDE or SCSI controller and may be a management controller such as a RAID controller. The RAIN elements 124 may be physically distributed and may be co-located in one or more racks that share resources such as cooling and power. Each node 18 (Figure 1) is independent of the other nodes 18, and the failure or inactivity of one node 18 does not affect the availability of the other node 18, and the data stored in one node 18 Can be restored from data stored in other nodes 18.
[0037]
In certain typical implementations, the RAIN element 124 is an Intel-based installation mounted on a motherboard that supports the PCI bus and 256 MB of random access memory (“RAM”) housed in a traditional AT or ATX case. A computer using a commodity component such as a microprocessor may be provided. The SCSI or IDE controller may be implemented on the motherboard and / or by an expansion card connected to the PCI bus. If the controller is implemented only on the motherboard, a PCI expansion bus may be used as an option. In certain implementations, the motherboard implements a PCI expansion card that is used to implement two mastering EIDE channels and two additional mastering EIDE channels so that each RAIN element 124 contains up to four or more EIDE hard disks it can. In a particular implementation, each hard disk may comprise an 80 gigabyte hard disk with a total storage capacity of 320 gigabytes or more per RAIN element. The hard disk capacity and configuration within the RAIN element 124 can be easily increased or decreased to meet the needs of a particular application. The jacket also houses support mechanisms such as a power source and a cooling device (not shown).
[0038]
Each RAIN element 124 runs an operating system. In certain implementations, a Unix variant operating system such as UNIX or Linux may be used. However, it is contemplated that other operating systems including DOS, Microsoft Windows, Apple Macintosh OS, OS / 2, Microsoft Windows NT, etc. can be equally replaced by making predictable changes in performance. The selected operating system forms a platform for executing application software and processes and implements a file system that accesses mass storage via a hard disk controller. Various application software and processes are implemented on each RAIN element 124 and use appropriate network protocols (eg, User Datagram Protocol (“UDP”), Transmission Control Protocol (TCP), Internet Protocol (IP), etc.) Network connectivity can be provided through the interface.
[0039]
With respect to FIG. 3, a logic flow chart is shown to represent the steps of entering a computer file into the hash file system of the present invention, wherein the hash value of the file is stored in advance in a set or database Matched.
[0040]
Process 200 begins by inputting computer file data 202 (eg, “File A”) into the hash file system (“HF”) of the present invention, and a hash function is performed at step 204. The data 206 representing the hash of file A is then compared at decision step 208 with the set of contents including the hash file value. If the data 206 is already in the set, the hash value of the file is added to the directory list at step 210. The contents of the set 212, including the hash value and corresponding to the data, are provided in the form of an existing hash value 214 for the comparison operation of decision step 208. On the other hand, if the hash value of file A is not currently in the set, the file is split into hash pieces at step 216 (as described more fully below).
[0041]
With respect to FIG. 4, a further logic flowchart is provided to represent the steps in process 300 for breaking up a digital sequence (eg, a file or other data sequence) into hash pieces. This process 300 will eventually generate a probabilistically unique hash value for each piece as well as a number of data pieces.
[0042]
The file data 302 is divided into pieces based on the commonality with other pieces of the system or the possibility of pieces that will later become apparent in step 304. The result of the operation of step 304 on the file data 302 generates four file pieces 306 named A1-A5 in the representative example.
[0043]
Each of the file pieces 306 is then manipulated at step 308 by placing each through an individual hash function operation such that each of the A1-A5 pieces 306 is assigned a stochastic unique number. As a result of the operation of step 308, each of the pieces 306 (A1-A5) will have an associated, stochastic unique hash value 310 (shown as A1-hash-A5-hash, respectively). The file splitting process of step 304 is described in further detail below, along with the unique “sticky bit” operations disclosed herein.
[0044]
With further reference to FIG. 5, another logic flow diagram is shown to represent a process 400 that compares the hash value 310 of each piece 306 of the file with the existing hash value 214 maintained in the set 212. In particular, at step 402, the hash value 310 of each piece 306 of the file is compared to the existing hash value 214 and the new hash value 408, and the corresponding new data piece 406 is added to the set 212. In this way, hash values 408 that are not pre-existing in the database set 212 are added along with the data pieces 406 associated with them. The process 400 will also generate a record 404 that indicates that one hash value for all file pieces is equivalent to the hash value 310 of the various pieces 306.
[0045]
Further, FIG. 6 shows another logic flow diagram illustrating a process 500 that compares a file hash or directory list hash value with an existing directory list hash value and adds the new file or directory list hash value to the database directory list. Process 500 operates on stored data 502 comprising a cumulative list of file names, file metadata (eg, date, time, file length, file type, etc.) and file hash values for each item in the directory. At step 504, a hash function is performed on the contents of the directory list. Decision step 506 functions to determine whether the hash value of the directory list is a set 212 of existing hash values 214. If the determination is affirmative, process 500 returns to add another file hash or directory list hash to the directory list. In contrast, if the directory list hash value is not already in the database set 212, the directory list hash value and data are added to the database set 212 in step 508.
[0046]
Still referring to FIG. 7, a comparison 600 of a representative computer file (ie, “File A”) piece 306 and their corresponding hash value 310 is shown both before and after editing a particular piece of a typical file. In this example, the record 404 includes the hash value of file A as well as the hash value 310 of each of the pieces of files A1-A5. A representative edit of file A can produce a corresponding change in hash value A2-b of hash value 310A along with a change in data of piece A2 (represented by A2-b) of file piece 306A. The edited file piece generates an updated record 404A that includes the modified hash value of file A and the modified hash value of piece A2-b.
[0047]
Still referring to FIG. 8, the composite data (eg, composite data 702 and 704) derived by the system and method of the present invention is substantially the same as the explicitly represented data 706, but with a “recipe” or formula. A conceptual diagram 700 is shown illustrating the fact that it will be created. In the illustrated example, the recipe includes the result of a function using the concatenation of data represented by the corresponding hash 708 or the data represented by the hash. The data block 706 may be a variable length quantity as shown, and the hash value 708 is derived from the data block associated with it. As already mentioned, the hash value 708 is a stochastic unique ID of the corresponding data piece, but a completely unique ID can be used instead, or can be mixed with a stochastic unique ID. . Composite data 702, 704 can also refer to other complex data at many levels, while the hash value 708 of the composite data can be derived from the value of the data generated by the recipe or the hash value of the recipe itself. It should also be noted that there are.
[0048]
Still referring to FIG. 9, how the hash file system and method of the present invention organizes data 802 and optimizes the reuse of redundant sequences by using hash value 806 as a pointer to the data they represent. In another conceptual diagram 800, the data 802 may be represented as a distinct byte sequence 808 (atomic data) or a group of sequences (composite) 804.
[0049]
The diagram 800 illustrates the immense commonality of recipes and data that is reused at every level. The basic configuration of the hash file system of the present invention is essentially a “tree” or “bush” in which a hash value 806 is used instead of a conventional pointer. A hash value 806 is used in the recipe to indicate another hash value that can be data or the recipe itself. Thus, in essence, the recipe indicates another recipe, the other recipe further indicates another recipe, eventually shows some specific data, and the specific data itself indicates another recipe that shows more data. It can be shown, and finally only data.
[0050]
Further with reference to FIG. 10, a simplified diagram 900 illustrating an exemplary 160-bit hash value 902 hash file system address translation function is shown. Hash value 902 includes a data structure with a front 904 and a back 906 as shown, and diagram 900 allows the use of hash value 902 to go to a particular node location in a system that includes the corresponding data. Fig. 2 illustrates a particular "0 of 1" operation used.
[0051]
The diagram 900 shows how the front 904 of the data structure of the hash value 902 can be used to indicate a hash prefix to the stripe identification (“ID”) 908, and then the stripe ID It illustrates how it is used to map to an IP address and map an ID class to an IP address 910. In this example, “S2” indicates stripe 2 of index node 37 912. The index stripe 912 of node 37 then indicates the stripe 88 of data node 73 indicated by reference number 914. Then, in operation, the hash value 902 itself can be used to indicate which node of the system contains the associated data, and another part of the hash value 902 indicates which data stripe is at that particular node. And another portion of the hash value 902 can be used to indicate where the data is in the stripe. Through this three-step process, it can be quickly determined whether the data represented by the hash value 902 already exists in the system.
[0052]
With further reference to FIG. 11, a simplified schematic diagram of the index stripe splitting function 1000 used in the system and method of the present invention is shown. In this figure, the typical function 1000 can be used to effectively split stripe 1002 (S2) into two stripes 1004 (S2) and 1006 (S7) so that one stripe is sufficiently full. It is shown that. In this example, odd entries are moved to stripe 1006 (S7), while even entries remain in stripe 1004. This function 1000 is an example of how stripe entries can be handled as the overall system size increases and complexity increases.
[0053]
With further reference to FIG. 12, a simplified diagram 1100 of the overall functionality of the system and method of the present invention is used, for example, to back up data for a typical home computer having multiple programs and document files 1102A and 1104A on the first day. The program file 1102B remains the same on the second day, while one of the document files 1104B is edited on the second day (Y.doc) and the third document file (Z.doc) is added.
[0054]
FIG. 1100 details how a computer file system can be divided into pieces and then listed as a series of recipes on a global data protection circuit (“gDPN”) to recover the original data from the pieces. Show. This very small computer system is shown in the form of “snapshot” on “first day” and then on “second day” thereafter. “Program File H5” and “My Document H6” are illustrated by the numeral 1106 on “First Day”, the former is represented by the recipe 1108, the first executable file is represented by the hash value H1 1114, The second executable file is represented by the hash value H2 1112. The document file is represented by a hash value H6 1110, the first document is represented by a hash value H3 1118, and the second document file is represented by a hash value H4 1116. After that, “Program File H5” and “My Document H10” indicated by the number 1120 in “Second Day” indicate that “Program File H5” has not been changed, but “My Document H10” has been changed. ing. H10 indicated by the number 1122 indicates that “X.doc” is still represented by the hash value H3 1118, while “Y.doc” is now represented by the hash value H8 in the number 1124. Show. The new document file “Z.doc” is represented by the hash value H9 at the numeral 1126.
[0055]
In this example, it can be seen that on the second day, some files were changed and others were not changed. In the modified file, some of those pieces have not been modified and others have been modified. Through the use of the hash file system of the present invention, a “snapshot” of the computer system can be created on the first day (creating the recipe needed to play the computer file so that it can exist next), and then on the second day. Several reuses of recipes from previous days, re-formulation of other recipes, and addition of new recipes describe the current system. In this way, the computer system is reproducible in its entirety at any time both on the first or second day and on any subsequent day.
[0056]
Further with reference to FIG. 13, a comparison 1200 of various pieces of a particular document file marked by a number of “sticky bytes” 1204 can be seen both before editing (1st day 1202A) and after editing (2nd day 1202B). It is shown that one of the pieces is changed and the other piece remains the same.
[0057]
For example, on the first day, file 1202A comprises variable length pieces 1206 (1.1), 1208 (1.2), 1210 (2.1), 1212 (2.), 1214 (2.3) and 1216 (3.1). On the second day, pieces 1206, 1208, 1210, 1214 and 1216 remain the same (and thus have the same hash value), while piece 1212 is edited to produce piece 1212A (and thus has a different hash value). .
[0058]
Data sticky bytes (or “sticky points”) are a unique fully automated method of subdividing computer files so that common block elements can be found on multiple related and unrelated computers without having to communicate between computers. . The means of finding sticky points is essentially mathematical in nature and is performed equally and properly regardless of the data content of the file. In the hash file system of the present invention, all data objects are indexed and stored and retrieved using, for example (but not limited to) industry standard checksums such as MD4, MD5, SHA or SHA-1. It's okay. In operation, if two files have the same checksum, it can be considered very likely that they are the same file. Using the systems and methods disclosed herein, data sticky points can be generated with a standard mathematical distribution and standard deviation with a small percentage of the target size.
[0059]
A data sticky point is an array of n bytes that is statistically rare. In this case, 32 bytes are used in the example because it is easy to implement in current microprocessor technology.
[0060]
A 32-bit rolling hash can be generated for file “f”: // f [i] is the i th byte of file “f”.
// scramble is an integer 256 entry array, each 32 bits long.
// These integers are generally chosen to evenly span the range.
int t = 8 // Target number of postfix 0.
int hash = 0;
int sticky bits;
for (int i = O; i <filesize; i ++)
hash = hash >> 1 | scramble [f [i]];
// In each byte of the file, the hash represents the rolling hash of the file.
sticky bits = (hash-1) ^ hash;
// sticks bits is a variable and has one of the hash numbers corresponding to the number of postfix 0s of the “hash”.
number of bits = count ones (stick-bits);
if (number of bits> t)
output sticky point (i);
}
[0061]
The sticky point is defined to be a rolling hash having at least the number of trailing zeros as a target number having a hash represented in binary. Statistically, this algorithm finds points that are 2 ^ t apart, where t is a post zero target number. In this example, if t = 8, the algorithm finds sticky points that are 2 ^ 8 = 256 bytes apart on average.
[0062]
A 32-bit rolling hash can be generated for an f file, where:
f [i] is the i-th byte of file f
A scramble is a 256-entry array of random elements, each n bits long:
int t = 8 // Zero target number
int target distance = 256; // 2 to the 8th power
int hash = 0;
int sticky bits;
int distance = 0;
int last point = 0;
for (int i = O; i <filesize; i ++) [
hash = hash >> 1 | scramble [f [i]];
// The hash for each byte of the file represents the rolling hash of the file.
sticky bits = (hash-1) ^ hash;
// sticks bits is a variable and has one of the numbers corresponding to the number of trailing 0s of the “hash”.
number of bits = count ones (stick bits);
distance = i-last point;
if (number of bits^* distance / target distance> t)
last point = i;
output sticky point (i);
}
}
[0063]
Although the hash function used to implement the hash file system of the present invention requires reasonably complex calculations, it is sufficient within the capabilities of current computer systems. Hash functions are inherently based on probability, so if two different data objects accidentally have the same hash value, any hash function may produce incorrect results. However, the systems and methods disclosed herein are at a level where the probability of collision is acceptable for reliable use (ie, an opportunity of 10 to the 24th power) and is acceptable with conventional computer hardware operations. This problem is mitigated with a well-known and investigated hash function that reduces to a much lower level than the error rate that is being done.
[0064]
As used herein, the term “Internet infrastructure” includes a variety of hardware and software mechanisms, but the term mainly refers to the ability to move data packets from one network node to another. Refers to routers, router software, and physical links between these routers. Also, as used herein, a “digital sequence” is represented by computer program files, computer applications, data files, network packets, streaming data such as multimedia (including audio and video), telemetry data, and digital or numeric sequences. Any other data form that can be included can be included, but is not limited to such. Probabilistically unique identifiers generated by the hash file system and method of the present invention may also be used in network applications as URLs.
[0065]
Although the principles of the present invention have been described above with reference to specific implementations and applications of the systems and methods of the present invention, the foregoing description is by way of example only and is intended to limit the scope of the invention. It was clearly understood that this is not the case. In particular, it will be appreciated that the teachings of the foregoing disclosure suggest other variations to those skilled in the art. Such alternative embodiments may include other features that are already known per se and that can be used in place of or in addition to features already described herein. Although the claims are expressly set forth in this application for specific feature combinations, the scope of the disclosure herein also relates to the same invention as the claims, and Any novel feature or combination of features disclosed explicitly or implicitly obvious to those skilled in the art, or a generalized example thereof, alleviating all of the same technical problems as encountered It should also be understood that variations are included. Applicant thereby reserves the right to develop the features and / or combinations of the features into new claims during the entire procedure of this application or of further applications derived therefrom.
[Brief description of the drawings]
FIG. 1 is a high-level diagram of a representative networked computer environment in which the system and method of the present invention can be implemented.
FIG. 2 is a more detailed conceptual diagram of a possible operating environment for utilizing the system and method of the present invention, where files maintained in any number of computers or data centers can be located, for example, in geographically diverse locations. It may be stored in a distributed computer system through an internet connection to a redundant array (“RAIN”) rack of multiple independent nodes located.
FIG. 3 is a logic flow chart representing the steps of inputting a computer file into the hash file system of the present invention, where the hash value of the file is checked against the hash value of the file previously maintained in the set or database.
FIG. 4 is a further logic flow diagram representing the steps of breaking up a file or other data sequence into hash pieces, with a number of data pieces and corresponding stochastic unique hash values for each piece. Will be generated.
FIG. 5 is another logic flow chart showing a comparison of the hash value of each piece of a file with an existing hash value of a set (or database), where the product of the record is a hash value for all files and It shows that the hash values of the various pieces are equivalent, and a new data piece and a corresponding new hash value are added to the set.
FIG. 6 is another logic flowchart illustrating the steps of comparing a file hash or directory list hash value with an existing directory list hash value and adding a new file or directory list hash value to the set directory list.
FIG. 7 is a comparison of representative computer file pieces and their corresponding hash values both before and after editing a particular piece of a typical file.
FIG. 8 The composite data derivable by the system and method of the present invention is effectively identical to the explicitly displayed data, but instead is represented by a “recipe” (eg, its corresponding hash) It is a conceptual diagram of the fact that it may be generated by concatenation of data or the result of a function using data represented by a hash.
FIG. 9 illustrates how the hash file system and method of the present invention can be used to organize data and optimize reuse of redundant sequences by using hash values as pointers to the data they represent. The data may be represented as a distinct byte sequence (atomic data) or a group of sequences (composite).
FIG. 10 is an illustrative simplified diagram of a typical 160-bit hash value hash file system address translation function.
FIG. 11 is a simplified exemplary diagram of an index stripe splitting function used in the system and method of the present invention.
FIG. 12 is a simplified diagram of the overall functionality of the system and method of the present invention, showing the day 1 being used to back up data for a typical home computer having a large number of programs and document files. One of the document files is edited on the second day, and a third document file is added.
FIG. 13 is a comparison of various pieces of a particular document file marked by a number of “sticky bytes”, both before and after editing, whereby one of the pieces is changed and the other The pieces remain the same, showing the figure.

Claims (63)

A method for managing data including digital sequences,
A computer system using a function to generate a stochastic unique identifier for the digital sequence;
The computer system comparing the stochastic unique identifier to a list of other identifiers stored in a storage device corresponding to other digital sequences;
If the stochastically unique identifier is previously not in said list, a step of the computer system to split the digital sequence into a plurality of shorter digital sequence,
The computer system using a function to generate a stochastic unique identifier for each of the plurality of shorter digital sequences;
Comparing the said list a unique identifier the stochastically said computer system corresponding to said plurality of shorter digital sequence,
If the stochastic unique identifier corresponding to the shorter digital sequence is not previously in the list, the computer system adds the shorter digital sequence and the corresponding stochastic unique identifier to the list;
The computer system removes the shorter digital sequence and the corresponding stochastic unique identifier from the list if the stochastic unique identifier corresponding to the shorter digital sequence is pre-existing in the list; or Screening the method. The computer system dividing the digital sequence into a plurality of shorter digital sequences;
Further comprising The method of claim 1, comprising the steps of: the computer system to generate a like plurality of stochastically unique identifier corresponding to each of the plurality of shorter digital sequence. The dividing step, to generate a short digital sequence than said having a variable length individually The method of claim 1, wherein. The dividing step is based on the content of the digital sequence, The method of claim 1, wherein. The dividing step, the is based on meta-data describing the digital sequence, The method of claim 1, wherein. The dividing step, to generate a short digital sequence than said having a substantially unchanged length The method of claim 1, wherein. Generating the same plurality of stochastic unique identifiers;
The running individually hash function computer system for short digital sequence from the said comprising the step of generating a like plurality of stochastically unique identifier, the method of claim 1. Further comprising The method of claim 1, wherein the step wherein the computer system is to use at least a portion of the stochastically unique identifier as an indicator to the location of said list for said comparing step. A data storage system,
A storage device for storing at least one list that maintains a portion of the digital sequence included in the data and a corresponding probabilistically unique identifier for each of the portion of the digital sequence;
Connected to the storage device, receives at least one new digital sequence , generates a stochastic unique identifier for the digital sequence using a function, and converts the stochastic unique identifier to another digital sequence corresponding with, and compared to a list of other identifiers stored in the storage device, if the stochastically unique identifier is previously not in the list, divide the new digital sequence into a plurality of shorter digital sequence, A controller that uses a function to generate a stochastic unique identifier for each of the shorter digital sequences, wherein any of the stochastic unique identifiers for each of the plurality of shorter digital sequences one of determining whether the currently maintained in the list, the maintained at least one list Any one of the stochastically unique identifier is not determined there, the controller adding a short digital sequence than corresponding to said any one of stochastically unique identifier to the at least one list When,
A data storage system comprising: The data storage system of claim 9 , wherein the at least one list comprises a plurality of lists. The data storage system of claim 10 , wherein each of the plurality of lists includes a portion of the stochastic unique identifier. The data storage system of claim 10 , wherein at least one of the plurality of lists is physically moved from the other of the at least one list. The data storage system of claim 10 , wherein the plurality of lists are partitioned based on the stochastic unique identifier. The data storage system of claim 12 , wherein the plurality of lists are coupled by a network. The data storage system of claim 9 , wherein the at least one list is physically moved from the at least one partitioning mechanism. The data storage system of claim 9 , wherein the list comprises a physically distributed database. The data storage system of claim 9 , wherein the controller and the storage device are coupled by a network. The data storage system of claim 17 , wherein the network comprises a public network such as the Internet. The data storage system of claim 18 , wherein the controller and the storage device are physically distributed. The data storage system of claim 9 , wherein the stochastic unique identifier is generated by a hash function. The data storage system of claim 20 , wherein the hash function comprises an industry standard digest algorithm. The data storage system of claim 21 , wherein the hash function comprises one of an MD4, MD5 SHA or SHA-1 algorithm. 21. The data storage system of claim 20 , wherein the stochastic unique identifier is generated by a checksum. The data storage system of claim 9 , wherein the digital sequence is variable length. The data storage system of claim 9 , wherein the digital sequence is invariant length. 16. The data storage of claim 15 , wherein the controller is effective to utilize at least a portion of the stochastic unique identifier for the plurality of the shorter digital sequences as a locator associated with the list partitioning. system. The data storage system of claim 9 , wherein the digital sequence comprises a data file. The data storage system of claim 9 , wherein the digital sequence comprises a data stream. The data storage system of claim 9 , wherein the digital sequence comprises an executable file. The data storage system of claim 9 , wherein the digital sequence comprises a database record. The data storage system of claim 9 , wherein the digital sequence comprises a database index. The data storage system of claim 9 , wherein the digital sequence comprises a digital device image. The data storage system of claim 9 , wherein the digital sequence comprises a network packet. The data storage system of claim 9 , wherein the digital sequence comprises a digitized analog signal. A computer program for managing data including digital sequences,
Computer readable program code configured to cause a computer to generate a stochastic unique identifier of a digital sequence using a function;
Computer readable program code configured to cause the computer to compare the stochastic unique identifier with a list of other identifiers corresponding to other digital sequences;
The stochastically case a unique identifier previously not in the list, the computer, the computer readable program code that is configured to split the digital sequence into a plurality of shorter digital sequence,
The computer is configured to use a function to generate a stochastic unique identifier for each of the shorter digital sequences, and each shorter digital sequence is configured to be associated with each stochastic unique identifier. Computer readable program code,
Computer readable program code configured to cause the computer to compare the stochastic unique identifier with the list;
If the stochastic unique identifier is not previously in the list, the computer is added to the list with the stochastic unique identifier and a shorter digital sequence corresponding to the stochastic unique identifier. A computer program comprising computer readable program code configured to be performed . The computer readable program code configured to cause the computer to generate is
Including computer readable program code configured to cause a computer to perform a hash function of the digital sequence to generate the stochastic unique identifier;
36. The computer program according to claim 35 . 38. The computer program of claim 36 , wherein the computer readable program code configured to cause a computer to perform a hash function is executed by an industry standard digest algorithm. 38. The computer program of claim 37 , wherein the computer readable program code configured to cause a computer to perform a hash function is executed by one of an MD4, MD5, SHA or SHA-1 algorithm. The computer readable program code configured to cause a computer to generate is
36. The computer program of claim 35 , comprising computer readable program code configured to cause a computer to generate a checksum of the digital sequence to generate the stochastic unique identifier. 36. The computer program of claim 35 , further comprising computer readable program code configured to cause a computer to create a directory list that includes the stochastic unique identifier of the digital sequence. 36. The computer program of claim 35 , wherein the computer readable program code configured to cause a computer to generate the shorter digital sequence having individually variable lengths. 36. The computer program of claim 35 , wherein the computer readable program code configured to cause a computer to generate the shorter digital sequence having a substantially invariant length. The computer readable program code configured to cause a computer to generate the same plurality of stochastic unique identifiers,
36. The computer program of claim 35 , comprising computer readable program code configured to cause a computer to individually execute the hash function of the shorter digital sequence to generate the plurality of similar stochastic unique identifiers. 36. The computer of claim 35 , further comprising computer readable program code configured to cause the computer to add the plurality of shorter digital sequences and the corresponding similar plurality of stochastic unique identifiers to the list. program. 36. The computer program of claim 35 , further comprising computer readable program code configured to cause a computer to utilize at least a portion of the stochastic unique identifier as an index into a location table of the list for the comparing step. . The storage device establishes a correspondence between each shorter digital sequence and each stochastic unique identifier so that each shorter digital sequence is associated with each stochastic unique identifier. the method of claim 1 comprising the step. 48. The method of claim 46 , wherein the identifier and the corresponding shorter digital sequence are maintained in at least one data list. 48. The method of claim 46 , wherein at least some of the identifiers can be used as pointers to the location of the corresponding shorter digital sequence in the at least one data list. 47. The method of claim 46 , wherein the shorter digital sequence comprises at least a portion of a data file and the identifier is uniquely associated with content of the at least a portion of the data file. 47. The method of claim 46 , wherein the shorter digital sequence comprises at least a portion of a data stream, and the identifier is uniquely associated with content of the at least a portion of the data stream. 47. The method of claim 46 , wherein the at least some of the digital sequences comprise at least some executable files and the identifier is uniquely associated with content of the at least some executable files. The function execution step includes:
47. The method of claim 46 , wherein the method is performed by performing a hash function of the shorter digital sequence to generate the stochastic unique identifier. 53. The method of claim 52 , wherein performing the hash function is performed by an industry standard digest algorithm. 54. The method of claim 53 , wherein performing the hash function is performed by one of an MD4, MD5, SHA or SHA-1 algorithm. Causes the computer to establish a correspondence between each of the plurality of shorter digital sequences and each of the stochastic unique identifiers so that each shorter digital sequence is associated with each stochastic unique identifier. 36. The computer program of claim 35 , comprising computer readable program code configured as follows. 56. The computer program of claim 55 , wherein the identifier and the corresponding shorter digital sequence are maintained in at least one data list. 57. The computer program of claim 56 , wherein at least some of the identifiers can be used as pointers to the location of the corresponding shorter digital sequence in the at least one data list. 56. The computer program of claim 55 , wherein the shorter digital sequence comprises at least a portion of a data file and the identifier is uniquely associated with content of the at least a portion of the data file. 56. The computer program of claim 55 , wherein the shorter digital sequence comprises at least a portion of a data stream and the identifier is uniquely associated with content of the at least a portion of the data stream. 56. The computer program of claim 55 , wherein the shorter digital sequence comprises at least a portion of an executable file and the identifier is uniquely associated with content of the at least a portion of the executable file. The computer readable program code configured to cause a computer to execute a function comprises:
56. The computer program of claim 55 , executed by computer readable program code configured to cause a computer to perform a hash function on the shorter digital sequence to generate the stochastic unique identifier. 62. The computer program of claim 61 , wherein the computer readable program code configured to cause a computer to perform a hash function is executed by an industry standard digest algorithm. 64. The computer program of claim 62 , wherein the computer readable program code configured to cause a computer to perform a hash function is executed by one of the MD4, MD5, SHA or SHA-1 algorithms.

Images