KR100860821B1 - Computing system, method for establishing an identifier and recording medium with a computer readable program for use in a commonality factoring system - Google Patents

Abstract

A computer file system constructed based on hash and / or constant strings, different or varying lengths 304 and capable of removing or screening redundant copies of cluster blocks (or portions of data blocks) of data from the system. Is disclosed. The hash file system of the present invention provides a hash value for the computer files or file fragments 306 that can be generated by a checksum generator, engine, or algorithms such as industry standard MD4, MD5, SHA or SHA-1 algorithms. 310). Alternatively, this hash value may be generated 308 by a checksum program, engine or algorithm, or other means for effectively generating a unique hash value for a block of data of an undetermined size based on a mathematical algorithm.

Checksum, packet, hard disk, RAID, NAS, MD5

Description

Computing system, method for setting identifier, and recording medium with computer readable program for use in common element differentiation system.

The present invention generally relates to the field of hash file systems and common element differentiation systems. In particular, the present invention relates to systems and methods for determining a correspondence between electronic files in a distributed computer data environment and specific applications thereto.

Economic, political and social power are increasingly managed by data. Data represents transactions and goods. Political power is analyzed and revised based on data. Human interactions and relationships are defined by data exchange. Thus, efficient distribution, storage, and management of data is expected to play an increasingly essential role in human society.

The amount of data that must be managed in the form of computer programs, databases, files, etc. is increasing exponentially. As computer processing power increases, operating systems and application software become larger. Moreover, the desire to access larger data sets, such as multimedia files and large databases, is increasing the amount of data managed. This growing data load must be transferred between computing devices and stored in an accessible form. The exponential growth rate of data is expected to outperform communications bands and storage capacity, making data management more urgent using conventional methods.

In conventional data storage systems many factors must be balanced and often compromised. As the amount of data is growing extremely large, pressure continues to reduce the cost per bit stored. In addition, the data management system must be extensible to address not only current needs but also future needs as well. Preferably, the storage system is gradually scalable such that the user can purchase only the capacity required at any particular time. High reliability and high availability are also taken into account as data users become increasingly unbearable for loss, corruption, and invalid data. However, conventional data management structures must compromise these factors, so no structure provides a cost-effective, reliable, high availability, scalable solution.

Conventional RAID (Redundant Array of Independent Disks) is a way of storing (and thus redundant) the same data in different locations on multiple storage devices, such as hard disks. By placing data on multiple disks, input / output (“I / O”) operations can be superimposed in a balanced manner, improving performance. Since the use of multiple disks increases the average failure interval ("MTBF"), the stored data also redundantly increases fault-tolerance. RAID systems rely on hardware or software controllers to appear as a single logical hard disk for the operating system, hiding the complexity of actual data management. However, RAID systems are difficult to scale due to cabling and physical limitations of the controller. In addition, since the utilization of the RAID system is highly dependent on the function of the controller itself, in the event of a controller failure, data stored after the controller becomes unavailable, i.e., after failure, cannot be utilized. Moreover, RAID systems tend to be expensive solutions as they need to be specialized rather than commodity hardware.

NAS (network-attached storage) refers to hard disk storage that is set to its own network address rather than to an application server. File requests are mapped by NAS file servers. NAS can use hardware or software-based RAID to provide transparent I / O operation. In addition, the NAS can further improve fault tolerance by automating data mirroring to one or more other NAS devices. Since NAS devices can be added to a network, they allow for expansion of the total storage available to the network. However, NAS devices are limited in the capabilities of conventional RAID controllers within RAID applications. NAS systems are also a limited solution because they do not allow mirroring and parity between nodes.

In addition to data storage problems, data transmission is evolving rapidly with improvements in long-range networks ("WAN") and internetworking technologies. For example, the Internet has created a world-wide networked environment with a nearly ubiquitous approach. Despite rapid network infrastructure improvements, the rate of increase in the amount of data that needs to be transmitted is expected to outweigh the improvements in available bandwidth.

Coolly, the way data is typically managed is incompatible with hardware devices and infrastructures that have been developed to manipulate and transmit data. For example, a computer is a general purpose machine that is easily programmed to perform a variety of virtually unlimited functions. In many cases, however, computers are equipped with fixed, slow-changing data sets that limit their general purpose to function for special purposes. For commercial computers, the improvement in processing speed, peripheral performance, and data storage capacity is most significant. However, many data storage solutions are not scaled by the storage controllers on which they are based, but are not capable of taking advantage of these improvements because they are limited. Likewise, the Internet has been developed as a fault tolerant, multipath interconnection network. However, conventionally, network resources are implemented in certain network nodes such that failure of the nodes renders the resource unavailable despite the fault tolerance of the network to which the node is connected. There is a continuing need for high availability, high reliability, and highly scalable data storage solutions.

The specification is constructed based on constant, different, or varying length hashes and / or digit strings to remove or screen redundant copies of blocks of data (or portions of data blocks) from a system. A system and method for a computer file system are disclosed. Also disclosed herein are systems and methods for computer file systems, where industry standard Message Digest 4 ("MD4"), MD5, Secure Hash Algorithm ("SHA") Or hashes may be generated by a checksum generator, engine, or algorithm, such as SHA-1 algorithms. Also disclosed herein are systems and methods for computer file systems, where pseudo-random values from input text of a non-linear probablistic mathematical algorithm or other data / numeric sequence. The hashes may be generated by a checksum program, engine, algorithm or other means that generates a probabilistic unique hash value for a block of data of indeterminate size based on any industry standard technique for generating the data.

The systems and methods of the present invention allow for potentially specific large amounts of unfactored storage devices to be reduced in size in several order of magnitude, thereby automatically redefining data redundancy in certain applications disclosed herein. Can be used to factor out. In this regard, the systems and methods of the present invention allow all computers to share data simply, efficiently, securely, and to provide uniquely advantageous means for reading, writing, or making reference to data, regardless of particular hardware or software characteristics. do. The systems and methods of the present invention are particularly effective for networked computers or computer systems, but can also be applied with comparable results to separate data storage devices.

The hash file system of the present invention advantageously solves a number of problems that present difficulties with conventional storage architectures. For example, the systems and methods of the present invention eliminate the need to manage large sets of directories and files along with copies, and system resources that inevitably occur with slightly different copies. Maintaining and storing copy files presents difficulties for traditional corporate and personal computer systems and generally requires human intervention in "clean up disk space". The hash file system of the present invention effectively eliminates this problem by removing the disk space used for copying and almost completely eliminating the disk space used for partial copying. For example, a traditional computer system that copies a gigabyte directory structure to a new location requires another gigabyte of storage. In certain applications, the hash file system of the present invention reduces the disk space used for this operation by up to 100,000 times.

Currently, some file systems have a mechanism for removing copies, but even if the system is extended, none of these operations can copy these operations in a short time (in technical terms, the system copies the copies at 0 (I) ("regular time order") time). Implying urea differentiation). This means that one unit of time is constant, as opposed to other systems that require O (N ^** 2), O (N), or O (log (N)) time, with time being factorized Means the amount of the device. Element differentiation of the storage device at an inconsistent time may only be satisfactory for a system with less storage, but as the system size increases, even an efficient system among non-uniform elemental differentiation systems is not satisfactory. The hash file system of the present invention is designed to tier the storage device on a scale that has never been attempted before, and in the first embodiment it can tier 2 million petabytes of storage with greater scalability. have. Existing file systems cannot manage data on this scale.

Moreover, the hash file system of the present invention can be used to provide inexpensive, worldwide computer system data protection and backup. Because computer file systems rarely change more than a few percent of their total storage between each backup operation, the factorization function works very efficiently for typical backup data sets. In addition, the hash file system of the present invention can be the basis of an efficient messaging (email) system. The email system is basically a data copying mechanism where the author fills out a message and sends it to the list's recipients. Email systems effectively implement "send" operations by copying data from one place to another. The author generally keeps copies of the messages he sent, and each recipient owns his own copies. These copies are conversely attached to the reply and also maintained (ie, copies of the copies). A feature of the common element differentiation of the present invention is that it allows email users to keep this familiar copy-oriented paradigm transparent while eliminating this huge inefficiency.

As is already well known, since most of the data of a computer system is hardly changed, the hash file system of the present invention is according to the needs of the system, for example, every hour or minute (or less) every day the system is present. At intervals, it allows for a complete snapshot of the entire system. In addition, since conventional computer systems often provide a limited version of files (ie, the VAX® VMS® file system from Digital Equipment Corporation), the hash file system of the present invention also provides significant advantages in this regard. Versioning in conventional systems exhibits both good and bad aspects. In the former example, it helps to prevent accidents, while the latter example requires regular removal to reduce the disk space consumed. The hash file system of the present invention provides versioning of files with little overhead by factoring the same or edited copies using very little extra space. For example, saving 100 revisions of a typical document requires about 100 times the original file space. Using the hash file system disclosed herein, these modifications would require only three times the original space (depending on the size of the document, the degree and type of editing, and external factors).

Still other potential applications of the hash file system of the present invention include web-serving. In this regard, a hash file system can be used to efficiently distribute web content because the method of hashing the common parts (hashing) also achieves a uniform distribution across all hash file system servers. This even distribution allows multiple arrays of servers to function as huge web server zones with an evenly distributed load. In other applications, the hash file system of the present invention can be used as a network accelerator as long as it can be used to reduce network traffic by sending a proxy (hash) to the data instead of the data itself. Many percentages of current network traffic are redundant data moving between locations. Sending a proxy for the data allows the local caching mechanism to work effectively, making it possible to significantly reduce traffic on the Internet.

In particular, as disclosed herein, the hash file system and method of the present invention may be implemented using a 160-bit hashsum as general purpose pointers. This is a conventional file system that uses pointers allocated from the central station (i.e. 32-bit "inode" in Unix are allocated by the kernel's file system in a lock-step operation that ensures uniqueness). Is different. In the hash file system of the present invention, these 160 bits of hash islands are allocated by a hashing algorithm without a central station (ie, without locking, without synchronization).

Known hashing algorithms produce stochasticly unique numbers that span the range of values uniformly. In the case of the hash function SHA-1, the range is between 0 and 10e ⁴⁸ . This hashing operation is performed by examining only the contents of the stored data, and thus can be completely separated and asynchronously performed without interlocking.

Hashing is an operation that can be identified by any component of the system and eliminates the need for trusted operations between the components. Hence, the hash file system and method of the present invention disclosed herein serves to eliminate the critical bottleneck of the traditional large scale distributed file system, that is, the trusted encompassing central authority. This allows the construction of large scale distributed file systems that can operate without the risk of inconsistencies and without any bottleneck limitations in any prior art.

1 is a high level explanatory diagram of an exemplary networked computer environment in which the systems and methods of the present invention may be implemented.

FIG. 2 is a more detailed conceptual diagram of a possible operating environment for using the system and method of the present invention, in which files maintained on any number of data centers or computers are arranged, for example, in geographically separated locations. A conceptual diagram that can be stored within a distributed computer system via an Internet connection to a Redundant Array of Independent Nodes (RAIN) racks.

FIG. 3 is a logical flow diagram illustrating the steps in the entry of a computer file into the hash file system of the present invention, in contrast to hash values for files in which a hash value for a file has been previously maintained in a set or database. Flowchart checked.

4 is a logical flow diagram illustrating the steps of dividing a file or other data sequence into hashed fragments to produce multiple data fragments as well as corresponding stochastic unique hash values for each fragment.

5 compares hash values for each fragment of a file with existing hash values in the set (or database), the equivalent of a single hash value for all file fragments with hash values of several fragments. And other logical flow diagrams illustrating the creation of records representing a new data fragments and corresponding new hash values added to the set.

Figure 6 is another logical flow diagram illustrating the comparison of file hash or directory list hash values with existing directory list hash values and adding new file or directory list hash values to the set directory list.

7 illustrates a comparison of fragments of a representative computer file with corresponding hash values before and after editing a particular fragment of an example file.

8 illustrates the results of a function using data represented by hashes or concatenation of data represented by synthetic data derivable by the systems and methods of the present invention, but clearly represented by the corresponding hashes. A conceptual diagram showing the fact that it is identical to the data that can instead be generated by a recipe.

FIG. 9 is another conceptual diagram illustrating how the hash file system and method of the present invention is used to organize data to optimize reuse of redundant sequences using hash values for the data they represent as pointers. A conceptual diagram that can be represented as an explicit byte sequence or composites of sequences.

10 is a simplified diagram illustrating a hash file system address translation function for an example 160 bit hash value.

11 is a simplified diagram of an index stripe splitting function for use in the systems and methods of the present invention.

12 is a simplified diagram of the overall functionality of the system and method of the present invention for use in backing up data for a representative home computer having multiple program and document files on Day 1, wherein one of the document files is a third document. Drawing edited on Day 2 with the addition of a file.

FIG. 13 shows a comparison of several fragments of a particular document file marked by a number of "sticky bytes" before and after editing, where one of the fragments is changed while the other fragments remain the same. drawing.

The foregoing and other features and objects of the present invention and the manner in which they are made will become more apparent and better understood by reference to the following description of the preferred embodiments in conjunction with the accompanying drawings.

In a particular embodiment of the hash file system of the present invention as disclosed herein, the application is a high utilization and high reliability data storage system for the rapid advances of commercial computer devices and the robust nature of internetworking technologies such as the Internet. It is about. In particular, the specification includes one or more data blocks (including one or more code (s) for their data blocks (including but not limited to files, directories, drive images, software applications, digitized voice, and rich media content). (B) a hash file system is disclosed that manages the correspondence of the < RTI ID = 0.0 > (x) < / RTI > Or another identifier. The system itself is not limited to personal computers, supercomputers, distributed or non-distributed networks, storage area networks ("SAN") using network buses (IDEs), SCSI, or other disk buses, network-attached storage ("NAS"). Or other computer system including other systems capable of storing and / or processing data.

In certain embodiments of the hash file system disclosed herein, the code (s) may include one or more hash or checksum generation engines, programs, including, but not limited to, MD4, MD5, SHA, SHA-1, or derivatives thereof. Or using an algorithm. In addition, the code (s) may comprise a hash or checksum generation engine, program, or algorithm, including, but not limited to, MD4, MD5, SHA, SHA-1, or other ways of generating stochastic unique identifiers based on data content. It may include portions of variable or fixed length codes derived using. In certain embodiments disclosed herein, a file search or lookup for retrieving data or checking for the presence / availability of the data may include routing information for searching, retrieving, or checking for sign representation or for the presence / availability of the data. Can be accelerated by searching in all or a smaller portion of the code.

Disclosed herein is a system and method for a hash file system, where a sign enables identification of a copy in a system that overlaps with data provided to the system for identification and / or storage of redundant copies in the system. This code allows for a redundant copy of the data and / or the screening of data portions within the system or data and / or portions of data provided to the system without loss of data integrity, so that the data is evenly distributed over the available storage capacity for the system. Can be dispersed. The systems and methods of the present invention as disclosed herein do not require a central operating point and process and / or input / output through any computer, supercomputer or other device capable of storing and processing data provided to the system. ("I / O") Balance the load. Redundant copies of data and / or portions of data provided herein include the generation, iterative generation of intelligent boundaries to screen other data in the system, future data provided to the system, or future data stored by the system. , Or enable maintenance.

The invention is illustrated and described in terms of a distributed computing environment such as an enterprise computing system using public communication channels such as the Internet. However, an important feature of the present invention is that it is easily scaled up and down to meet the needs of a particular application. Thus, unless otherwise specified, the present invention is applicable to very large and complex network environments as well as small network environments such as conventional LAN systems.

Referring to FIG. 1, the present invention can be used in connection with a novel data storage system on the network 10. In this figure, the exemplary internetwork environment 10 may include the Internet including a global internetwork formed by logical and physical connections between multiple WANs 14 and LANs 16. The internet backbone 12 is represented by routers carrying main lines and data traffic. The backbone 12 is formed by the largest network in a system operated by major Internet service providers (“ISPs”), such as, for example, GTE, MCI, Sprint, UUNet, and America Online. Although single connection lines have been used for convenience to represent the connection of the WANs 14 and LANs 16 to the Internet backbone 12, in practice multiple paths between multiple WANs 14 and LANs 16 are used. It will be appreciated that there are routable physical connections. This makes the internetwork 10 robust when faced with single or multiple failures.

It is important to distinguish network connections from internal data paths implemented between peripherals in a computer. “Network” includes a general purpose system that includes primarily switched physical connections, which enables logical connections between processes operating on nodes 18. The physical connections implemented by the network are typically independent of the logical connections established between the processes using the network. In this way, a hybrid set of processes in the range of file transfers, mail transfers, and the like can use the same physical network. Conversely, a network can be formed from a hybrid set of physical network technologies that are not visible to processes that are logically connected using the network. Since the logical connection between the processes implemented by the network is independent of the physical connection, the internetwork can easily be extended to a virtually infinite number of nodes over a long distance.

In contrast, internal data paths such as system buses, peripheral interconnect ("PCI") buses, intelligent drive electronics ("IDE") buses, and small computer system interface ("SCSI") buses, may be used for special purpose purposes within a computer system. Defines the physical connections that implement the connection. These connections implement physical connections between physical devices as opposed to logical connections between processes. These physical connections are characterized by limited distances between components, a limited number of devices that can be coupled to the connection, and a limited format of devices that can be connected via the connection.

In a particular implementation of the invention, the storage device may be located at nodes 18. The storage of any node 18 may include a single hard drive or may include a managed storage system, such as a conventional RAID device having multiple hard drives organized into a single logical volume. Importantly, the present invention manages redundant operations between nodes so that the designated configuration of storage within any given node is less involved than in nodes.

Optionally, one or more nodes 18 may implement storage allocation management (“SAM”) processes that manage data storage through nodes 18 in a distributed collaborative manner. Preferably, the SAM processes have little or no centralized control over the entire system. SAM processes provide data distribution between nodes 18 to implement recovery in the manner of fault tolerance between network nodes 18 in the same manner as the paradigm found in RAID storage subsystems.

However, because SAM processes operate through nodes rather than on a single node or on a single computer, they are capable of better fault tolerance and a higher level of storage efficiency than conventional RAID systems. For example, SAM processes may recover even when network node 18, LAN 16, or WAN 14 are not available. Moreover, even if a portion of the Internet backbone 12 is not available due to errors or congestion, the SAM process may recover using data distributed on the nodes 18 that remain accessible. In this way, the present invention strengthens the properties of the internetwork, providing unprecedented usefulness, reliability, fault tolerance and robustness.

2, a more detailed concept of an exemplary network computing environment in which the present invention is implemented is shown. The internetwork 10 of the preceding figures (or the Internet 118 of the figures) may be computing devices and mechanisms 102 ranging from a supercomputer or data center 104 to a handheld or pen based device 114. It allows for a hybrid set of interconnected networks 100. While these devices inherently require other data storage devices, they share the ability to retrieve data through the network 100 and operate that data within their own resources. Personal computer or workstation-class devices such as IBM compatible device 108, Macintosh device 110, and laptop computer 112, as well as mainframe computers (e.g., VAX station 106 and IBM AS / 400 station ( Heterogeneous computing devices including 116) are easily interconnected via the internetwork 10 and the network 100. FIG. Although not shown, mobile and other wireless devices may be connected to the internetwork 10.

Internet-based network 120 includes a set of logical connections between a plurality of internal networks 122, some of which are made through the Internet 118. Conceptually, the Internet-based network 120 is similar to the WAN 14 (FIG. 1) in that it enables logical connections between geographically distant nodes. Internet-based network 120 may be implemented using the Internet 118 or other public and private WAN technologies, including leased lines, fiber optic channels, and the like.

Similarly, internal network 122 is conceptually similar to LAN 16 in that it allows for a logical connection over a more limited distance than WAN 14. Internal networks 122 may be implemented using various LAN technologies, including Ethernet, optical distribution data interface (“FDDI”), token ring, AppleTalk, fiber channel, and the like.

Each internal network 122 connects one or more Redundant Arrays of Independent Nodes (RAIN) components 124 to implement RAIN nodes 18 (see FIG. 1). Each RAIN component 124 includes one or more mass storage devices such as a processor, memory, and hard disk. RAIN components 124 also include hard disk controllers, which can be a management controller such as a conventional IDE or SCSI controller, or a RAID controller. RAIN component 124 may be physically distributed or co-located in one or more racks that share resources such as cooling and power. The failure or failure of one node 18 does not affect the utilization of the other nodes 18, in that data stored on one node 18 can be reconstructed from data stored on other nodes 18. Each node 18 is independent of the other nodes 18.

In one particular embodiment, the RAIN components 124 are 256-byte random access memory ("RAM") mounted in a conventional AT or ATX case and an Intel-based microcomputer mounted on a mother mode that supports a PCI bus. It may include computers using commercially available components such as processors. The SCSI or IDE controller can be implemented by an expansion card connected to the motherboard or to the PCI bus. Although these controllers are only implemented on the motherboard, a PCI expansion bus can optionally be used. In certain embodiments, the motherboard is a PCI expansion card and two PCI expansion cards used to implement two additional mastering EIDE channels, such that each RAIN component 124 includes up to four or more EIDE hard disks. It may include an EIDE channel. In certain embodiments, each hard disk may comprise 80 gigabyte hard disks for a total storage capacity of 320 gigabytes or more per RAIN component. Hard disk capacity and configuration within the RAIN components 124 can be easily increased or decreased to meet the needs of a particular application. The case also accommodates support mechanisms such as power and cooling devices (not shown).

Each RAIN component 124 runs an operating system. In certain embodiments, UNIX variant operating systems such as UNIX or Linux may be used. However, other operating systems such as DOS, Microsoft Windows, Apple Macintosh OS, OS / 2, Microsoft Windows NT, etc. are equally replaceable, but performance changes are expected. The selected operating system forms a platform for executing application software and processes, and implements a file system for accessing mass storage devices via hard disk controller (s). Various application software and processes are implemented on each RAIN component 124 to provide network interface using appropriate network protocols such as User Datagram Protocol ("UDP"), Transmission Control Protocol (TCP), Internet Protocol (IP), and the like. Can provide network connectivity.

Referring to Figure 3, a logical flow diagram illustrating steps of entering a computer file into the hash file system of the present invention, wherein the hash value for the file is checked for hash values for files previously held in the set or database. The logical sequence that is shown is shown.

Process 200 is initiated by input of computer file data 202 (eg, "File A") into the hash file system ("HFS") of the present invention, and a hash function is performed in step 204. Next, at decision step 208, data 206 representing the hash of File A is compared with the contents of the set containing the hash file values. If data 206 representing the hash of File A already exists in the set, then the hash value of the file is added to the directory list in step 210. The contents of the set 212 including the hash values and corresponding data are provided in the form of existing hash values 214 for the comparison operation of decision step 208. On the other hand, if the hash value for File A does not currently exist in the set, then the file is split into hashed fragments in step 216 (which will be described in more detail below).

Referring to FIG. 4, a logic flow diagram is shown that illustrates the steps of process 300 for dividing a digital sequence (eg, a file or other data sequence) into hashed fragments. This process 300 ultimately leads to the generation of multiple data fragments as well as corresponding stochastic uniquely hash values for each fragment.

File data 302 is divided into fragments at step 304 based on the commonality with other fragments in the system or the likelihood that future fragments will be found to be common. The result of the operation of step 304 on the file data 302 is the generation of five file fragments 306 named A1 through A5 in the representative example shown.

Each of the file fragments 306 is then computed by locating it through a separate hash function operation in step 308, assigning a probabilistic unique number to each of the fragments 306 (A1 through A5). The result of the operation in step 308 is that each of the fragments 306 (A1 through A5) has an associated probabilistic unique hash value 310 (shown as A1 hashes to A5 hashes, respectively). The file splitting process of step 304 will also be described in greater detail below in conjunction with the only " sticky bite " operation disclosed herein.

Referring to FIG. 5, another logic representing the comparison process 400 of the hash values 310 of each fragment 306 of the file and the existing hash values 214 maintained in the set 212. Flowchart is shown. In particular, at step 402, the hash values 310 for each fragment 306 of the file are compared with existing hash values 214, such that new hash values 408 and corresponding new data fragments ( 406 is added to the set 212. In this manner, hash values 408 that were not previously provided in database set 212 are added along with their associated data fragments 406. Process 400 also results in the creation of record 404 showing the equivalent of a single hash value for all file fragments along with hash values 310 of various fragments 306.

Referring to Figure 6, another logical flow diagram illustrating a process 500 for comparing file hash or directory list hash values to existing directory list hash values and adding new file or directory list hash values to the database directory list. It is. Process 500 is stored on stored data 502 including a list of file names accumulated in the directory, file meta-data (eg, date, time, file length, file type, etc.), and a hash value of the file for each item. It works. In step 504, a hash function is executed on the contents of the directory list. The decision step 506 is operative to determine whether a hash value for the directory listing exists within the set of existing hash values 214. If present, process 500 reverts to adding another file hash or directory list hash to the directory list. Alternatively, if a hash value for the directory list does not yet exist in the database set 212, the data and hash value for the directory list are added to the database set 212 at step 508.

Referring to FIG. 7, a comparison of the corresponding hash values 310 with fragments 306 of a representative computer file (ie, “File A”) is shown before and after editing a particular fragment of an example file. . In this example, the record 404 includes the hash values 310 of each of the fragments A1-A5 of the file as well as the hash value of File A. The edited representation of File A changes with the data for fragment A2 (hereinafter referred to as " A2-b ") in file fragments 306A and the corresponding hash value A2-b of hash values 310A. Can cause a change. The edited file fragment produces an updated record 404A that includes the modified hash value of File A and the modified hash value of fragment A2-b.

Referring to FIG. 8, synthetic data (synthetic data 702 and 704) derived by the systems and methods of the present invention are clearly represented, but instead are data 706 generated by a “recipe” or formula. A conceptual representation 700 is shown that represents the fact that they are substantially the same. In the example shown, this preparation includes the concatenation of the data represented by the corresponding hashes 708, or the result of a function using the data represented by the hashes. As shown, data blocks 706 have a variable length and hash values 708 are derived from their associated data blocks. As mentioned above, hash values 708 are probabilistic unique identifiers of corresponding data fragments, but indeed unique identifiers may be used instead or together. In addition, the composite data 702, 704 can refer to other composite data of multiple levels of depth and the hash values 708 for the composite data can be derived from the data value the recipe produces or the hash value of the recipe. have.

Referring to FIG. 9, how the hash file system and method of the present invention are used to construct data 802 to optimize reuse of redundant sequences using hash values 806 for the data they represent as pointers. Another conceptual representation 800 is shown, where data 802 may be represented as an explicit byte sequence 808 or a group of sequences 804.

Reference numeral 800 shows a vast common part of the manufacturing process and the data to be reused at each level. The basic structure of the hash file system of the present invention is mainly a "tree" or "bush" structure, where hash values 806 are used instead of conventional pointers. Hash values 806 are used to point out other hash values in the manufacturing process that can be data or the recipe itself. In essence, the recipe indicates other recipes that ultimately dictate some specific data that may be themselves, and dictates other recipes that point to more data, resulting in the non-data.

Referring to FIG. 10, a simplified diagram illustrating a hash file system address resolution function for an example 160 bit hash value 902 is shown. Hash value 902 includes a data structure that includes a front end 904 and a rear end 906 as shown, and diagram 900 enables the use of hash value 902 to include corresponding data. Describes a particular "0 or 1" operation used to advance to the location of a particular node in the system.

Reference numeral 900 indicates how the front end 904 of the hash value 902 data structure can be used to indicate the hash prefix as a stripe identifier (" ID ") 908, i. It is described whether the class can be used to map to IP address 910. In this example, " S2 " indicates a stripe (2) 912 of the index node 37. Index stripe 912 at node 37 indicates stripe 88 at data node 73, indicated by reference numeral 914. In a later operation, one portion of its own hash value 902 may be used to indicate which node in the system contains related data, and another portion of hash value 902 may be used at a particular node. May be used to indicate a stripe of data, and another portion of hash value 902 may be used to indicate the location of data within that stripe. Through this three-step process, it can be quickly determined whether the data represented by hash value 902 has previously existed in the system.

Referring to FIG. 11, FIG. 11 illustrates a simple example of an index stripe split function for use in the systems and methods of the present invention. In this description, an example function 1000 is shown that can be used to effectively divide stripe 1002 (S2) into two stripes 1004 (S2) and 1006 (S7), where one stripe should not be too full. do. In this example, the odd entry has been moved to stripe 1006 (S7), and the even entry has kept stripe 1004. Function 1000 is an example of how stripe entries can be handled as the overall system size grows and becomes complex.

12, the overall functionality of the system and method of the present invention for use in the backup of data for a representative home computer having multiple program and document files 1102A and 1104A on Day 1 is shown briefly, Here one of the document files 1104B is edited on Day 2 with the addition of a third document file (Z.doc) (Y.doc) while the program files 1102B remain the same on Day 2.

Details of how a computer file system can be divided into fragments to reconstruct the original data from the fragments and listed as a series of recipes on the Global Data Protection Network (“gDPN”) are shown in Description 1100. . A very small computer system is shown in the form of a "snapshot" in "Day 1" followed by "Day 2". In "Day 1", "Program File H5" and "My Document H6" are denoted by reference numeral 1106 and the former is represented by Formula 1108, the first executable file is represented by hash value H1 1114, The second executable file is represented by hash value H2 1112. The document file is represented by hash value H6 1110, the first document is represented by hash value H3 1118, and the second document is represented by hash value H4 1116. Thereafter, in "Day 2", "program file H5" and "my document H10" indicated by reference numeral 1120 indicate that "program file H5" has not changed, but "My document H10" has changed. H10 denoted by reference numeral 1122 indicates that "X.doc" is still represented by hash value H3 while "Y.doc" is currently represented by hash value H8 of reference numeral 1124. The new document file "Z.doc" is represented by the hash value H9 at reference numeral 1126.

In this example, we can see that some files have changed on Day 2, but others have not. In the changed files, some of their fragments were not changed and others were changed. Through the use of the hash file system of the present invention, a "snapshot" of the computer system can be made on Day 1 (since they exist, creating a recipe that is essential for the reconstruction of computer files) and on Day 2 some of the recipes of the previous date This is done by reusing it, rebuilding other things, and then adding new things that describe the system. In this manner, the computer system can be regenerated at any time on Day 1 or Day 2 as well as on any subsequent dates.

Referring to FIG. 13, there is shown a comparison of several fragments of a particular document file marked by a number of “sticky bytes” 1204 before (Day 1 1202A) and after (Day 2 1202B) where fragments are shown. One of them changes while the other fragments remain the same.

For example, at Day 1, file 1202A includes variable length fragments 1206 (1.1), 1208 (1.2), 1210 (2.1), 1212 (2.), 1214 (2.3) and 1216 (3.1). On Day 2, fragments 1206, 1208, 1210, 1214, and 1216 remain the same (and therefore have the same hash value), but fragment 1212 was edited to produce fragment 1212A (thus having different hash values). .

Data sticky Byes (or "sticky points") are unique and sub-divide computer files so that common components can be browsed on computers that are not associated with multiple related computers without communication between computers. In a completely automated way. The meaning of searching for data sticky points is essentially completely mathematical and is performed the same regardless of the data content of the files. In the hash file system of the present invention, all data objects can be indexed, stored and retrieved using, for example, industry standard checksums such as, but not limited to, MD4, MD5, SHA, or SHA-1. If two files have the same checksum at the time of operation, it can be considered that they are most likely the same file. In the systems and methods disclosed herein, data sticky points can be generated with standard deviations that are a small percentage of the mathematical standard variance and the target size.

The data sticky point is an array of statistically rarely occurring bytes. In this case, an example of 32 bytes is provided because of ease of implementation in current microprocessor technology.

A 32 bit rolling hash can be generated for file "f".

// f [i] = is the ith byte of the file "f".

// scrambel is a 256 entry array of integers with each // being 32 bits wide;

// those integers are typically chosen to uniformly // span the range.

int t = 8 // target number of trailing zeros

int hash = 0;

int sticky_bits;

for (int i = O; i <filesize; i ++)

hash = hash >> 1┃scarmble [f [i]];

// At every byte in the file, hash represents the // rolling hash of the file.

sticky_bits = (hash-1) ^ hash;

// sticks_bits is a variable which will have the // number of ones in the hash

// that correspond to the number of trailing zergo in // the "hash".

number_of_bits = count_ones (stick_bits);

if (number_of_bits> t)

      output_sticky_point (i);

}

The sticky point is defined as being a rolling hash with a number of trailing zeros as a target number with a hash expressed in binary. Statistically speaking, this algorithm will find points spaced 2 ^ t, where t is the target number of trailing zeros. In this example, t = 8, we will average the sticky points spaced 2 ^ 8 = 256 bytes apart.

A 32-bit rolling hash can be generated for f file at

f [i] = is the ith byte of the file f.

scramble is a 256 entry array of random elements with each being n

bits wide;

int t = 8 // target number of trailing zeros

int target_distance = 256; // 2 to the power of 8

int hash = 0;

int sticky_bits;

int distance = 0;

int last_point = 0;

for (int i = O; i <filesize; i ++) {

       hash = hash >> 1┃scarmble [f [i]];

       // At every byte in the file hash represents the // rolling hash of the file.

       sticky_bits = (hash-1) ^ hash;

       // sticks_bits is a variable which will have the // number of ones that correspond to the number of // trailing zergo in the "hash".

       number_of_bits = count_ones (stick_bits);                 

       distance = i-last_point;

if (number_of_bits ^* distance / target_distance> t)

       last_point = i;

output_sticky_point (i);

       }

}

The hash functions used to implement the hash file system of the present invention require moderately complex operations, but this is well within the capabilities of today's computer systems. Hashing functions are inherently probabilistic and can produce inaccurate results if two different data objects have the same hash value. However, the systems and methods disclosed herein have been well known and studied to reduce the likelihood of a collision to an acceptable level (i.e. trillion units) for reliable use, with much lower error rates allowed in conventional computer hardware operation. The above problem is solved by hashing functions.

The term "Internet infrastructure" as used herein refers to a variety of hardware and software mechanisms, but the term basically refers to routers, router software, and routers that function to transfer data packets from one network node to another network node. Refers to the physical link between them. As used herein, “digital sequence” includes, but is not limited to, computer program files, computer applications, data files, network packets, streaming data such as multimedia (including audio and video), data by telemetry, and digital Or any other form of data that can be represented by a sequence of numbers. Probabilistic unique identifiers generated by the hash file system and method of the present invention may also be used as URLs in network applications.

While the principles of the invention have been described above in conjunction with specific operations and system configurations, it will be clearly understood that the foregoing description is illustrative only and is not intended to limit the scope of the invention. In particular, it is appreciated that one of ordinary skill in the art would understand the foregoing disclosure and suggest other modifications. Such modifications may inherently include other features that may be used in place of or in addition to those already known and already disclosed herein. Although the claims have been formulated in this application for specific combinations of features, mitigation of any or all of the same technical problems as faced by the present invention, whether or not related to the same invention as claimed in any claim. Without departing from the scope of the present specification, any novel feature or combination of any novel feature, clearly or potentially disclosed, or any generalization or modification apparent to those skilled in the art. Applicant reserves the right to prepare new claims for such features and combinations of features during the continuation of this application or any further application derived therefrom.

Claims (85)

delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete At least one list for maintaining portions of digital sequences and corresponding probabilistic unique identifiers for each of the portions of digital sequences; And At least one process unit, The at least one process unit, Receive at least one new digital sequence; Split the new digital sequence into a plurality of shorter digital sequences, and perform at least one partitioning mechanism for generating a stochastic unique identifier for each of the shorter digital sequences; And perform a comparison mechanism to determine if any one of the probabilistic unique identifiers for each of the plurality of shorter digital sequences is currently maintained in the list. delete delete delete delete delete delete delete 27. The computing system of claim 25, wherein the at least one process unit is configured to perform the at least one partitioning mechanism, and the at least one list is connected by a network. 34. The computing system of claim 33 wherein said network comprises a public network, such as the Internet. 35. The computing system of claim 34 wherein said at least one partitioning mechanism and said at least one list are physically distributed. 27. The computing system of claim 25 wherein the probabilistic unique identifiers are generated by a hash function. 37. The computing system of claim 36 wherein said hash function comprises an industry standard digest algorithm. 38. The computing system of claim 37 wherein said hash function comprises an MD4, MD5, SHA or SHA-1 algorithm. 37. The computing system of claim 36 wherein the probabilistic unique identifiers are generated by a checksum. 26. The computing system of claim 25 wherein said digital sequences are of variable length. 26. The computing system of claim 25 wherein said digital sequences are of fixed length. 26. The computing system of claim 25 wherein the comparison mechanism is operative to use at least one portion of the probabilistic unique identifiers for each of the plurality of shorter digital sequences as a locator correlated with the list partitions. The method of claim 25, wherein the digital sequence, A feature comprising a data file; A feature comprising a data stream; A feature comprising an executable file; A feature comprising a database record; A feature comprising a database index; A feature comprising a digital device image; A feature comprising a network packet; or Features include digitized analog signals Computing system having at least one of the. delete delete delete delete delete delete delete 27. The system of claim 25, wherein at least one of the probabilistic unique identifiers and corresponding ones of the plurality of shorter digital sequences that are not determined to be maintained within the at least one list are added to the at least one list. Computing system. delete delete delete delete delete delete delete delete delete delete delete delete delete delete A method of setting an identifier for at least a portion of a digital sequence, the method comprising: Performing a hashing function on the at least one portion of the digital sequence to produce a probabilistic unique sign for the at least one portion of the digital sequence; Establishing a corresponding relationship between said at least one portion of said digital sequence and said probabilistic unique code; Using the probabilistic unique sign as the identifier Identifier setting method comprising a. 67. The method of claim 66 wherein the identifier and at least one portion of the digital sequence corresponding to the identifier is maintained in at least one data list. 68. The method of claim 67, wherein at least one portion of the identifier is available as a pointer to the location of at least one portion of the digital sequence corresponding to the identifier in the at least one data list. The method of claim 66, Said at least one portion of said digital sequence comprises at least one portion of a data file and said identifier points to a content of said at least one portion of said data file; Said at least one portion of said digital sequence comprises at least one portion of a data stream, said identifier indicating a content of said at least one portion of said data stream; or The at least one portion of the digital sequence includes at least one portion of an executable file, and wherein the identifier indicates the contents of the at least one portion of the executable file Method of setting an identifier having at least one of the features. delete delete delete 67. The method of claim 66, wherein performing the hashing function is performed by an Industry Standard Digest Algorithm. 74. The method of claim 73, wherein performing the hashing function is performed by one of an MD4, MD5, SHA, or SHA-1 algorithm. A recording medium having a computer readable program implemented therein for setting an identifier for at least a portion of a digital sequence, The computer readable program includes a plurality of computer readable codes, When the computer readable codes are executed on a computer, The computer performs a hashing function on the at least one portion of the digital sequence to produce a probabilistic unique sign; The computer establishes a correspondence between the at least one portion of the digital sequence and the probabilistic unique code; And the computer is configured to use the stochastic unique code as the identifier. 76. The recording medium of claim 75, wherein the identifier and at least a portion of the digital sequence corresponding to the identifier are maintained in at least one data list. 77. The computer program product of claim 76, wherein at least one portion of the identifier is usable as a pointer to a location of at least one portion of the digital sequence corresponding to the identifier in the at least one data list. One recording medium. 76. The method of claim 75, Said at least one portion of said digital sequence comprises at least one portion of a data file, said identifier indicating a content of said at least one portion of said data file; Said at least one portion of said digital sequence comprises at least one portion of a data stream, said identifier indicating a content of said at least one portion of said data stream; or Said at least one portion of said digital sequence comprises at least one portion of an executable file, said identifier indicating the contents of said at least one portion of said executable file A recording medium having a computer readable program comprising at least one of the following features. delete delete delete 76. The recording medium of claim 75, wherein the computer readable code configured to cause the computer to perform a hashing function is performed by an industry standard digest algorithm. 83. The computer program product of claim 82, wherein the computer readable code configured to cause the computer to perform a hashing function is performed by one of an MD4, MD5, SHA, or SHA-1 algorithm. Recording media. delete delete

Images