JP2013536514A - Method and system for extending data storage system functionality - Google Patents

Abstract

A method and system for extending the function of a data storage system having data organization means and data storage resources, the method separating the data storage function of the data storage system from the other functions of the data storage system. And transferring at least part of the data storage function to the intercept system.
[Selection] Figure 3

Description

  The present invention relates generally to data storage systems, and more particularly to a method and system for extending the functionality of a data storage system.

  This application claims priority from US Provisional Application No. 61 / 376,905, filed Aug. 25, 2010, the entire contents of which are hereby incorporated by reference.

  The file system on the computer allows the computer operating system to act as a trusted third party that enforces security and naming protocols between communicating processes, even if these processes are not active at the same time. It becomes possible to operate. One of the functions of the file system is to classify and organize the supplied data so that it can be retrieved later. In order to perform this function, the file system must manage storage resources. Normal storage resources appear in the file system as a continuous byte sequence, a continuous block sequence, or as a key value store in an object storage system.

  The file system is usually part of the operating system on the computer, but may exist as an extension of the operating system in the application or as a pure application that is accessed over a network connection using the client file system protocol. . Various examples of these are known in the art. In either case, the file system is designed to use a particular type of storage resource, such as a disk, and is subject to the constraints associated with that storage resource. These constraints relate to, for example, performance, capacity, parallelism, scalability, physical location, etc. Furthermore, the file system has a function for applying operations such as encryption, compression, duplication, or other conversion including context-dependent processing (data polymorphism) to stored data in advance. If not, there is a restriction in terms of the function of the file system itself. In this technical field, the term context-dependent generally refers to a program characteristic that changes depending on the operation being performed in the program. For example, contextuality facilitates documentation for a particular property in use, and in contextual or contextual processing of data, different processing is performed on the data depending on how the data is used or where it is used. Can be.

  Prior systems and methods that address some of these limitations have used virtual devices that create virtual storage resources used by the file system. Due to the lack of available context that exists in the file system and is not passed to the storage resource, this approach cannot provide context-sensitive processing of the data and does not actually It is not possible to remove the restriction caused by the data mapping performed by the file system with respect to the data existing in the file, for example, the restriction on the total data storage capacity.

  Accordingly, it is an object of the present invention to provide a novel system and method for extending the functionality of a data storage system, for example, to extend the functionality of a data storage system to perform context-sensitive data processing. It is.

  According to an embodiment of the present invention, a method for extending the function of a data storage system having data organization means and data storage resources, wherein the data storage function of the data storage system is separated from other functions of the data storage system. A method is provided comprising the steps of: transferring at least a portion of the data storage functionality to an intercept system.

  According to one aspect of the invention, the intercept system is a complementary storage resource. The data organization means may be selected from the group consisting of a file system, a key value store, and a database.

  According to another aspect of the invention, the separating step is performed by providing data organization means, data storage resources, and intercept means for communicating with the intercept system, the intercept means intercepts file system operations, Determine whether the function of the operation should be handled by the data storage resource or by the intercept system.

  According to another aspect of the present invention, the intercepting means can operate the file system while the operation holds the context and before the operation is decomposed into a plurality of independent operations suitable for the data storage resource. To intercept.

  According to another aspect of the invention, at least another portion of the data storage function is maintained in the data storage system. According to another aspect of the present invention, all of the data storage functionality is transferred to the intercept system.

  According to another aspect of the invention, the data storage system is selected from the group consisting of a file system, a key value store, an object store, and a network protocol.

  According to another aspect of the invention, the data storage system is shared between multiple external interfaces.

  According to another aspect of the invention, the intercepting means includes a user space application program that cooperates with an operating system facility or a file system facility. According to another aspect of the invention, the intercepting means includes a file system protocol proxy application running on the network. According to another aspect of the invention, the intercepting means includes a mini-filter driver that intercepts file system operations in the operating system kernel.

  According to another aspect of the invention, the intercept system uses one or more complementary storage resources to perform its function, and the one or more complementary storage resources are independent of the data storage resources. .

  According to another aspect of the invention, the intercept system performs capacity expansion of the data storage system. According to another aspect of the invention, the intercept system improves the performance of the data storage system by changing one or more characteristics of the data on the data storage resource. The one or more characteristics are preferably selected from the group consisting of storage format, storage location, and storage order.

  According to another aspect of the invention, the method further comprises performing de-duplication by the intercept system. According to another aspect of the invention, the method further comprises performing data polymorphism with an intercept system. According to another aspect of the invention, the method further comprises performing an independent access control mechanism for data by the intercept system. According to another aspect of the invention, the method further comprises the step of versioning the data with an intercept system.

  According to another aspect of the invention, the method further comprises performing caching of the data storage system either single-level or multi-level, wherein the performing step is performed by an intercept means. According to another aspect of the invention, performing locality optimization by pushing less frequently used data to a remote data storage system and pulling frequently used data to nearby data storage systems The step of executing and executing is performed by intercept means. Remote data storage systems include data storage systems that are either physically remote or take a long time to access.

  In accordance with another aspect of the present invention, the method provides name-based virtualization that makes file names that are not valid in current data storage systems appear to be valid by referencing data on other data storage systems. The step of executing is executed by the intercept means.

  According to another aspect of the invention, the method further comprises performing data backup and data replication, the performing step being performed by an intercepting means.

  According to another aspect of the present invention, the method further comprises performing data virtualization including allowing data to be physically located in different data storage systems with the same name, and performing the steps comprises: Performed by intercept means.

  According to another aspect of the invention, the method further comprises providing a unique function for selected data in an intercept system rather than a data storage resource, the unique function comprising performance characteristics, deduplication, data polymorphism, independent Selected from the group consisting of access control mechanisms, versioning, caching, locality, replication, and data virtualization. The selection data is identified using a selection mechanism that employs metadata pattern matching with one or more selected from the group consisting of name, timestamp, size, history information, physical information, and context information.

  According to another embodiment of the present invention, a system for extending the function of a data storage system having data organization means and data storage resources, wherein the data storage function of the data storage system is separated from other functions of the data storage system. A system is provided comprising separation means and means for migrating at least part of the data storage function to an intercept system.

  According to another aspect of this embodiment, the intercept system may be a complementary storage resource.

  According to another aspect of this embodiment, the separating means includes a data organization means, a data storage resource, and an interceptor that communicates with an intercept system, the interceptor intercepts file system operations, Determine whether it should be handled by a data storage resource or by an intercept system.

  According to another aspect of this embodiment, the interceptor performs file system operations while the operations hold context and before the operations are broken down into multiple independent operations suitable for data storage resources. To intercept.

  According to another aspect of the present embodiment, at least another part of the data storage function is maintained in the data storage system. Alternatively, all of the data storage function is transferred to the intercept system.

  According to another aspect of this embodiment, the data storage system is selected from the group consisting of a file system, a key value store, an object store, and a network protocol.

  According to another aspect of the present embodiment, the data storage system is shared among a plurality of external interfaces.

  According to another aspect of this embodiment, the interceptor includes a user space application program that cooperates with an operating system facility or a file system facility.

  According to another aspect of this embodiment, the interceptor includes a file system protocol proxy application that runs on the network.

  According to another aspect of this embodiment, the interceptor includes a mini-filter driver that intercepts file system operations in the operating system kernel.

  According to another aspect of this embodiment, the intercept system uses one or more complementary storage resources to perform its functions, and the complementary storage resources are independent of the data storage resources.

  According to another aspect of this embodiment, the intercept system performs capacity expansion of the data storage system.

  According to another aspect of the present embodiment, the intercept system improves the performance of the data storage system by changing one or more characteristics of the data on the data storage resource.

  According to another aspect of this embodiment, the one or more characteristics are selected from the group consisting of a storage format, a storage location, and a storage order.

  According to another aspect of this embodiment, the intercept system performs data deduplication.

  According to another aspect of this embodiment, the intercept system performs data polymorphism.

  According to another aspect of this embodiment, the intercept system performs an independent access control mechanism for data.

  According to another aspect of this embodiment, the intercept system performs data versioning.

  According to another aspect of this embodiment, the interceptor performs data storage system caching on a single level or multi-level basis.

  According to another aspect of this embodiment, the interceptor optimizes locality by pushing frequently used data to a remote data storage system and pulling frequently used data to nearby data storage systems. Execute.

  According to another aspect of this embodiment, the remote data storage system includes a data storage system that is either physically remote or takes time to access.

  According to another aspect of this embodiment, the interceptor provides name-based virtualization that makes file names that are not valid in current data storage systems appear to be valid by referring to data on other data storage systems. Execute.

  According to another aspect of the present embodiment, the interceptor performs one of data backup and data replication.

  According to another aspect of this embodiment, the interceptor performs data virtualization including allowing data to be physically located in different data storage systems with the same name.

  According to another aspect of this embodiment, the intercept system is selected from the group consisting of improved performance characteristics, deduplication, data polymorphism, independent access control mechanisms, versioning, caching, locality, replication, and data virtualization. Run one or more.

  According to another aspect of the present embodiment, the data organization means is selected from the group consisting of a file system, a key value store, and a database.

  Embodiments will now be described by way of example only with reference to the accompanying drawings.

1 shows a high level architecture of a system according to the present invention.

Fig. 3 illustrates a computer system that may use and / or implement the present invention.

1 shows an embodiment of a method according to the invention.

  The present invention provides novel systems and methods for changing any constraints associated with data, including but not limited to constraints that can be changed using virtual storage resources. The present invention provides a data storage system other than a file system that provides a higher level of abstraction than the actual storage resource as a primary interface, such as databases and specific object databases, key value stores, predetermined network protocols, and predetermined The shared data system can be provided with this function.

  Furthermore, in the present invention, all the conventional restrictions of the data storage system can be changed without cooperation of the file system itself or without changing the file system itself. Therefore, in the present invention, it is not required to completely overhaul the new file system nor implement the new file system, and the existing system, the interface between the old and new systems, and the development without developing the new file system. A new system along the way will enable the benefits and enhancements described herein to be implemented. In general terms, the present invention attaches to the file system an intercept system that can selectively intercept file system operations, including contextual information, before being broken down into operations suitable for storage resources. That is, the file system has contextual operations and instructions for executing them supplied by the computer operating system. When the file system interacts with storage resources, these contextual operations and instructions are lost, but the interaction between the file system and storage resources is only about data retrieval, storage, and cataloging. Because. The intercept system according to the present invention provides a function required by the file system at an intercept point at which the operation of the intercept system is transparent to the file system. Therefore, the present invention separates the storage function of the data storage system from other functions (for example, file naming, lock, supply, security, etc.) and arbitrarily assigns the actual storage function to another component to Methods and systems for extending storage resources and system functionality are provided.

  Referring to FIG. 1, one embodiment of the present invention is shown, in which a data storage system 100, an interceptor 200, and an intercept system 300 are shown. Details of a preferred embodiment for separating the various functions distributed in a conventional data storage system from the data storage system itself and dividing it to be executed by the data storage system or intercept system are described below. Separation of functions and subsequent distribution between the data storage system 100 and the intercept system 300, in particular, distribution of functions between the data storage system 100 and the intercept system 300 in a non-cooperative environment is It is considered new in the technical field. However, although the preferred embodiment of the present invention relates to an add-on mechanism for an existing non-cooperative data storage system, it provides an architectural enhancement that makes the collaborative data storage system scalable and provides enhanced functionality. It may be a characteristic, examples of which are described in more detail below.

  Referring to FIG. 3, another embodiment of the present invention is shown, in which file system operations are intercepted 315 to extend the functionality of the data storage system, and the functionality of the operations depends on the data storage resources of the intercept system. Separating the data storage function 310 of the data storage system from other functions of the data storage system by determining whether it should be handled, and migrating 320 at least a portion of the data storage function to the intercept system. A method comprising: Alternatively, the intercepted data storage function persists 325 to the data storage resource.

  The present invention generally operates within the context of a computer system and allows for the expansion of data storage capabilities associated with the general computer system shown in the exemplary example of FIG. As shown, the computer system 20 includes a central processing unit (“CPU”) 24, a random access memory (“RAM”) 28, an input / output (“I / O”) interface 32, a network interface 36, a nonvolatile memory A number of physical and logical components including a storage 40 and a local bus 44 that allows the CPU 24 to communicate with other components. The CPU 24 executes an operating system and a number of software systems. The RAM 28 is a volatile storage that is relatively responsive to the CPU 24. The I / O interface 32 can receive an input from one or more devices such as a keyboard and a mouse, and outputs information to an output device such as a display and / or a speaker. The network interface 36 enables communication with other systems. The non-volatile storage 40 stores an operating system and a program. During operation of the computer system 20, programs and data are read from the non-volatile storage 40 and placed in the RAM 28 for easy execution.

  The data storage system 100 is generally known in the art, and for the purposes of the present invention, any machine, device, or apparatus that uses a predetermined identifier (eg, a file name) for data , And later, at least a portion of the data can be read based on the demand using the identifier. The data storage system 100 has data organization means 105, referred to herein as storage resources 110 (shown as non-volatile storage 40 in FIG. 2), optionally physically incorporated independent of operation. Data is stored in a primitive form according to the characteristics of the storage resource using the machine or module to be stored, but the data storage system can later read the stored data from the storage resource by itself. . The data storage system 100 may be shared among a plurality of external interfaces.

  Typical examples of the data organization means 105 include a file system, a key value store, a database, and other machines such as a web cache and a page file that are layered in combination on these. Typical examples of the storage resource 110 include a physical disk, a memory such as a RAM, a RAID array, a perforated tape, a document, and the like. In a representative example of a preferred embodiment, the storage system 100 may be a Windows® NTFS file system and the storage resource may be a hard disk drive.

  The intercept means 200 is a system that intercepts and processes data operations such as file system operations. In accordance with the present invention, intercept means 200 is more specific for storage resources that, while virtually all storage resources, are available for data operations but do not include contexts. Intercept operations before breaking them down into appropriate operations The context may be, for example, application information including the source of the data request, details about the data being requested, details about the intended use of the data, and other information that requires information about the current data operation.

  According to the representative example above, intercept means 200 is a mini-filter or legacy filter driver designed or configured to intercept file system operations at the appropriate level of the Windows operating system kernel. This driver may be mounted separately or a commercially available product may be procured. In other examples, the intercept mechanism is a file system stack using an application level file system provider, such as FUSE (User Space File System), built on other operating environments, a STREAMS driver for network interception, or transparent It can be provided by facilities such as, but not limited to, a physical proxy machine. The intercept means may be a custom designed software module that incorporates the functionality described herein. The intercepting means 200, also called interceptor, may be a file system protocol proxy application.

  The intercept unit 200 manages a contact point between the data storage system 100 and the intercept system 30. The intercept system 300 has an interest in certain operations of the data storage system. In a typical example, the predetermined operation may be any data input / output for redirecting data to another storage device. The intercept system 300 sets the intercept mechanism to pass all input / output operations to the intercept system 300 horizontally, instead of passing all input / output operations to a lower hierarchy in the data storage system 100, as conventionally performed.

  When receiving an intercepted operation, intercept system 300 has a number of options. (A) The intercept system 300 makes a decision not to process this operation by itself and instructs the intercept means 200 to continue as if there was no intercept, and the intercept means passes the operation to the lower hierarchy of the data storage system 100. Like that. (B) The intercept system 300 determines that the operation needs to be corrected before passing it to the lower tier of the data storage system 100. Or (c) The intercept system 300 determines that the operation should be handled by itself and takes over responsibility for the operation. In this case, intercept mechanism 200 must respond appropriately to the high and low tiers of the data storage system for this operation, including error handling.

  The intercept system 300 can perform a variety of tasks, including complementary storage resources, multimedia expansion devices, multimedia servers, home servers, application specific controllers, or tasks typically performed by storage resources. Any number of hardware devices such as systems may be used.

  In accordance with the present invention, an intercepting means 200 is provided that can cause an intercepted operation to be taken over transparently, ie, not recognized by a data storage system. As a result, when the data storage system is separated from the existing file system, the file system can be expanded with a new function. In contrast, conventional intermediate programs that perform some data interception function, such as virus inspection software, encryption shim, etc., utilize the original data storage system to perform the function.

  Furthermore, in the present invention, in an existing data storage system, the original semantics of the data storage system are maintained while separating the position of stored data or metadata from the data storage system. Thus, while the data storage system remains responsible for the existing data or metadata that it already stores, the intercept system is subject to new policies according to its own policy, unless the external semantics of the existing data storage system are changed. Responsibility for data or metadata, or modified data or metadata can be taken over.

  In the representative example described throughout this specification, the original file system remains the responsible entity for all metadata, and the intercept system takes over responsibility for some or all of the data. Specifically, in the present invention, the NTFS file system metadata including directories, the master file table (MFT), and the contents of each MFT entry are maintained on the file system storage resources as is usually the case. However, the data itself can be manipulated separately and stored independently of the original data storage system configuration.

  In the present invention, for example, in a legacy NTFS implementation that is not updated, the latest file system functions such as duplication, deduplication, caching, pooling, and the functions normally provided by storage resources (RAID level), and new special functions Numerous advantages are provided, such as means for supporting Since the file system external application interface presents unupdated semantics, the present invention enables these functions interchangeably for existing applications. That is, the file system remains unupdated and can be used by the operating system without being updated.

  As an example of a conventional attempt to solve the same problem as the problem of the present invention, a Windows (registered trademark) home of Microsoft (registered trademark) which is a data storage system having a simple expandability for home users. There is a server (WHS) project. WHS initially included a feature called Drive Extender that uses a file-based paradigm to extend NTFS on Windows® systems, which means that the NTFS file system has all files at different destinations. The reference information (“Tombstone”) can be left in the original NTFS file system. This approach failed because it was not completely transparent to the application (the remaining reference information behaved differently from the original file), in fact, the function that the drive extender attempted to provide was initially virtual Reimplemented using the block storage resource model, that is, appearing as a storage resource to the file system. This feature was then completely removed from WHS. Other examples of the original drive extender approach are known in the art, for example, in a hierarchical storage management system. On the other hand, in the present invention, the file system semantics and metadata remain in the original storage resource so that they can be referred to as necessary, while the data itself is not subject to restrictions in the configuration of the original data storage system. Can be manipulated and stored.

  Although the example in the previous paragraph is presented with respect to a disk storage system, the approach of the present invention can be applied to non-cooperative file systems, other non-cooperative processes such as key value stores, object stores, various caches, databases, etc. It may also be applied to an extension to a dynamic data storage system. With the ability to separate and place the data separately, or simply as an additional process to the original data storage system design, the existing data storage system can be used and many new functions and corresponding Functional characteristics can be provided to the application. Applicants believe that methods and systems for extending data storage system functionality beyond the limitations of the file system and the data storage system itself by separating the data storage function from the other functions of the data storage system are conventional. Not disclosed by technology.

  One advantage of incorporating the system and method according to the present invention is the potential for improved performance. Redirect data can be stored in storage resources that are faster than those used by the original data storage system, and / or can be stored in a manner that allows for faster reading and / or writing. You can use this mechanism at the same time. For example, the storage system metadata may be retained in the original storage resource, and the data itself may be stored in a high-speed storage resource. In this example, the intercepting unit acquires the metadata from the original storage resource and instructs the file system to acquire the data from the high-speed storage system. And when information and data are passed to the file system to be processed by the operating system, the data appears as if it were obtained directly from the original storage resource, but faster. For example, performance can also be improved by caching data in secondary storage resources that have high speed or other improved characteristics.

  In one embodiment of the invention, redirect data is stored in a sparse file on a fast storage resource. A sparse file is a file including a hole to which no data is assigned in a storage resource, and the data is virtually represented. One of the problems with file caching systems is representing the files to be cached in a space efficient format, which usually stores the actual data and stores the internal file locations and data about these locations. It is necessary to generate a data packing module that performs conversion between positions. In order to cache data at the sub-file level as needed, data that has not yet been acquired from the original file needs to be represented virtually instead of physically. The use of sparse files allows any file system that supports sparse files to provide this functionality, and exempts the need to develop and provide this functionality in the present invention. Applicants believe that the use of sparse files for the purpose of caching real file data is new, and the value of the availability of existing features for the purpose of providing this internal functionality of the subfile cache is Those skilled in the art will understand.

  The present invention may be used in a data compression context and can compress data in a variety of ways that are not limited by the characteristics of the original data storage system. For example, in normal stream compression or deduplication compression, data is not stored again if it already exists somewhere in the system. This type of data compression is usually not directly possible at the level of data storage systems based on the teachings of the prior art. Those skilled in the art will appreciate the distinct advantages of being able to provide data compression in any file system and data storage resource system, regardless of the file system and data storage resource system used.

  Another possible application is data polymorphism, where data can be represented differently depending on the context in which it is used. A variety of data polymorphism applications are known in the art, but these are typically performed at the application level and are constrained by the data storage system. In accordance with the method of the present invention, data can be manipulated in an intercept system, ie, at the data storage system level, and context can be changed and presented to the file system, so operating system and application resources, or changes Is not required.

  In the context of a document management system, the present invention allows versioning of redirected data so that the semantics of the data storage system are maintained and extended semantics are provided that allow retrieval of older versions of data. . In current systems, old versions of data need to be updated to incorporate extended semantics or read out in such a way that old data is distinguished from new data.

  The redirect data input / output may be an input to a distributed data storage system that manages the location of the source data based on the time-related utility of the data to itself. Again, in this regard, these characteristics can be implemented at the data storage system level and are presented to the application accessing the file system without any changes.

  In the context of name virtualization, data no longer needs to reside in the storage resources of the original data storage system and can be located in completely different locations without adaptation to the file system. Data virtualization is also possible and a single named data no longer needs to reside on the same storage resource, but can be distributed across multiple data storage systems and / or storage resources. In the context of data replication, the input and output of redirect data can drive the replication process to ensure data redundancy.

  For the data and operations described above, well-known implementation examples such as the use of virtual machine implementations exist mainly at the application level, but the disadvantages of these implementation examples are described in the background section of the invention. . The present invention allows these characteristics and functions to be provided at the data storage system level, thus making these characteristics and functions independent of the file system or operating system.

  Various modifications and implementations of the invention may be made without departing from the spirit of the following claims, for example, single-level or multi-level caching of a data storage system may be performed by an interceptor, Is performed by intercept means, and locality optimization is performed by pushing infrequently used data to a remote data storage system and pulling frequently used data to nearby data storage systems Is done. A remote data storage system is either a data storage system that is physically remote or takes a long time to access. In addition, name-based virtualization can be performed by interceptors, and by referring to data on other data storage systems, filenames that are not valid in current data storage systems can appear to be valid. it can. Data backup and replication is also possible by having the interceptor communicate directly with the file system. Data virtualization is also possible at the data storage system level, data can be physically located in different data storage systems with the same name, and the execution stage is executed by the interceptor means.

  According to another aspect of the invention, the method comprises providing a unique function for selected data in an intercept system rather than a storage resource. The unique function is selected from the group consisting of performance characteristics, deduplication, data polymorphism, independent access control mechanisms, versioning, caching, locality, replication, and data virtualization. The selection data is preferably identified using a selection mechanism that employs metadata pattern matching with one or more selected from the group consisting of name, timestamp, size, history information, physical information, and context information. The

  The above embodiments are intended as examples of the present invention, and the above embodiments are within the scope of the present invention which is defined only by the claims appended hereto. Changes or modifications may be added by a trader.

Claims (59)

A method for extending the function of a data storage system having data organization means and data storage resources,
Separating the data storage function of the data storage system from other functions of the data storage system;
Transferring at least part of the data storage function to an intercept system.   The method of claim 1, wherein the intercept system has complementary storage resources. The separating is performed by providing intercept means for communicating with the data organization means, the data storage resources, and the intercept system;
The interception means intercepts a file system operation and determines whether the function of the operation should be handled by the data storage resource or by the intercept system. Method.   The intercepting means intercepts file system operations while the operations hold context and before the operations are broken down into independent operations suitable for the data storage resource. Item 4. The method according to Item 3.   The method according to claim 1, wherein at least another part of the data storage function is maintained in the data storage system.   The method according to any one of claims 1 to 4, wherein all of the data storage functions are transferred to the intercept system.   The method according to any one of claims 1 to 6, wherein the data storage system is selected from the group consisting of a file system, a key value store, an object store, and a network protocol.   The method according to claim 1, wherein the data storage system is shared among a plurality of external interfaces.   5. A method according to claim 3 or 4, wherein the intercept means comprises a user space application program cooperating with an operating system facility or a file system facility.   The method according to claim 3 or 4, wherein the intercepting means comprises a file system protocol proxy application running on a network.   5. A method according to claim 3 or 4, wherein the intercepting means comprises a mini filter driver for intercepting file system operations in an operating system kernel.   4. The intercept system of claim 3, wherein the intercept system uses one or more complementary storage resources to perform its functions, and the one or more complementary storage resources are independent of the data storage resources. Method.   The method according to any one of claims 1 to 12, wherein the intercept system performs capacity expansion of the data storage system.   14. The intercept system improves the performance of the data storage system by changing one or more characteristics of data on the data storage resource or by using complementary storage resources. The method of any one of these.   The method of claim 14, wherein the one or more characteristics are selected from the group consisting of a storage format, a storage location, and a storage order.   The method according to claim 1, further comprising performing de-duplication by the intercept system.   The method according to any one of claims 1 to 16, further comprising performing data polymorphism by the intercept system.   The method according to any one of claims 1 to 17, further comprising executing an independent access control mechanism for data by the intercept system.   The method according to any one of claims 1 to 18, further comprising the step of versioning data by the intercept system. Further comprising performing caching of the data storage system in either a single level or a multi-level;
The method of claim 12, wherein the performing is performed by the intercepting means.   The method of claim 12, wherein the complementary storage resource is a commonly used operating system memory cache to cache data.   Further performing locality optimization by pushing infrequently used data to a remote data storage system and pulling frequently used data to nearby storage resources or data storage systems The method of claim 12, wherein the step of providing and executing is performed by the intercepting means.   23. The method of claim 22, wherein the remote data storage system comprises a data storage system that is either physically remote or takes time to access.   Performing name based virtualization, which makes file names that are not valid in current data storage systems appear to be valid by referencing data on other data storage systems; The method of claim 12, wherein the performing is performed by the intercepting means.   The method of claim 12, further comprising performing one of data backup and data replication, wherein the performing is performed by the intercepting means.   Further comprising performing data virtualization including allowing data to be physically located in different data storage systems with the same name, wherein said performing is performed by said intercepting means. Item 13. The method according to Item 12.   Providing a unique function for selected data in the intercept system rather than the data storage resource, the unique function comprising performance characteristics, deduplication, data polymorphism, independent access control mechanism, versioning, caching, locality 27. A method according to any one of claims 1 to 26, selected from the group consisting of:, replication, and data virtualization.   The selection data is identified using a selection mechanism that employs metadata pattern matching with one or more selected from the group consisting of name, timestamp, size, history information, physical information, and context information. Item 28. The method according to Item 27.   29. A method according to any one of claims 1 to 28, wherein the data organization means is selected from the group consisting of a file system, a key value store, and a database.   The method of claim 12, wherein the complementary storage resource is a sparse file. A system for extending the function of a data storage system having data organization means and data storage resources,
Separating means for separating the data storage function of the data storage system from other functions of the data storage system;
Means for migrating at least part of the data storage function to an intercept system.   32. The system of claim 31, wherein the intercept system has complementary storage resources. The separation means comprises an interceptor in communication with the data organization means, the data storage resource, and the intercept system;
The system according to claim 31 or 32, wherein the interceptor intercepts a file system operation to determine whether the function of the operation is to be handled by the data storage resource or by the intercept system. .   The interceptor intercepts file system operations while the operations hold context and before the operations are broken down into independent operations suitable for the data storage resource. 34. The system according to 33.   35. A system according to any one of claims 31 to 34, wherein at least another part of the data storage function is maintained in the data storage system.   35. A system according to any one of claims 31 to 34, wherein all of the data storage functions are transferred to the intercept system.   37. The system according to any one of claims 31 to 36, wherein the data storage system is selected from the group consisting of a file system, a key value store, an object store, and a network protocol.   The system according to any one of claims 31 to 37, wherein the data storage system is shared among a plurality of external interfaces.   35. The system of claim 33 or 34, wherein the interceptor comprises a user space application program that cooperates with an operating system facility or a file system facility.   35. A system according to claim 33 or 34, wherein the interceptor comprises a file system protocol proxy application running on a network.   35. The system of claim 33 or 34, wherein the interceptor includes a mini-filter or legacy filter driver that intercepts file system operations in an operating system kernel. The intercept system uses one or more complementary storage resources to perform its functions,
The system of claim 33, wherein the one or more complementary storage resources are independent of the data storage resources.   34. The system of claim 33, wherein the intercept system performs capacity expansion of the data storage system.   34. The system of claim 33, wherein the intercept system improves the performance of the data storage system by changing one or more characteristics of data on the data storage resource.   45. The system of claim 44, wherein the one or more characteristics are selected from the group consisting of storage format, storage location, and storage order.   46. A system according to any one of claims 31 to 45, wherein the intercept system performs data deduplication.   47. A system according to any one of claims 31 to 46, wherein the intercept system performs data polymorphism.   48. A system according to any one of claims 31 to 47, wherein the intercept system implements an independent access control mechanism for data.   49. A system according to any one of claims 31 to 48, wherein the intercept system performs data versioning.   35. A system according to claim 33 or 34, wherein the interceptor performs caching of the data storage system in either a single level or a multi-level.   43. The system of claim 42, wherein the complementary storage resource is an operating system memory cache typically used to cache data.   34. The interceptor performs locality optimization by pushing infrequently used data to a remote data storage system and pulling frequently used data to a nearby storage resource or data storage system. Or the system according to 34.   53. The system of claim 52, wherein the remote data storage system comprises a data storage system that is either physically remote or takes time to access.   35. The interceptor performs name-based virtualization that makes file names that are not valid in current data storage systems appear valid by referring to data on other data storage systems. The described system.   35. A system according to claim 33 or 34, wherein the interceptor performs one of data backup and data replication.   35. A system according to claim 33 or 34, wherein the interceptor performs data virtualization comprising allowing data to be physically located in different data storage systems with the same name.   32. The intercept system performs one or more selected from the group consisting of performance characteristics enhancement, deduplication, data polymorphism, independent access control mechanism, versioning, caching, locality, replication, and data virtualization. 57. The system according to any one of to 56.   58. The system according to any one of claims 31 to 57, wherein the data organization means is selected from the group consisting of a file system, a key value store, and a database.   43. The system of claim 42, wherein the complementary storage resource is a sparse file.

Images