JP2009501382A - Maintaining writing order fidelity in multi-writer systems - Google Patents

Abstract

Write Order Fidelity (WOF) is for fully active implementations where multiple access nodes at geographically distant sites read and / or write data in a distributed data system in parallel in a “fully active” manner. Maintained. From the perspective of the host at various geographic locations, a synchronized, cache-coherent view of the data is provided. Data transfer is asynchronous. Time-ordered data images are created and maintained so that operations can be restarted after a partial system failure that has not yet been asynchronously transferred over the network, but that causes the loss of data that has been acknowledged to the source host . Time-order asynchronous data transfer is implemented as a change pipeline that reflects the contributions from all nodes. WOF also improves network performance and reduces bandwidth consumption.

Description

(Citation of related application)
This application is a non-provisional patent application and claims priority to US Provisional Patent Application No. 60 / 699,935 (filed on July 14, 2005, agent case number: 019417-008700US). The entire contents are cited by reference for all purposes herein.

(Copyright notice)
Part of the disclosure of this patent document contains elements that are subject to copyright protection. The copyright owner has no objection to reproduction by any one of the patent documents or patent disclosures displayed in the US Patent and Trademark Office patent file or record, but otherwise, no rights of any copyright Also possess.

(Background of the Invention)
The present invention relates to a system and method for providing recovery of the contents of a data storage system after a failure or other possible cause of data loss or corruption, and more particularly, a distributed data storage network. The present invention relates to a system and method for providing write order fidelity to a storage system having a data writer operating in parallel at multiple locations.

  In current storage networks, and in certain storage networks that contain storage resources and geographically separated access nodes that are interconnected by the network, the distance between the nodes can be reduced if writes must be replicated or transmitted synchronously. If it increases, write performance is greatly hindered. Furthermore, it is highly desirable to minimize the required bandwidth between locations. Therefore, the method of asynchronously transmitting data is used when writing is confirmed before the data is transmitted to the remote site node.

  It is also desirable that the data access be localized in order to improve the access speed to the data block requested by the host device. Caching a block at an access node provides localization, but the cached data must remain coherent to modifications at another access node that may have cached the same data. Don't be.

  Furthermore, such complex storage applications need to tolerate failures of their supporting storage systems, local storage networks, networks interconnecting nodes, and access nodes. In the event of a failure, asynchronous data transmission indicates the possibility of loss of data held at the failed site. The consistent data image needs to be constructed from the remaining storage contents from the application perspective. An application must make some assumption about which writes or pieces of data are being written to a storage system that has survived a storage system failure, specifically by writing to a given block. All writes confirmed completed by a storage system that maintains write ordering, such as when a fix is lost, all subsequent writes to the block or its associated volume in that volume are also lost Against.

  As used herein, write order fidelity (“WOF”) refers to a group of related properties, each describing the contents of the storage system after recovery from some type of failure. That is, after the storage system recovers from a failure, it is a property that an application can assume about the contents of the storage system. Write order fidelity (WOF) provides a guarantee that the remaining data is consistent after recovery from a failure. Complex applications such as file systems or databases rely on this consistency property to recover from storage system failures. Even simpler applications that are not explicitly written to recover from their own failures or back-end storage failures should benefit from these post-failure guarantees.

When implementing a strict sense of WOF, an application generates a stream of writes {W _i | i ≧ l} to a storage system that supports the application. The underlying storage system exhibits strict write order fidelity if, after any failure of the storage system, the state of the storage system at recovery reflects part of the prefix of the write sequence from the application. In other words, there is some i ≧ 0 such that all writes {W _j | j ≦ i} are committed to storage and no write {W _j | j> i} is committed to storage.

  Strict WOF assumes that writes are fully ordered, but this is straightforward for a single controller or a set of tightly coupled storage controllers that communicate through shared memory. However, the cost for generating a complete order for writing is enormous for controllers that communicate via messages, even within a site. The cost of ordering is unacceptable even when latency between controllers reaches a few milliseconds.

  Traditionally, the “active-passive” approach is a site-to-site where only one writer or host processor has read-write access to a given volume of blocks and the other processor has read access only. Used for asynchronous transmission of data. A “fully active” environment, where reads and writes to a given volume of blocks can be done randomly from any node, is highly desirable, but the approach to WOF and the interaction with caching on all access nodes of the system A change in method is required.

(Summary of Invention)
Embodiments in accordance with the present invention provide write order fidelity (WOF) to distributed storage systems in which storage access nodes, commonly referred to as storage controllers, and storage systems are interconnected by a network. Obviously, any network that allows communication between nodes can be used. While various embodiments allow for fully active operation where writes and / or reads to any given data volume may be initiated from any node of the system, a similar system is conventional active-passive. Obviously, it can also be used in modes.

  The WOF is obtained in some aspects by using a delta set distributed or fragmented across various sites and / or nodes that use a cache coherency layer. Various embodiments use distributed cache coherency to ensure that the most recent writes to any given node are immediately reflected in subsequent reads by the site. Without providing a WOF to a fully active site, thus providing a coherent application view. This reflects the most recent writes, with data blocks written to any given delta set distributed across the distributed nodes being coherent and the corresponding partial deltas being stored on various nodes of the system I can also guarantee that.

  In one embodiment, one or more host visible volumes of data are managed as a WOF group. Even if multiple data volumes constitute a WOF group, WOF is guaranteed for all blocks from all data volumes in the WOF group as if all blocks were in a single volume.

  In one embodiment, a delta set multi-stage pipeline is used to collect newly written block images to exchange block images between nodes and commit the block images to back-end storage. A system-wide barrier mechanism ensures that the delta set pipeline proceeds simultaneously on all nodes of the system. Obviously, the barrier need not be system-wide if not all nodes are participating in the management of a given WOF group. New writes to any of the multiple data volumes that make up a WOF group are stored in an “open delta” cache for that WOF group. Upon any trigger or other event, the current open delta should be closed. A message to close the open delta is broadcast to that WOF group so that any outstanding writes can be completed and the delta can be closed. A new delta is opened to receive a new write. The latest closed delta, which can exist as a unique fragment in different WOF groups, undergoes an “exchange stage”, whereby each site can obtain a complete copy of the closed delta. After all the modified blocks are exchanged between nodes and the next barrier occurs and demarcates the next progression of the pipeline, the complete closed delta enters the “commit phase” to provide stable storage. It can be made permanent by writing.

  These and other embodiments of the present invention, and the benefits and functions thereof, will be described in more detail in conjunction with the following text and accompanying figures.

  Various embodiments according to the invention are described in connection with the drawings.

(Detailed description of the invention)
The systems and methods according to the various embodiments are such that each single site is written order fidelity, as opposed to existing active / passive implementations where a single writer typically pushes data over the network to the passive site. Overcoming the above and other deficiencies in existing data storage systems by providing various fully active implementations that can read and / or write data in parallel while maintaining (WOF). These and other objectives can be achieved in one aspect by combining an improved delta set approach to write order fidelity with an approach for providing distributed cache coherency.

  In one aspect, the time order image of the data is created and maintained so that after a storage system failure, the operation can be performed without having a corrupted file system, database, or other application inconsistent with the view of the data. Can be restarted. Typically, existing systems must transmit data in blocks known in the art as the basic unit of data transfer and maintain the order of those blocks. This creates a significant number of short transactions and can cause degradation of network and / or system performance with increasing latency. One way to reduce the number of transactions is to group the blocks into deltas or delta sets. Using a delta set increases the granularity of the data transfer over the network, thus increasing the efficiency of the network transfer, and the multiple of individual blocks written multiple times within the time interval captured by the delta set. If a transfer is eliminated so that an individual block is written many times, the block is written only once using a delta. The boundary between deltas provides a data write order consistency image.

  The delta set is described, for example, in US Pat. No. 6,823,336 issued November 23, 2004, which is incorporated herein by reference. Other descriptions of how to mirror delta and remote data can be found in, for example, US Pat. No. 7,055,059, issued May 30, 2006, and “Seneca: remote mirrowing done write” (USENIX Technical Conference Summary, pages 253-256, June 2003), each of which is incorporated herein by reference.

  Prior to the present invention, delta set-based WOF implementations supported only active-passive data access. In this scenario, the WOF implementation is fairly relatively simple because the “active” site can fully control the transition between WOF images. A simple active-passive implementation of WOF involves maintaining only two delta sets with memory areas allocated to each of the two partial deltas at the active and passive sites. After a decision is made at the active node to advance the delta image, new writes are simply accepted into the alternate buffer. A message is sent to the passive site instructing the switch and transferring data from the newly closed delta. When the data “push” to the passive site is complete, both sites can commit the data just exchanged to disk. When this is complete, the active site can toggle back and forth between delta set buffers to close the current delta. Enhancements to traditional delta set-based WOF implementations include maintaining more delta sets, which are typically maintained as alternating sets of buffers, and the exchange of closed buffers reduces short-term network bandwidth. Permits delays because of saturation. Further details of the basic use of the delta set are described in US Pat. No. 6,823,336, incorporated by reference, but no further details are described here.

  Potentially, a more advanced approach is needed to bring the potential of multiple nodes to multiple sites writing to a common data volume. In one embodiment, the distributed cache coherency mechanism is based on US patent application Ser. No. 11 / 177,924 filed Jul. 7, 2005, entitled “Systems and Methods for Providing Distributed Cache Coherence”, and Jul. 7, 2004. No. 60 / 586,364, all of which are hereby incorporated by reference. These methods provide a mechanism in which a cache maintained in multiple local and network separated nodes with the possibility of read / write access from all nodes is kept coherent. In other words, the data image distribution order is provided as a system in which all hosts access a single disk drive. The use of distributed cache coherency with delta sets allows for a synchronous image of data even if the actual data movement is asynchronous. Data movement maintains geographic order overall write order fidelity, which allows the system to restart at any consistent point in time.

  In one aspect, a directory manager module (“DMG”) can be used to provide a cache coherency mechanism for shared data across a set of distributed data access nodes. A set of nodes used to cache data from a shared data volume is called a shared group. In general, a DMG module includes software that executes in a node's processor or other intelligence module (eg, ASIC). The DMG module may be implemented on a single node or distributed across multiple intercommunication nodes. In some aspects, an access node is embodied as a controller device or node communicatively coupled to a storage network such as a storage area network (SAN) and provides access to data stored in the storage network. enable. However, it will be apparent that the access node can also be embodied as other network devices such as intelligent fabric switches or hub adapters. Furthermore, any network node can be configured to operate as an access node with DMG functionality (eg, DMG can run on a desktop computer with network connectivity). US Pat. No. 6,148,414 (incorporated herein by reference) discloses controller devices and nodes in which implementation of aspects of the invention are particularly useful.

  For one embodiment of the present invention, FIG. 1 includes a plurality of network clients 102 (a) -102 (N) communicatively coupled to a plurality of access node devices 104 (a) -104 (N). A basic network configuration 100 is shown. Each access node device includes a processor component 106, such as a microprocessor or other intelligence module, a cache 108 (eg, RAM cache) and / or other local storage, a communication port (not shown), and an instance of the DMG module 110. . In general, “N” is used herein to indicate that there is a plurality, so the number “N” when referring to one component is not necessarily the same as the number “N” of another component. It does not have to be equal. Each client 102 can be communicatively coupled to one or more of the access nodes 104 over the local network connection 112 for speed and other reasons, or as required for a particular application and geographic location. Can be communicatively coupled to node 104 in any of a number of connection schemes, including, for example, a direct wiring or wireless connection, an Internet connection, any local area network (LAN) type connection , Any metropolitan area network (MAN) connection, any wide area network (WAN) type connection, VLAN, any proprietary network connection, etc. Each node 104 also typically includes or is communicatively coupled to one or more other nodes, such as a high-speed network such as a storage area network (SAN), LAN, WAN, MAN, Infiniband, etc. On the above network 116, it is communicatively coupled to one or more storage resources 114 each including one or more disk drives. In one aspect, node 104 is preferably coupled to one or more storage resources 114 over a local network connection. The nodes may be located in physical proximity to each other, or at least one may be located remotely (eg, geographically remote) from other nodes. The access node may be any other communication network, such as on the network 116 and / or on the PCI bus or backbone or Fiber channel network, or on the same network 112 that the client device 102 is using to communicate with the access node 104 or It is also possible to communicate with other nodes on the media.

  Distributed cache coherency helps reduce bandwidth requirements between geographically distant access nodes by enabling localized (cached) access to remote data. Data access generally cannot be localized unless the data can be cached locally, and caching the data locally unless the cached data can remain coherent to modifications at the remote access node Unsafe. The DMG embodiment can meet cache coherency accuracy requirements and can keep the overhead for creating practical and useful localized cache accesses low enough. Although the embodiment described in FIG. 1 shows a single DMG directory 110, the method for distributed cache coherency taught in the above patent manages coherency on a peer-to-peer basis, and also has multiple nodes and It can be extended to large distances between nodes. Distributed cache coherency can be combined with the use of delta sets to provide write order fidelity (WOF).

For one embodiment, write order fidelity (WOF) and “dependency WOF” can be more formally defined as follows: An application can use an update operation to update its persistent state, which includes metadata update W ₁ (eg, writing an entry in the database recovery log) and data update W ₂ (eg, And write the modified data to the database). For the application itself or the storage of any suitable disaster recovery, application executes data update W ₂ Wait for metadata updates W ₁ is completed. The timing relationship between these two writing is here but is represented as W ₁ → W _2, W ₂ is considered that it is "dependent" for W _1, W ₁ is taken to W ₂ considered a "necessary" It is done. Using such an approach, W ₁ records the information to allow that the W ₂ is lost it can not be interpreted correctly. In such a storage system, dependent writes can be monitored by noting that the start time of W ₂ is after the completion time of W ₁ .

Such a storage system is considered to exhibit “dependency” write order fidelity if only one of the following cases is true for two writes W ₁ → W ₂ after recovering from any failure: (A) Neither W ₁ nor W ₂ is applied to the storage, (b) both W ₁ and W ₂ are applied to the storage, or (c) only W ₁ is applied to the storage.

  The dependency WOF defines a partial order, not the whole, in a write operation. In some embodiments, this can use a globally synchronized time concept, but other embodiments avoid this problem. Furthermore, writes that are not ordered by the dependency WOF are just writes that are performed in parallel by the application. According to one aspect, dependency write order fidelity is an appropriate definition of the consistency of a network controller, such as a NetStorage product. US Pat. No. 6,148,414 (incorporated herein by reference) discloses a useful network controller embodiment.

In one aspect, the storage system determines that the write pair W ₁ and W ₂ are dependencies from an application perspective. Storage device, since W ₁ is only capable of monitoring that W ₂ arrives after completing the storage device is not easy to determine these dependencies. There is no dependency between these two writes, and a clear ordering may just be a coincidence. Without assistance from the application (although that is not expected), storage systems usually cannot distinguish between a coincidence match and a real dependency write pair. For safety, the storage system, all cases in which W ₂ arrives after the W ₁ is completed can be assumed to represent the true dependence writing. This creates a stronger partial order than is needed, but the true write dependency becomes part of this strong ordering.

  In one aspect, an environment with multiple active cache coherent initiators uses a global time concept to provide dependent WOFs. The original interpretation of such use implies an absolute global ordering of writes. The cost per write of this original approach is very high, and it is not worth considering seriously, especially in geographically distant situations. One alternative involves grouping batches of writes into deltas so that the boundaries between deltas follow dependent write order fidelity as described above.

  Within the delta used to represent system changes as described above, writes are reordered, providing some degree of optimization. When a delta is closed, all writes contained in the delta depend on the same or previous delta writes. If the required write is postponed to a later delta, the WOF is no longer stored across the delta boundary. The operation of closing the delta should therefore be coordinated across all participating nodes.

  Delta, in one embodiment, is global across all nodes at all sites participating in the WOF group. Each node collects locally written delta fragments. Since each site needs a complete copy to commit to local storage, the sites exchange fragments and assemble a complete copy. In one aspect, these partial deltas are exchanged between sites before being applied locally. After the exchange, each site can commit the delta locally to the underlying storage. Local commits are made atomically and in the correct order with respect to other deltas. The atomicity requirement means that the delta is made permanent before it is applied. Changing the degree of persistence can affect the strength of the WOF consistency guarantee. For example, placing the delta on stable storage is safer than copying the delta to volatile memory on a redundant device.

  For the WOF solution, you can see the underlying storage operating atomically through a set of consistent states. If each site collects the entire global delta before applying the delta locally to the storage, a link failure between the sites will not keep the storage in a consistent state. Delta is applied as a whole or not at all.

In one aspect, the network administrator groups front-end volumes that need to be consistent with each other into a WOF group. For example, all data and log volumes in the database can be placed in the same WOF group. Each WOF group, in one embodiment, is managed by several subsets of nodes and sites in the entire system. New writes to the WOF group are collected in the current open delta Δ _i cache, and the cache backend generates a delta.

  For the WOF solution, you can see the underlying storage operating atomically through a set of consistent states. If each site collects the entire global delta before applying the delta locally to the storage, a link failure between the sites will not keep the storage in a consistent state. Delta is applied as a whole or not at all. Thus, due to the implicit time order given by the complete delta, the atomic writing of the delta to the underlying storage can preserve the WOF properties, whether or not they are complete.

In one aspect, new writes to the WOF group are collected in the current WOF delta Δ _i cache, and the cache back end generates a delta. A delta that collects new writes is considered "open". In one aspect, the decision to close the open delta to stop accepting new writes and advance the delta pipeline is made periodically based on some time or space constraint. The decision to close the delta can be made by an external trigger or as part of recovery from a system error condition. Closing a delta, opening a new delta, and generally advancing the delta pipeline is referred to as "delta rollover." In one aspect, the node making the decision to close the delta can send or broadcast a notification to the WOF group. As each node receives the notification, it begins delaying the acknowledgment to a new write operation. This ensures that all write order dependencies go from Δ _i to Δ _{i + 1} , or are contained within Δ _i . When all writes incomplete at the time of receiving the notification is completed, delta _i is closed. Then Δ _{i + 1} is opened and at the same time Δ _i goes to the exchange phase, it starts collecting new writes. It is important to minimize the time that writes to the WOF group are delayed to ensure that the application does not sense disruptions in I / O. If a general ordered broadcast service imposes too much overhead on these broadcasts, a lighter two-phase commit protocol can be used, but this minimizes the nodes involved in the commit protocol.

  Delta pipeline rollover operations imply periodic performance impacts. Since WOF integrity guarantees may only be needed in case of a catastrophic failure, alternative implementations choose to recover the delta boundary during error recovery, thereby eliminating the need for global barriers during normal operation. It's okay.

Recently closed deltas can exist as fragments in the cache of the access node of the WOF group. In one aspect, the cache coherency protocol ensures that each fragment contains unique, non-redundant “dirty” data. Each site assembles a complete copy of the delta by exchanging blocks or partial deltas contained in these fragments. In one aspect, the atomic commit phase can be simplified if the copy of Δ _i for the site is on a single node. However, this imposes strict constraints on the global size of the delta and can make the exchange phase and the decision to close the delta more complex. In another aspect, the writes within the collected delta can be reordered to take advantage of overwrites or adjacent write segments.

In one aspect, the delta is made permanent before Δ _i can be committed. The persistence operation can overlap with the exchange phase. For strong consistency guarantees, data describing deltas and associated metadata can be written to stable storage. Data can be copied one or more times to another node at the same site, thereby providing a “protected copy” and providing a less strong guarantee. Such a protection approach can be faster, but may impose additional memory constraints. A weaker guarantee does not require persistence. Without persistence, there is no protection against multiple failures, but data integrity can be maintained in the event of a site disaster.

After Δ _i is exchanged and made permanent, in one aspect, the sites harmonize and begin applying Δ _{i to} the back-end storage. This cannot be done truly atomic and can be interrupted. Persistent copies are protected from these failures as long as some nodes can redo the commit. When the delta commits, interrupting the intersite link has no effect because the operation is purely local to the site.

  The introduction of WOF in one aspect only affects the order in which dirty data comes from the cache back end, and has no effect on the cache coherency protocol. A read request to any node always returns the latest copy of the data. A cache write that hits a closed delta cannot invalidate the previous copy, but instead creates a new copy in the open delta. This new copy shadows any previous version of the block, including a persistent copy stored in the underlying storage for cache coherency purposes.

(Example dependency WOF implementation)
In one aspect, an exemplary WOF implementation has a dependent write order and is fully active across multiple sites and within multiple sites. Such an implementation fully supports fully active WOF across sites and across nodes within the site. Implementations can also be delta-based, where each delta contains a consistent batch of writes. The system is collecting deltas as application writes to the network controller node. For example, at intervals on the order of about 5-30 seconds, the system synchronizes the current delta closure locally and globally. A new delta is opened immediately to begin collecting new writes. The closed delta is then written atomically to backend storage. The most recently committed delta defines the repair point in case of failure.

  Such an implementation may also provide support for WOF consistency groups. Databases, journal file systems, and other similar applications often separate their data volumes from their metadata and log volumes. Administrators must be able to group these volumes and provide WOF across them as a set, not just individually.

  This paragraph gives a more specific example of WOF implementation. FIG. 2 shows an exemplary two-site system 200 that includes site A 202 and site B 204 with one leg 206 of distributed RAID 1 (DR1) at site A and one leg 208 at site B. First, each site collects writes in the current open delta D1 210, shown schematically in the open box (210a, 210b) at the top of each site. Writes (W1, W2, W3, W4) are collected locally at each node in the site according to normal cache coherency protocols. US patent application Ser. No. 11 / 177,924 (incorporated by reference above) discloses useful cache coherency systems and methods.

  Finally, for several different reasons, the system can decide to close the current delta. For example, a user configurable timer may have expired. Administrators can use such configurable timers to reduce the amount of data that can be lost if a failure occurs. Another possible reason is that an external API is called by the application. The application can use this call to indicate when the data is perfectly consistent from that point of view. Another possible reason is that the system may run out of resources of one or more nodes. One of the nodes collecting the delta may determine that the node needs to close the delta early.

  The system in one aspect synchronizes the closing of the current delta across sites so that the delta boundaries adhere to the ordering consistency of dependent writes. In other words, each delta edge can define a consistent view of storage. As shown in FIG. 3, once the current delta is closed, a new delta D2 300 can be immediately opened to collect new writes from the application. The system then exchanges the partial delta collected at each site via the inter-site link 302 so that each of sites A and B can have a complete copy of the closed delta (210c and 210d, respectively). .

  The closed complete deltas 210c, 210d can be applied to each leg 206, 208 of DR1 via the appropriate link 400, as shown in FIG. This apply process is not necessarily an atomic operation. That is, the operation may be interrupted by various failures. Depending on the type of failure, you may need to restart the process that applies the delta.

  FIG. 5 is a flowchart illustrating the steps of an exemplary method 500 according to such an approach. In this method, front-end volumes are grouped into WOF groups, each implemented by a subset of nodes and sites in the entire data system (502). New writes to any of the WOF groups are stored in the open delta cache of that WOF group (504). At some trigger event, the current open delta is closed (506). A close delta message is broadcast to the WOF group, which may complete any outstanding writes and close the delta (508). A new delta is then opened (510) to receive a new write. Recently closed deltas that exist as unique fragments in different WOF groups go through an exchange phase, whereby each site gets a complete copy of the closed delta (512). The complete closed delta is made permanent by writing to stable storage (514). After the closed delta is made permanent, the site applies the closed delta to each site's backend storage (516).

  The situation 600 of FIG. 6 illustrates the above process that proceeds in parallel when a link failure occurs. When writes are being collected for open delta D3, closed delta D2 is swapped and committed delta D1 is applied in this case to the disk that is a leg of the distributed RAID 1 system for sites A and B respectively. ing. If the link 602 for exchanging write data between sites fails, the front end I / O is interrupted and no further writes are collected at D3. Fragments of D2 cannot continue to be exchanged, and each site will either until the link is restored or until the administrator declares that one site must resume operation before the link is restored , Keep “dirty” data for that delta. However, the committed delta D1 can continue to be applied to the leg, as shown in situation 700 of FIG. 7, because D1 is no longer dependent on the active link. Even if the link is broken after this is complete, the DR1 legs will match.

  If the link failure is temporary, normal operation resumes when the link recovers. If Dirty Data is lost because Site B has only a few copies of the system confirmed, Sites A and B discard the contents of Deltas D2 and D3 and O service can be resumed. The content of the storage seen from site A is consistent with the last successfully exchanged and committed delta D1. While writing to only the DR1 local leg, Site A can update the bitmap log so that when the link is restored, the two legs of DR1 can be synchronized.

(Effects of WOF performance)
The WOF implementation can be a significant change to the host write flow from the front end to the back end physical storage. Some aspects of the implementation can affect the net performance of the system as perceived by the host.

  For example, in one aspect, all nodes engaged in the WOF group must close the current delta synchronously. This has an obvious performance impact, especially in multi-site configurations. However, it should be noted that during this synchronization interval, writes are successful and incoming writes receive data from the host but are not confirmed until the delta close is complete. In addition, by collecting changes in deltas, these changes can be streamed over inter-site links more efficiently than smaller individual write operations.

  As mentioned above, the operation is vulnerable to failure because application of committed deltas to storage is not an atomic operation. It can be determined which of the committed deltas that have not yet been applied to storage are protected against failure. This corresponds to the selection of how strong the consistency guarantee is. There are several implementation options in this area, each with different performance tradeoffs.

(Delta collection stage)
In one aspect, the first stage of the WOF pipeline is the collection of new writes into the open global delta. As a result of the coherency protocol at the entrance of the cache, there are no two dirty copies of the same block. Thus, the intersection of partial deltas from different nodes or different sites is an empty set. The entire global delta is a union of all partial deltas.

  Since dirty blocks are added to the open delta, they need to be linked to the open delta so that the blocks will advance through the pipeline with the remaining deltas in a consistent write order. To achieve this according to one embodiment, two metadata can be added to the cache block data structure that is used only for dirty data. The first metadata is a delta identifier that stores the delta number to which this block belongs. This identification is unchanged. The delta number is a system-wide global number that is incremented each time a new delta is opened as the pipeline progresses. A second metadata that can be added is a delta list that allows a cache block to be stored with its peers from the same delta. For this purpose, a single linked list is sufficient.

  Both metadata fields can be assigned as soon as a new write is entered into the open delta. In addition, to protect against node failure, when an incoming write is copied to another access node, the delta ID is protected copy so that it can be recovered if the node that received the write fails. Can be added to the metadata stored with it.

  When a new write arrives in an already dirty block in the cache somewhere in the system, one of at least two actions can be performed. If the dirty block is in an open delta, the dirty block can simply be invalidated. However, if the dirty block is in the old delta, invalidating the dirty block would violate dependency write consistency. In at least one embodiment, in this case, a new write must be entered into the open delta and then the old contents of the block must be shadowed for cache coherency purposes.

  When a new write arrives in an already dirty block in the cache somewhere in the system and the write modifies only a portion of it instead of completely overwriting the block, there are special actions beyond those described above. May be necessary. The entire block of data to be modified can be transferred to the node that processes the write in such a way that the previous contents of the block are not lost due to node failure. One simple approach is to erase the block from storage before sending it to the node that handles the new write for the node that holds the current dirty contents of the block, This method is not suitable for WOF because it violates dependency write consistency. Instead, the cache protection mechanism described above can be extended, for example, as follows. (1) The old contents of the block are transferred to the write processing node that creates the protected copy as described above. (2) The original holder of the block invalidates the protected copy corresponding to the copy of the block. (3) The node that processes the write applies the data from the application to the block and updates its protected copy. However, when this node is in open delta, it must obey the restrictions described in the previous paragraph regarding simply deleting the old write and its protected copy. (4) Finally, application writing is confirmed.

  In one aspect, all write requests need to be sent with the requester's delta ID in the DMG. This can be achieved in the DMG / cache API, for example by using the following example.

一態様において、リクエスタは、ブロックのダーティコピーを、ディレクトリから受信するとすぐに保護し、それからホストからの新しいデータを受け入れる前に、書き込みの分散完了を開始し得る。このアプローチは、共有者に対するＤＭＧロックとリモート無効化が、可能な限り早急に完了され、それにより同ページについての後の分散操作を妨げる可能性が低くなるという利点がある。このアプローチに欠点があるとすれば、ライタが、イニシエータへの書き込みを完了する前に、２つの保護操作を実行する（１つはホストが書き込みを転送する前、２つ目はその後）必要があることである。代替的にリクエスタは、書き込みの分散完了を開始する前に、ホストへの書き込みの継続と完了を許可することができる。かかるアプローチは、二重の保護操作を回避する。

In one aspect, the requester may protect the dirty copy of the block as soon as it is received from the directory, and then initiate a write distribution completion before accepting new data from the host. This approach has the advantage that the DMG lock and remote invalidation for the sharer is completed as soon as possible, thereby reducing the likelihood of preventing subsequent distributed operations on the page. If this approach is flawed, the writer needs to perform two protection operations before completing the write to the initiator (one before the host transfers the write and the second after). That is. Alternatively, the requester can allow continuation and completion of writes to the host before initiating write distribution completion. Such an approach avoids double protection operations.

(Collection during closure migration)
When the open delta is closed, a new delta is opened to collect the next batch of new writes. Since the delta closure pipeline transition is not an instantaneous operation, there is a transition period in which new writes are treated separately. Specifically, the write data is normally accepted into the cache, but the confirmation of the completion of writing to the host is delayed until the transition period ends.

  A delta closure is a state transition that moves the delta of a WOF pipeline from an open state to an exchange state. Events that can trigger delta closure operations include normal timers with variable intervals, node memory constraints, external trigger APIs, and recovery from system error conditions. Any node can be a source of closure triggers, but timer triggers should only come from a single designated node. Since multiple nodes independently make the decision to trigger a delta closure asynchronously, the closure barrier mechanism should be able to withstand duplicate triggers. Triggers for delta closure are described in more detail elsewhere herein.

  The delta closure can be tuned in one aspect with a distributed barrier mechanism such as a two-stage commit protocol. A barrier mechanism according to one embodiment includes multiple stages. One such stage is a barrier start stage where messages are broadcast to all nodes of the WOF group. The message can start at any node, which then becomes the leader of the remaining barrier round. If there is a race condition and multiple nodes are broadcasting a barrier start notification, the first one will be the winner (using an order broadcast service such as virtual synchronization).

  Another stage is the barrier confirmation stage, where each member of the WOF group sends a point-to-point message to the current round leader when that member reaches the barrier. Since the barrier confirmation message can carry a data payload, information related to the barrier can be shared without additional communication overhead.

  Yet another stage is the barrier end stage, where the leader sends a broadcast message when all the incomplete barrier confirmation messages have been collected. The barrier end message includes the merged data from the barrier confirmation message, if applicable.

(Use barrier)
In one aspect, the WOF implementation has a strict pipeline that proceeds in a lockstep fashion, so there is only a single barrier to control the pipeline progression for each WOF group. A barrier can be started at any node by a delta closure trigger, and that node becomes the next round barrier leader by broadcasting a barrier start message to all nodes. As each node receives this broadcast, it can increment the global delta id, which pushes all new write requests to the next delta. The node can delay confirmation of completion of a new write request to the host.

  The node can wait until all in-progress write requests are completed in the recently opened delta and, if necessary, can wait for the current exchange and commit stage to complete. The node can then notify the barrier leader that this node is ready to continue by sending a message.

  The exchange phase can be made more efficient by including information such as the size of the partial delta in the barrier confirmation message. However, by delaying write completion, the duration can be minimized because the barrier can affect the host application. Once the barrier leader has collected all the barrier confirmation messages, it can broadcast a barrier end message. As each affected node receives this notification, they accept all writes that should have been completed to the new open delta to the host and complete successfully. Further writing can be allowed. The node can initiate an exchange protocol for the previous open delta and begin a commit protocol for the previously exchanged delta, as will be described below.

(Exchange stage)
In one aspect, two things happen when the delta is in an exchange state. First, each site's partial delta is transferred to all other participating sites, so that each site has a complete copy of the delta. Second, make each copy permanent or secure before each site begins committing to backend storage.

  The degree of safety may range from an unprotected delta that is vulnerable to any failure to a mirrored on-disk journal. Journaling not only protects against node and site failures, but also means that data can be safely evicted. However, evicting a safely exchanged block from the cache results in a performance penalty of reading back from the journal during the commit phase. Due to this performance penalty, an implementation that keeps the entire commit delta in memory until committed to storage is preferred in at least one embodiment. In such an implementation, the journal is write-only unless it is needed for disaster recovery purposes.

  One approach to addressing the safety question is to take the middle with n-way protection. However, the problems with this approach include the high memory consumption of the replication approach. Every site needs enough memory to store n duplicate copies of everything, not just two complete deltas that are in exchange and committed with partially open deltas. Furthermore, in a system where the site has a disproportionate number of nodes, the smaller site has a smaller memory pool for use in the WOF pipeline, so the larger site uses some of the memory available to the WOF. It must be left unused. This problem occurs in any in-memory solution, but is exacerbated by the use of additional memory in the protection solution.

  In one aspect, if the node receiving the write is unable to allocate protected space for the block being written, a simple approach to handle the write is to immediately push the written block to disk; In other words, this individual writing is processed by write-through. However, that approach violates dependency write ordering, so protected space must be reserved in advance. After a node failure, a larger protected space must be obtained to re-protect insecure data or remain in degraded mode while the pipeline is flushed. The protection approach does not provide security guarantees for a single node site and loses data integrity after a site failure or some n node failures. With the journaling approach, data integrity is never lost.

  For these and other reasons, journaling is preferred for delta safety in at least some embodiments. Every node can use a small amount of protected space to keep its partially open delta safe from node failure. If the block is secured in the journal at all sites during the exchange phase, the protected copy is no longer needed and can be invalidated and reused. Simple node failures are handled by reprotecting insecure data, reading a secure delta from the journal, and restarting the exchange phase. More serious failures that lose all copies of unjournaled blocks must be treated like link failures, and all sites continue to write their commit deltas atomically, but open and The exchange data is discarded and the host application may need to be restarted. The failure processing will be described in more detail later.

(Memory supply)
In one aspect, each node at each site with a leg of a distributed RAID 1 (“DR1”) implementation participating in the WOF group may have memory provisioned for the entire WOF pipeline. The cumulative in-site memory usage for the WOF group described herein includes the total memory usage M and the memory W allocated to the WOF group.

一態様において、ＷＯＦグループを作成するとき、管理者は、全参加サイトに適用するＷを指定する。他のＷＯＦグループがすでに使用可能なスペースを消費しているため、またはその他の理由で要求されたＷの値が割り振れない場合、ＷＯＦグループ作成は失敗し得、他の以前に作成されたＷＯＦグループが、正常に操作を継続することになる。あるいは、ＷＯＦグループは不十分なリソースで作成されてもよいが、他の使用からリソースを取り戻しを始める。十分なリソースが取得されたとき、新しいＷＯＦグループは、操作を開始することができる。ストレージシステムは、長時間経って十分なリソースが収集できない場合、管理者にＷＯＦグループ作成をキャンセルするオプションを提供してもよい。

In one aspect, when creating a WOF group, the administrator specifies W that applies to all participating sites. If other WOF groups are already consuming available space, or if the requested W value cannot be allocated for other reasons, WOF group creation may fail and other previously created WOFs The group will continue to operate normally. Alternatively, the WOF group may be created with insufficient resources, but begins to reclaim resources from other uses. When sufficient resources are acquired, the new WOF group can begin operation. The storage system may provide an option to the administrator to cancel WOF group creation if sufficient resources cannot be collected over a long period of time.

At the n-node site, M is evenly distributed across the participating nodes and each node i is limited to p _i = P / n for its partial open delta. Furthermore, the exchange protocol distributes the full delta E / n to each node, and the RMG is most likely to place a replica on the neighboring nodes, ie about R / n ends up at each node.

  From the value provided for W, the maximum size of P can be calculated, depending on the number of sites s in the WOF group. If there are more sites, a higher percentage of W is used by a fully exchanged delta. If the same upper limit is used for P at every site, the size of the fully exchanged delta can be defined as:

従って、オープンデルタの最大サイズは、以下のように定義できる。

Therefore, the maximum size of the open delta can be defined as follows:

この値は、Ｐの理論上の最大値である。複製目的の制約は、オープンデルタをさらに制限し得る。サイトの各ノードｉは、オープンデルタのｐ_ｉを収集する責任がある。そのデータは、ジャーナル上で安全にされるまで複製され得る。ＷＯＦグループのためのサイト内複製要件は、サイトの各ノードに対する複製の必要性の合計によって定義される。

This value is the theoretical maximum value of P. Constraints for replication purposes can further limit open deltas. Each node i of the site is responsible for collecting the open delta p _i. The data can be replicated until secured on the journal. The intra-site replication requirement for a WOF group is defined by the sum of replication needs for each node in the site.

各ノードは、現在のオープンデルタの内容のみでなく、交換段階の最後に安全にされるまで、最新に閉じられたデルタの内容も複製する。

Each node replicates not only the contents of the current open delta, but also the contents of the most recently closed delta until it is secured at the end of the exchange phase.

ＷＯＦグループのための複製スペースは、事前割り振りすることができ、簡単に使用できるので、複製要求は失敗しない。しかし、理想的な割り振りは、常に可能なわけではない。Ｉ／ＯがＷＯＦ区画で開始する前に、ＲＭＧはｒ_ｉを予約するよう求められ、要求されている全てが予約できない場合、ｐ_ｉはそれに応じて減らされる。ｐ_ｉの修正された値は、それからデルタクロージャトリガーに対するスペース制約として使用することができる。

The replication space for the WOF group can be pre-allocated and easily used so that replication requests do not fail. However, an ideal allocation is not always possible. Before the I / O starts at the WOF partition, the RMG is asked to reserve r _i, and if not all requested are reserved, p _i is reduced accordingly. modified values of p _i may then be used as a space constraint for the delta closure trigger.

(Memory reservation)
In one implementation, memory for the WOF pipeline, including one for the associated protected copy, is reserved in advance so that normal WOF operations do not have to deal with temporary memory shortages. In an alternative implementation, it would be possible to allocate memory for the WOF pipeline.

(Exchange protocol)
The exchange protocol can ensure that all sites have a local copy of the full exchange delta. During the delta closure protocol, each node can build a descriptor for the data in its partial delta and a second descriptor for the associated metadata, each of which is sent to the round leader with a barrier check . The round leader then distributes these descriptors to all nodes participating in the WOF group in a barrier end broadcast. Each participating site node then uses these descriptors to fetch the missing delta fragment from that site. In one implementation, these descriptors are implemented as keys for the Remote Direct Memory Access (Remote Direct Memory Access, RDMA) area.

  Two tokens, one for exchange and one for security (eg journaling), can be circulated through the nodes of the site. One by one, the node can obtain an exchange token and use the previously communicated descriptor to fetch data from another site until the node exchange area is full. Since all nodes have received the same descriptor in the barrier end broadcast, each node can just start, excluding the previous node, and continue in sequence through the region. Since the node exchange area may fill up after only partially transferring the remote area, the fetching node may cause the transfer to stop at the block boundary. When data is received, the data can be divided into blocks, which are assigned the appropriate delta id and added to the correct delta list as described above.

  After finishing the exchange part, the node passes an exchange token to its neighbors. A safety token follows the exchange token. Nodes may secure their exchanged data (and their local partial deltas) in chunks because they do not have to wait until the entire node exchange area is full. The security token may only be used for the security protocol, but it needs to be serialized like a disk journal. After securing the exchange, the node passes a security token to its neighbors. When the last node releases the security token, the exchange phase ends and the pipeline can proceed. When the next barrier start broadcast is received, the descriptor of the previous exchange round can be discarded and a new descriptor can be created for the next exchange.

(Safety protocol)
In one aspect, the safety protocol is responsible for ensuring that the exchanged data is secure before the commit phase. If journaling is used to achieve this goal, the administrator can provide local disk space for journaling when the WOF group is created. It is wise to use mirrored disks to reduce the possibility of journal failure. A very small space is required for the journal. For example, two fully exchanged deltas are sufficient. Once the delta is committed, the journaled version is no longer needed.

  The following diagram summarizes the format of the on-disk journal J.

ジャーナルのそれぞれの名前付けされた要素は、ブロック境界で開始し、整数のブロックを占有する。典型的な名前付けされたジャーナルのセグメントの内容は以下のとおりである。
ＪＭ（ジャーナルメタデータ）：以下の通り、関連付けられたＷＯＦグループを識別し、現在のコミットデルタのスタートである。

Each named element of the journal starts at a block boundary and occupies an integer number of blocks. The contents of a typical named journal segment are as follows:
JM (journal metadata): identifies the associated WOF group as follows and is the start of the current commit delta.

Δ_ｉ（デルタ）：デルタを定義するために必要とされる全てのデータおよびメタデータを包含する。
ΔＳ（デルタスタートメタデータ）：以下のフィールド^１を^-含む、ジャーナルされたデルタの開始のマーカー。

Δ _i (delta): encompasses all data and metadata needed to define a delta.
[Delta] S (delta start metadata): field ¹ below ^- including, markers of the start of the journal delta.

ΔＥ（デルタエンドメタデータ）：以下のフィールドを含む、オンディスクデルタの終了のマーカー：

ΔE (Delta End Metadata): On-disk delta end marker, including the following fields:

Ｃ_ｉ（デルタチャンク）：デルタのデータのサブセットと、障害後にジャーナルからチャンクを回復するする必要があるメタデータを表わす。
ＣＳ（チャンクメタデータヘッダ）：デルタチャンクの開始をマークし、以下のフィールドを含む。

C _i (delta chunk): represents a subset of the data in the delta and the metadata that needs to be recovered from the journal after a failure.
CS (Chunk Metadata Header): Marks the start of a delta chunk and includes the following fields:

データ（チャンクデータ）：ｃｈｕｎｋＢｌｏｃｋｓのメタデータと同じ順序の、このチャンクに対応するブロックデータ。
ＣＥ（チャンクメタデータトレーラ）：デルタチャンクの終了をマークし、以下のフィールドを含む。

Data (chunk data): Block data corresponding to this chunk in the same order as the chunkBlocks metadata.
CE (Chunk Metadata Trailer): Marks the end of a delta chunk and includes the following fields:

（ジャーナル進行）
ジャーナルは、交換段階中に書き込める。ＷＯＦパイプラインが進行するとき、ジャーナルはデルタ切り替えも実行する。各ノードがバリア終了ブロードキャストを受信すると、それは、グローバルパイプライン進行の永続的な指示として、ＪＭのｃｏｍｍｉｔＤｅｌｔａＯｆｆｓｅｔおよびｃｏｍｍｉｔＤｅｌｔａＩｄのフィールドを更新して、最新にジャーナルされたデルタを指し示すことができる。全てのサイトは、この指示を同時に書き込むので、各サイトの全てのノードは、バリアを出るときに書き込みを実行することができる。かかる要件で、書き込みは全サイト障害によってのみ割り込むことができる。その場合、全ての存続するサイトは、ビットマップログの使用により、その現在のコミットデルタにおける全ての書き込みを保護するべきである。

(Journal progress)
The journal can be written during the exchange phase. As the WOF pipeline progresses, the journal also performs delta switching. As each node receives a barrier end broadcast, it can update the JM commitDeltaOffset and commitDeltaId fields to point to the most recently journaled delta as a permanent indication of global pipeline progress. All sites write this indication at the same time, so all nodes at each site can perform the write when they exit the barrier. With such a requirement, writes can only be interrupted by all site failures. In that case, all surviving sites should protect all writes in their current commit delta by using bitmap logs.

  Since the purpose of the exchange phase is to create a replica of the partial delta, a simple fault handling mechanism simply restarts the exchange phase. If the delta is secured but not fully committed, data can be read back into memory from the journal.

  Journal disk failures are rare because they are supported to be mirrored. If the journal disk fails, a temporary drop mode occurs. The WOF guarantee is lost only if a node or site failure occurs before the situation is corrected. The degradation mode can be terminated in at least two ways. First, journaling can be resumed if the administrator can correct the problem that caused the disk failure. Alternatively, by reducing the maximum size of the open delta, the WOF pipeline is gradually flushed and switched to write-through (or write-back, depending on administrator preference) until the journal disk is normal.

(Flow control)
Although the network-attached site should obviously provide enough bandwidth to accommodate the average write throughput rate, the system according to one embodiment has been expanded to withstand bursts of I / O traffic. At least two strategies can be used independently or together.

  In a first exemplary strategy, the system can maintain a queue of deltas that are closed but not yet replaced, or “before replacement”. If block replacement during the exchange phase does not complete before the next barrier triggers pipeline progress, or if the memory allocated for the delta collection phase is found to be insufficient to cause pipeline progress You can open a new open delta to accept new writes. Recently closed deltas can be kept in a first-in first-out (FIFO) queue of a “delta set before exchange”. When the exchange phase is complete, the next old pre-exchange delta can be advanced to the exchange phase. Thus, the system can provide a buffer for short bursts of write traffic that exceed the capacity of the network.

  This concept can be extended in one aspect by combining two or more pre-exchange delta sets when the pre-exchange delta queue becomes long. Combining pre-exchange deltas in this embodiment is done in the same way at all nodes of the system so that the resulting larger delta set represents a consistent write time range throughout the system. When combining two or more deltas, the union of all written blocks can be used. If each block has been written more than once at a given node and has been reborn into one or more delta sets, the latest reborn (the block image of the most recent pre-exchange delta set) can be used. . Combining delta sets reduces the number of times a commonly written block is sent, creates a larger stream of transfers in the exchange phase, potentially resulting in higher network efficiency, and larger time intervals and data The volume has the advantage of amortizing the cost of the barrier operation. A potential disadvantage is that there is an adjustment cost to trigger and manage the combination of pre-exchange delta sets, and this extension has to be restarted from the previous “post-exchange” delta image. If a node is lost when it must, the amount of data that can be lost may be slightly increased. This can therefore be used advantageously as a recovery mechanism when the system is behind due to a long queue of delta sets before exchange.

  In a second exemplary strategy, the rate at which write data is accepted from the host into the WOF group is reduced or “suppressed”. In this method, a delay is inserted before writing confirmation to the host. The delay is increased or decreased to reflect the extent to which the system lags behind the exchange of delta sets. Slowing down individual writes tends to average the write performance so that the delivered performance returns to match the current network capacity.

  By using both methods, it can withstand short bursts without loss of system performance, and at the same time allow the mechanism to persistently exceed writes without damaging applications with short write timeouts. A solution according to one embodiment is provided.

  Another solution basically treats the box as DR1 through every stage of the dependency WOF protocol except commit. In the commit phase, the node does nothing but does not allow the data to be evicted and the data is retained until the data is replicated to the passive site. Then, if there is a site disaster at the active site, the view of the passive site storage host seen through the node front end will be consistent. At that point, the saved delta can be written to back-end storage. One potential drawback of this approach is that dirty data can be transferred twice over the inter-site link (once to exchange partial deltas, once the active SRDF site pushes the batch to the passive site. When).

(WOF subcomponent)
The WOF component in one aspect consists of a number of individual subcomponents. One such subcomponent is referred to herein as a “wof server”. A simple version of the wof server can be used, but the wof server can be responsible for group changes and fault handling, or simply provide an NMG broadcast mechanism. Messages sent through this service can be processed by all nodes, but the “wof client” code can determine group membership on the fly and act accordingly. The wof client can provide a way for the local cache to register the AMF partition and obtain the WOF group handle. The following API will suffice.

別の副構成要素は、ＣＯＭとＮＭＧとを両方使用する分離汎用バリア機構であるが、これについては後述する。このバリアは、ＷＯＦ構成要素内でのみ使用される。また別の副構成要素は、デルタＩＤジェネレータである。グローバルデルタｉｄジェネレータは、ロールオーバおよびデルタ世代比較をサポートできる。これは以下のようなＡＰＩをキャッシュにエクスポートできる。

Another subcomponent is a separate universal barrier mechanism that uses both COM and NMG, which will be described later. This barrier is only used within the WOF component. Another sub-component is a delta ID generator. The global delta id generator can support rollover and delta generation comparisons. This can export the following API to the cache:

トリガー機構をキャッシュと分離させておくために、ＷＯＦ構成要素は、定期タイマ、メモリ制約、またはユーザコマンド（ＷＯＦトリガー）に依存した、全てのトリガー決定を担当することができる。単一のサイトで交換される部分デルタのためにスペースを予約する必要がないので、最初のマイルストーンにおけるメモリ制約は任意に選択できる。これは以下のようなＡＰＩが使用できることを意味する。

In order to keep the triggering mechanism separate from the cache, the WOF component can be responsible for all trigger decisions that depend on periodic timers, memory constraints, or user commands (WOF triggers). Since there is no need to reserve space for partial deltas exchanged at a single site, the memory constraints at the first milestone can be chosen arbitrarily. This means that the following API can be used.

最後の関数は厳密なパイプラインで使用され、それによりＷＯＦはクロージャバリアを通過する適切な時期まで待つことができる。交換プロトコルが存在する後のマイルストーンでは、同様の関数呼び出しを使用して、交換完了を通知できる。

The last function is used in a strict pipeline so that the WOF can wait until the appropriate time to cross the closure barrier. At milestones after the exchange protocol exists, a similar function call can be used to signal the exchange completion.

(AMF abstraction)
An abstraction layer can be used on top of the AMF for delta writing. This may allow the WOF to distinguish between local AMF and DR1, and may handle bitmap logging if necessary. The following API additions can be used to be called by the cache at the amf_write location:

（デルタセット概念の拡張機能）
以下の章は、クラスタコントローラ、地理的ストレージ、キャッシュコヒーレンシ、および仮想化についての以下の特許および特許出願に開示されるもの（これらは参照により、全体として本書に組み込まれる）などの技術を拡張する。
６，１４８，４１４；６，９１２，６６８；６，８５７，０５９；５，８７５，４５６；６０／５８６，３６４；ＵＳ−２００３−０１８８６５５−Ａ１；ＵＳ−２００１−００４９７４０−Ａ１；およびＵＳ２００５−００７１５４５Ａ１。

(Extension of the delta set concept)
The following chapters extend technologies such as those disclosed in the following patents and patent applications on cluster controllers, geographic storage, cache coherency, and virtualization, which are incorporated herein by reference in their entirety: .
No. 6,148,414; 6,912,668; 6,857,059; 5,875,456; 60 / 586,364; US-2003-0188655-A1; US-2001-0049740-A1; and US2005-0071545A1 .

(Active / passive support)
As described above, the dependency WOF implementation allows active access to data at one or more sites, so that any site actively reads data that is asynchronously distributed between the sites, or Can write. One approach to extending the delta set concept is that only one of the many possible writers is actually actively writing to a given WOF group for any given volume at any given point in time. Recognize that can be. Multiple node reads and single node writes for a period are equivalent cases. If this condition can be detected dynamically, the storage system can be optimized to reduce the cost of delta broadcasts and to minimize the amount of data lost due to a system restart on the previous delta.

  If only one site is writing (instant primary site), the WOF solution can function like the previous WOF solution. In one example, the instantaneous primary site A 802 can survive and site B 804 fails the situation 800 of FIG. Site A can continue processing data without interruption or data loss. The main improvement over previous WOF solutions is that other sites can continue to read active data with guaranteed data coherency. This concept incorporates and extends the concept of coherency between nodes within a site and across geographies.

  Another advantage is that the definition of which site is primary can be very dynamic. A site can be an instantaneous primary site if it is the only site that writes to any asynchronous delta set (open or closed). This implementation may require the site to broadcast that the partial delta set is dirty upon the occurrence of its first write to the new delta set. Writes that dirty a partial delta set can be deferred until all sites acknowledge the notification. Thus, the surviving site will know how many delta set levels, if any, must be set back to ensure data integrity. If not, processing can continue without application restart or data loss. Even if the surviving site cannot be declared as a momentary primary site, data loss is minimized by retreating only the last delta set that may be considered a momentary primary site.

(Participating node without DR1 local leg)
The above example outlines participating nodes at various sites, each site having a DR1 local leg. The concept of fully active WOF does not require that all nodes actually have local backend data storage (DR1 local leg). This is particularly useful when the satellite site wants to access the data in a read / write manner without the cost of maintaining a full copy of the locally mirrored data volume.

  One embodiment allows such “satellite nodes” to join a WOF group. In this case, the satellite node creates an open delta and manages incoming writes in exactly the same way as the node at the site with the local DR1 leg. Satellite nodes participate in barrier operations in exactly the same way as other nodes. However, during the exchange phase, the satellite site only needs to participate in half, and changes from other participating nodes do not need to be copied to the satellite. Similarly, satellite nodes do not need to participate in the commit phase.

  Note that a given node can participate in satellite operations (without a DR1 leg) for some WOF groups managed by that node, and still maintain DR1 for others.

(Explicit passive site)
Another enhancement involves dynamically determining that a site is not writing and then making that site a passive site until the first write is detected from that site. With a write from a configured “passive” site, a message can be broadcast to all sites, so that they know that this previous passive site is now an active participant. In one aspect, the system should wait for a few partitions to pass, such as a 3-5 delta rollover, and a site not writing during that period should be considered a passive site until a later write. Judge that. The cost of such an approach includes the need to broadcast when the site becomes active again, so it is desirable to set a site as passive only if the site is likely to remain passive for at least some period of time.

  In operation, it may be useful to explicitly determine that a site is passive, by configuration or operator command. This may eliminate the need to be included in the first write broadcast. In addition to operational advantages, this reduces the latency of the first write to the partition.

  There are complex scenarios of sites that have potentially multiple sites that are explicitly declared as passive. Thus, notifications between implicit passives are made to determine the instantaneous primary site, as described in the previous chapter. Sites can also be declared as “explicitly active” with the same effect.

(Partial delta set synchronization after failure)
One presumed result of maintaining the “dirty delta set” bit described above is whether synchronization of a partial delta set (asynchronous delta set) can minimize data loss, and in some cases instantaneous It is possible to immediately determine whether a target primary site situation can be established.

  As shown in situation 900 of FIG. 9, the loss of site C 906 does not mean a loss of data or a loss of application backup, but this does not synchronize deltas D2 and D3 between sites A 902 and B 904. , To allow both A and B to become instantaneous primary sites. If the passive writer fails, no data is lost and the system can simply continue.

(Synchronous delta set)
The WOF described here can handle both asynchronous data transfer and synchronous data transfer. WOF can also handle a combination of synchronous and asynchronous data transfer. For example, when the transfer distance to the data center is about 100 km, the light speed waiting time is not so long. Since it is desired to have two protected copies of the data image at any given time, when the system writes to the host, the write is immediately inserted into the synchronous delta set. If there are two sites participating synchronously and a third site participating asynchronously in the middle of the continent, writing can be done from the host, and the block can be instantly just the local delta set. Not even the synchronization partner. The system then returns and confirms the write, which represents the safety of the written data. When replicating data with a synchronous delta of 100 km away, when the contract is complete and the write is confirmed, the system must not only exist at the local site in cache format but also 100 km away in cache format Is shown. It will happen sometime later that asynchronous images cross the country. From that point of view, for example, if all data storage on the west coast is lost, the data is fragile, but if only one area is lost, a copy in another area is safe. Every write is synchronously mirrored between the two sites, so one site may be lost, but the other site with a copy of the data still uses the data asynchronously across the country Can be pushed to.

  In some embodiments, a group of synchronous delta sets is declared so that writes to any open partial delta set are synchronously written to other partial delta sets that exist within a common delta set synchronization group It will be convenient. In one aspect, write replication can be implemented in a manner consistent with the delta set concept. For purposes of this description, write replication refers to placing write dirty data in two or more independent pools of cache memory before returning “write complete” to the host, which means that for any given node. Its purpose is to protect the data from loss due to failure. In another aspect, delta set synchronization groups implemented across geographic groups can be a convenient way to improve site fault tolerance. For example, two sites that are relatively close together can be declared as members of a synchronous delta set group, while other sites in other geographic regions have their own groups. Thus, any one site can be lost without data loss or operational interruption. At the same time, the impact of latency between regions is minimized.

  In the schema 1000 of FIG. 10, the two sites 1002, 1004 on the west coast are declared as members of the first delta set synchronization group 1010. Similarly, two sites 1006, 1008 on the east coast are declared as members of the second delta set synchronization group 1012. A write to any given partial delta set is replicated synchronously with another partial delta set in the delta set synchronization group (replication completes before the write returns on completion). Data is distributed throughout the continent using a normal post-close delta set data push operation. This also benefits from a persistent view of the delta set described below. As with other delta set behaviors, delta set synchronization groups can be defined on virtual volumes by virtual volume criteria.

(Cascading synchronous delta set replication)
Since the delta set that receives the write “pushes” the data to the other delta set, the relationship need not necessarily be symmetric. For example, box “A” may be required to replicate synchronously to box “B”, but “B” is “A” except through the exchange of partial delta information as part of normal post-close operations. May not have a request to replicate.

  This also allows cascade operation. For example, writing to the partial delta “A” results in synchronous writing to “B” and “C”, which goes from “B” to “D” and “E”, and from “C” to “F”. And synchronous writing to "G". An example of when this is useful is when writing is spread across multiple nodes within a site and then across multiple sites.

(Closed delta set is cache-safe)
As described above, completion of a contract for writing indicates that the data can be considered operationally safe. In traditional single site storage systems, administrators make conscious choices that balance performance with data safety. Specifically, the administrator selects from among write-through, write-back, and cache copy write-back. Cache replication writeback can be extended with the possibility of write n-way replication for additional protection.

The delta set structure allows a similar level of operational flexibility at the trade-off when the delta set is secure. Lineage includes that the notion of a delta set is considered “safe” when any of the following is true:
• All dirty data is written to all disk-resident mirror images.
Dirty data is written to the local disk and remote site cache.
Dirty data is replicated between a partial delta set of n caches at the local site and m caches at each of the p sites.
Dirty data is replicated to n caches at the local site.
A delta set is considered safe as soon as it is closed.

  In the above, the concept of replicating into n caches means exchanging between partial delta sets located in each of the n caches. There is no requirement that the delta set be an exported host for all these nodes. Some instantiation of partial delta sets can be used to purely protect dirty data from node failure.

(Snapshot and delta set integration)
As described elsewhere in this document, a snapshot refers to a logical point-in-time image of a real data set. Snapshots can be used for functions such as maintaining a backup window. When performing a backup, it is not desirable to shut down the storage system for a long time, for example, to back up data. What is desired is a point-in-time image of all data, which ensures that the data is always consistent across the backup tape.

  The snapshot is desired to be time consistent, so that the snapshot should reflect point in time. Furthermore, for some applications such as databases, the point-in-time should correspond to a point where the application is safe. For example, the database can commit after a series of reads and writes to erase the data in the database and commit the data to the database. In this case, the snapshot may correspond to a commit point. The commit can be performed when a confirmation response is returned from the storage system in which the last writing has been performed for the commit point. Agents or other triggering events can be used to trigger a snapshot, as is known in the art. Today's snapshots are typically implemented at the storage system layer.

  These snapshots can be advantageously combined with a delta set. The point-in-time image of the data needs to be WOF consistent even if the storage is distributed across the geographical area. In one aspect, the snapshot can be triggered by an agent or timer, for example corresponding to point-in-time. As soon as the snapshot is received, a rollover of the delta set can be triggered, resulting in an intra-domain point-in-time image corresponding to the desired snapshot point. The system can be started with new I / O's, allowing its delta set to go through to the commit point. When the commit point is reached and the open delta is written, the delta goes through the delta set pipeline. When all writes are complete, the system triggers a snapshot image on itself. In one aspect, the snapshot can be triggered to the underlying storage device, but this actually takes a physical snapshot based on what is on the disk. In another aspect, the delta set can be kept relative to its point-in-time.

  Each completed delta set represents a volume of data at a consistent point in time. A logical snapshot (a point-in-time logical image of a volume) is implemented by simply providing an indication to the previous delta set and allowing the exported volume to be based on a consistent data image when the delta set is closed Can do.

  It is important to allow snapshots to be triggered by an external source such as an application. For this reason, an interface for closing a delta set as described below can be provided.

  The snapshot can also be an exported read / write. From the delta set point of view, this creates a family of delta sets, each representing a lineage that develops from the origin of the delta set. From the user's point of view, the results look exactly like the previous logical snapshot family.

  Starting as a logical snapshot, an image that creates a physical (not logical) copy of the data by background copy can also be embedded. This can be done by first creating a logical snapshot as described above and then “cloning” the higher delta set to a separate media in the background.

(Extension to Delta Set disk image)
As described elsewhere in this document, the system in one aspect may have three delta sets that are open at any given time, each delta set having a different point-in-time. Represents. One delta set represents the point-in-time at which the data is finally committed to disk, one represents the point-in-time at which the data is about to be committed to disk, and one represents the point-in-time at the beginning of the exchange. Represent. There may also be many open deltas throughout the system. Instead of creating a single large base image that collapses all these deltas, the system can have many open deltas and have many views of various point-in-time data. In such a system, if it is desired to go back to the appropriate point in time, the system can easily go back to the appropriate delta by indicating the point in time t and the appropriate image.

  In one aspect, implementations can utilize techniques such as constant data protection (CDP). Many delta sets, each representing a point-in-time, are created, thereby displaying a picture of the data at that point-in-time by reading any given delta and the later delta of that delta So that CDP can be implemented. A delta set is connectivity in the sense that as the delta set becomes older, the need for fine-grained point-in-time decreases. As such, delta sets can be folded in a combined manner and combined in a manner that coarsens over time, reducing the overall amount of storage used.

  The snapshot concept can be extended to CDP, where delta sets can be used to begin merging delta sets that change in granularity over time. For this purpose, the delta can be retrieved from the cache and written to disk. This allows creation of delta sets stored in cache and disk, i.e. extending delta set storage on disk. By doing so, the system can implement both snapshot and CDP format functionality. Both delta sets and partial delta sets can be accommodated in random access memory or disk (or any medium in that respect).

(Merge delta set (using delta set connectivity))
If a delta set is considered to be a series of point-in-time images of data (snapshots), the lineage of delta sets is associative. Old delta sets can be merged without changing the view exported by the newer delta set.

  This allows the creation of a family of point-in-time images that represent relatively fine time increments at more recent time points. As the delta sets become “stale”, the delta sets can be merged together to create coarser time increments and reduce the overhead associated with maintaining point-in-time images.

  The process of “merging” delta sets will be apparent to those skilled in the art in light of the teachings and suggestions contained in this document. A “union” operation is possible except that there are changes to a given block in multiple delta sets and the latest changes can be used.

(Remote importer)
Sites that import WOF storage remotely but do not have a local DR1 leg may have special behavior. Like other full-fledged participating sites, these sites contribute to s to calculate P. In one aspect, both P and R are calculated correctly, but E = C = 0 because there is no local storage to write. The node at the importing site does not participate in the exchange or commit phase beyond supplying its RDMA key with a confirmation barrier message during delta close.

(Yotta disk and delta set integration)
As mentioned in this document, and as disclosed in US Pat. No. 6,857,059 (February 15, 2005), entitled “Storage virtualization system and methods”, Yota Disk is, for example, presented to the host, such as end user, any large size up to (for example, 10 ²⁴ bytes), a virtual disk image demand is mapped. In one embodiment, for example, a virtual disk image is used to create a mapping from the virtual disk image to the backend physical storage, which is the result of an I / O operation such as a write operation performed on the physical storage. As done dynamically. Storage remapping allows back-end storage to be managed without impact to the user, and multiple back-end partitions can be combined to provide a single virtual image. The disk image presents a potentially very large image to the user, isolates the user from the volume resizing problem, and allows for a generous amount of usage. This image may be supported by a management system that controls usage and growth rates and provides the ability to maintain core system processes such as creation, deletion, and attachment of other candidate disks.

  Yotta disks can be implemented with delta sets using a similar mechanism. Since storage may need to be allocated dynamically, a backend storage allocator can be used for both processes to allow the yotta disk and delta set to be implemented at about the same time. The delta set uses a mechanism described in US Pat. No. 6,857,059 (incorporated herein by reference), for example, a sparse disk image (where demand is mapped and released based on block references). sparse disk image). The difference is the system of time representation provided by the delta set.

(Close delta set dynamically)
Several mechanisms can be provided to close the delta set and thus open a new delta set. Such mechanisms include, for example:
• Set time interval • Time interval based on the number of transactions • Time interval based on the number of changes (writes) in the delta set or partial delta • Application triggers • Operator triggered triggers • Triggers triggered by other subsystems In the case of triggers triggered by error conditions and the above complex time intervals, the size of the interval is adjusted according to the changing conditions. For example, an overloaded network can be a reason for increasing the lifetime of a delta set to reduce its impact on the network. The “advanced alert” period may trigger a finer delta set to reduce the likelihood of data loss during periods of high data dependence.

  If any node can potentially trigger a delta set turnover, these triggers need not be consistent across the various nodes.

(Time Firewall)
Minsen Ji et al., “Seneca: remote mirroring done write” (USENIX Technical Conference summary page 253-256, June 2003) presented the concept of a time firewall. One embodiment described herein extends such a concept to include both a time-delayed read-only view of the data volume and a multi-writer fully active geographically distributed access view to the current data view. . In another aspect, various WOF optimizations are integrated with the concept.

  One of the concerns of interconnecting multiple sites is that logical errors (as opposed to physical failures) can quickly propagate between sites. For example, a virus inserted at one site quickly infects all sites sharing the data image. The point-in-time / delta set concept described above can provide an efficient mechanism for providing a “safe” window to the deployment of data.

  Sites operating in an active / passive manner can provide a read-only portal to data based on point-in-time images (delta sets) that lag behind active data. Read-only images can proceed automatically with delta sets that maintain a default “safe” time interval. In this way, normal processes continue to run on “real” data that can detect logical failures or corruption of the data. If such an event is detected, the progress of the “safe” read-only image is interrupted until the problem is corrected.

  The “remote” site (or any site) has a current delta set image that works like any other multi-writer delta set, and a “safe” window to the previous open delta set at the same time. Can have.

  In a specific example, two entities want to share data freely, but may want to avoid a situation where a virus is inserted into one data of one entity and then spread to another entity. Absent. Thus, in this example, it is not desirable to have just one large data repository. A solution according to one embodiment allows one of the entities to write only to the partition and then export the data to another agency, so that each entity has a different view of the data. One viewpoint is a synchronized image of data as if there were just two normal WOF sites. At some time delay point there is a second volume that is delayed for some time (such as 30 minutes) that is considered a safe view of that data. If something happens to one entity, the system can simply interrupt the progress of the save pointer until the problem is repaired and the virus is removed. There can be both a real-time image of the data and an image with a time delay of at least 30 minutes. A delayed image can update itself, such as with a delta set closure, to keep itself delayed by approximately 30 minutes, and indicate that something or someone has a problem and the update should be aborted As long as you don't, you can continue the time. The delayed image can then remain at a safe point until instructed otherwise.

(Nested delta set)
If there are two sites that are relatively close compared to other sites, a denser transfer of data can be performed between the adjacent sites, such as for synchronous replication. Delta sets can be nested so that pairs of neighboring sites can make frequent exchanges, but this is in a “meta” delta set closure. This meta delta set exchange can be finer in granularity than a normal delta set exchange. More frequent exchanges can provide more frequent updates at a relatively low cost due to the proximity of the site.

  Nesting delta sets allows a subset of nodes to switch sub-delta sets so that nodes that are geographically closer are more synchronized with a denser delta set than is used at a greater distance Can do. For example, 1100 in FIG. 11 illustrates the grouping of two geographically close sites into nested groups 1102. This helps minimize the chance of data loss and maximize the chances of a momentary primary site after a failure.

(RAID of the entire delta set)
A delta set implementation can mirror the changes between all nodes by exchanging elements of the partial delta set between all nodes after closing the delta set. Making changes "safe" by replicating between nodes does not have to mean just mirroring. For example, some RAID arrangement pattern may be used. In this case, “D” does not refer to the disk image, but refers to a node instance in the cache or on the disk.

  A “safe” arrangement should also recognize the physical reality. For example, multiple nodes in a site may need only one copy between them, and mirrors and other duplicate copies may be stored at another site.

  For example, if there are five sites that are geographically separated and performing a delta set exchange, rather than mirroring RAID 1 at all five sites, a RAID 5 type implementation can be used to store data in several subsets. This allows any site to be lost while still having access to the data. As is known in the art, RAID 5 includes functions such as data striping and parity checking so that no site has a full set of data. The exchange part of the delta set can be implemented as a RAID write. Each block need not be sent to every other host, but instead can be routed based on RAID striping. At any given time, data may only exist at two sites or three sites, or at one site and checksum. This reduces the need for a complete broadcast of all data.

(Redefine storage primitives using delta sets)
Conventional storage tiering includes, in order, cache, cache replication, virtualization, and conventional RAID. This can be replaced with another layering, including a coherency layer, a delta set layer, and then a physical resource allocator that indicates where everything is located. Combining the above, it is possible to redefine the storage archive with respect to the primitives surrounding the delta set.
So instead of the traditional stratification below,
・ Cache ・ Cache replication ・ Virtualization ・ Conventional RAID
These primitives can present a new layering as follows.
• Coherence layer • Delta set with properties and origins • Physical resource allocator that maps delta sets to physical devices The functionality of the various embodiments can be achieved by any suitable combination of hardware and software known in the art. Can be implemented. For example, software and logic can be stored as a plurality of instructions or program code on an information storage medium contained within or outside various components, accessories, and / or devices. Storage media and computer readable media for containing code or portions thereof may include any suitable media known or used in the art, including EEPROM, flash memory, or other Memory technology, CD-ROM, ROM, RAM, digital versatile disc (DVD), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage device, data signal, data transmission, or Implemented in any method or technique for storage and / or transmission of information such as computer readable instructions, data structures, program modules, or other data, including any other media accessible by the computer Volatile and non-volatile, although such removable and non media are not limited to, include various storage media and communication media. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will appreciate other means and / or methods for implementing various example aspects.

  Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. However, it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the claims.

FIG. 1 illustrates a distributed storage system that can be used in accordance with one embodiment of the present invention. FIG. 2 illustrates process steps for use by a multi-site storage system in accordance with one embodiment of the present invention. FIG. 3 illustrates process steps for use by a multi-site storage system in accordance with one embodiment of the present invention. FIG. 4 illustrates process steps for use by a multi-site storage system in accordance with one embodiment of the present invention. FIG. 5 illustrates the steps of a method for storing data, according to one embodiment of the present invention. FIG. 6 illustrates process steps for use by a multi-site storage system in accordance with one embodiment of the present invention. FIG. 7 illustrates process steps for use by a multi-site storage system in accordance with one embodiment of the present invention. FIG. 8 illustrates process steps for use by a multi-site storage system in accordance with one embodiment of the present invention. FIG. 9 illustrates process steps for use by a multi-site storage system, in accordance with one embodiment of the present invention. FIG. 10 illustrates a geographically separated distributed storage system that can be used in accordance with one embodiment of the present invention. FIG. 11 illustrates a distributed storage system including nested groups that can be used in accordance with one embodiment of the present invention.

Claims (39)

A method for providing write order fidelity to a set of distributed data access nodes in a network comprising:
Storing a write request in a first cache of open deltas, the first cache corresponding to a first node receiving the write request;
In response to a trigger event, sending a message to each node of the set of data access nodes to close the open delta;
For each node that has a write request for the open delta, complete any pending write requests for the open delta and close the open delta;
Exchanging write request information between nodes, whereby each node is associated with a complete copy of the write request information for the closed delta;
Writing a complete copy of each to persistent storage;
Storing each complete copy of the closed delta in back-end storage of each node.   2. The method of claim 1, further comprising the step of grouping front end volumes into write order fidelity (WOF) groups, each WOF group including at least one site via the network. the method of.   The method of claim 2, wherein each site includes at least one of the nodes and all nodes are operable to read and write in parallel.   The method of claim 2, wherein a write for each WOF group is stored in the open delta cache for that WOF group.   The method of claim 1, further comprising executing the trigger event.   The method of claim 1, further comprising opening a new delta and receiving a new write when closing the open delta.   The method of claim 1, further comprising providing localized cache access to remote data of geographically distant nodes.   The writing of each complete copy to persistent storage includes the step of writing the metadata update entry to the recovery log when the metadata update entry has been written and writing the data update to the database. The method described in 1.   The method of claim 1, further comprising reordering write requests in an open delta before closing the delta.   The method of claim 1, further comprising triggering a data snapshot corresponding to closing the open delta.   Set time interval, time interval based on the number of transactions, time interval based on the number of changes / writes in a delta set or partial delta, application trigger, operator triggered trigger, trigger triggered by another subsystem, error condition The method of claim 1, further comprising generating the trigger event using a mechanism selected from the group consisting of triggers triggered by and combinations thereof.   The method of claim 1, wherein the network is selected from the group consisting of a storage area network (SAN), a local area network (LAN), a wide area network (WAN), and a metropolitan area network (MAN).   The method of claim 1, wherein sending a message comprises broadcasting the message to each node of the set of data access nodes. A system for providing write order fidelity to a set of distributed data access nodes in a network comprising:
A storage system for storing data;
A plurality of access nodes configured to access data of the storage system;
Each node of the plurality of access nodes is operable to store a write request in an open delta cache, the cache corresponding to the node receiving the write request, each node responding to a trigger event And is further operable to send a message to the plurality of access nodes in the data storage network to complete all write requests and close the open delta, each node exchanging write request information And is thereby further operable to associate each node with a complete copy of the write request information for the closed delta, each node storing a respective complete copy in persistent storage and then the closed delta Each persistent copy of the backend for that node Further operable, the system to apply to storage.   The system of claim 14, wherein at least one of the nodes is geographically remote from other nodes.   The system of claim 14, wherein at least one of the access nodes is operable to mirror data to a remote location.   The system of claim 14, further comprising a plurality of sites, each site including at least one of the nodes.   The storage system is further operable to determine when only one site is writing to the storage system so that exchange of write request information is deferred until multiple sites write to the storage system The system of claim 17, wherein   The storage system is further operable to determine when one of the sites is not writing data to the storage system, and sets the site as a passive site until the site needs to write The system of claim 14, wherein the system is operable.   The system of claim 19, wherein the storage system is further operable to broadcast a message to other sites to indicate the status of the passive site.   The system of claim 14, wherein the storage system is operable to perform asynchronous and synchronous data transfer.   Each node is operable to exchange write request information by exchanging write request information with a first subset of the plurality of access nodes, whereby the first subset of nodes The system of claim 14, exchanging the write request information with a second subset of access nodes.   The system of claim 14, wherein each node is operable to replicate writes to multiple caches.   The data storage system maintains three deltas, each delta at the point in time when the data is finally committed to disk, the point in time just before the data is committed to the disk, and at the start of the exchange The system of claim 14, wherein the system represents one of point-in-time.   The system of claim 14, wherein the storage system is further operable to merge deltas over time.   The system of claim 14, wherein the storage system is further operable to create a snapshot of any open delta at any point in time.   The storage system has a set time interval, a time interval based on the number of transactions, a time interval based on the number of writes in a delta set or partial delta, an application trigger, an operator triggered trigger, a trigger triggered by another subsystem 15. The system of claim 14, further operable to close the delta set using a mechanism selected from the group consisting of: a trigger triggered by an error condition, and combinations thereof.   The system of claim 14, wherein the network is selected from the group consisting of a storage area network (SAN), a local area network (LAN), a wide area network (WAN), and a metropolitan area network (MAN).   The system of claim 14, wherein each node is further operable to send the message to the plurality of access nodes by broadcasting the message to each node of the set of data access nodes. A computer program product embedded in a computer readable medium for providing write order fidelity to a set of distributed data access nodes in a network comprising:
Computer program code for storing a write request in a first cache of open deltas, the first cache corresponding to a first node that receives the write request;
Computer program code for sending a message to each node of the set of data access nodes to close the open delta in response to a trigger event;
For each node that has a write request for the open delta, computer program code to complete any pending write requests for the open delta and close the open delta;
Computer program code for exchanging write request information between nodes, whereby each node is associated with a complete copy of the write request information for the closed delta;
Computer program code to write a complete copy of each to persistent storage;
A computer program product comprising computer program code for storing a complete copy of each of the closed deltas in the back-end storage of each node.   Computer program code for grouping front end volumes into write order fidelity (WOF) groups, each WOF group further comprising computer program code for grouping including at least one site via the network 32. The computer program product of claim 30, comprising.   32. The computer program product of claim 30, further comprising computer program code for storing a write to each WOF group in the open delta cache for that WOF group.   32. The computer program product of claim 30, further comprising computer program code for providing localized cache access to remote data of geographically distant nodes. A method for providing write order fidelity to a set of distributed data access nodes in a network comprising:
Providing a plurality of write order fidelity (WOF) groups, each WOF group including at least one of the data access nodes;
Storing the write request in a first cache corresponding to a first node in a first WOF group that receives the write request;
In response to a trigger event, exchanging write request information between the nodes of the first WOF group, whereby each node is associated with a complete copy of the write request information;
Writing a complete copy of each to persistent storage;
Storing a complete copy of each in the backend storage of each node.   35. The method of claim 34, further comprising providing localized cache access to remote data of geographically distant nodes.   35. Writing each complete copy to persistent storage includes writing the metadata update entry to a recovery log when writing the metadata update entry is complete and writing the data update to the database. The method described in 1.   35. The method of claim 34, further comprising triggering a data snapshot corresponding to the state of the cached write request for the first WOF group. A method for providing write order fidelity to a set of distributed data access nodes in a network comprising:
Providing a plurality of write order fidelity (WOF) groups, each WOF group including at least one of the data access nodes;
Storing write requests in a cache corresponding to one of the plurality of WOF groups, wherein each WOF group is associated with a cache and has access to at least one node of the WOF group. A step operable to receive a write request from a writer;
In response to a triggering event for a WOF group, exchanging write request information between the nodes of the WOF group, whereby each node is associated with a complete copy of the write request information;
Storing each complete copy in persistent storage. A computer program product embedded in a computer readable medium for providing write order fidelity to a set of distributed data access nodes in a network comprising:
Computer program code providing a plurality of write order fidelity (WOF) groups, each WOF group including at least one of the data access nodes;
Computer program code for storing a write request in a cache corresponding to one of the plurality of WOF groups, each WOF group being associated with a cache and having access to at least one node of the WOF group Computer program code operable to receive a write request from a request writer of:
Computer program code for exchanging write request information between the nodes of the WOF group in response to a trigger event for the WOF group, whereby each node is associated with a complete copy of the write request information;
A computer program product comprising computer program code for storing a complete copy of each in persistent storage.

Images