WO2001035244A1 - Flexible remote data mirroring - Google Patents

Flexible remote data mirroring Download PDF

Info

Publication number
WO2001035244A1
WO2001035244A1 PCT/US2000/005443 US0005443W WO0135244A1 WO 2001035244 A1 WO2001035244 A1 WO 2001035244A1 US 0005443 W US0005443 W US 0005443W WO 0135244 A1 WO0135244 A1 WO 0135244A1
Authority
WO
WIPO (PCT)
Prior art keywords
mirroring
data
local
remote
unit
Prior art date
Application number
PCT/US2000/005443
Other languages
French (fr)
Inventor
Ron Mccabe
Robert Church
Tracy Camp
Stuart W. Card
David J. Schroeder
Original Assignee
Miralink Corporation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Miralink Corporation filed Critical Miralink Corporation
Priority to CA 2427920 priority Critical patent/CA2427920A1/en
Priority to EP00923070A priority patent/EP1240591A4/en
Priority to AU43252/00A priority patent/AU4325200A/en
Publication of WO2001035244A1 publication Critical patent/WO2001035244A1/en
Priority to HK03102020.7A priority patent/HK1051417A1/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0626—Reducing size or complexity of storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1446—Point-in-time backing up or restoration of persistent data
    • G06F11/1448—Management of the data involved in backup or backup restore
    • G06F11/1451—Management of the data involved in backup or backup restore by selection of backup contents
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
    • G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
    • G06F11/2058—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring using more than 2 mirrored copies
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
    • G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
    • G06F11/2069—Management of state, configuration or failover
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0614—Improving the reliability of storage systems
    • G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
    • G06F3/065—Replication mechanisms
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/0671—In-line storage system
    • G06F3/0683—Plurality of storage devices
    • G06F3/0689—Disk arrays, e.g. RAID, JBOD
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00—Arrangements for detecting or preventing errors in the information received
    • H04L1/08—Arrangements for detecting or preventing errors in the information received by repeating transmission, e.g. Verdan system
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L1/00—Arrangements for detecting or preventing errors in the information received
    • H04L1/12—Arrangements for detecting or preventing errors in the information received by using return channel
    • H04L1/16—Arrangements for detecting or preventing errors in the information received by using return channel in which the return channel carries supervisory signals, e.g. repetition request signals
    • H04L1/18—Automatic repetition systems, e.g. Van Duuren systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
    • G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
    • G06F11/2064—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring while ensuring consistency
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
    • G06F11/2056—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring
    • G06F11/2071—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant by mirroring using a plurality of controllers
    • G06F11/2074—Asynchronous techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/835—Timestamp
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/84—Using snapshots, i.e. a logical point-in-time copy of the data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/855—Details of asynchronous mirroring using a journal to transfer not-yet-mirrored changes

Definitions

  • the present invention relates to the remote mirroring of digital data from a server or other computer in order to provide better fault tolerance and/or disaster recovery, and relates more particularly to tools and techniques for increasing the flexibility of remote data mirroring by permitting its use in a wider variety of network configurations than those previously used
  • United States Patent No 5,537,533 describes tools and techniques for remote mirroring of digital data from a primary network server to a remote network server
  • a system includes a primary data transfer unit with a primary server interface and a primary link interface, and a remote data transfer unit with a remote link mterface and a remote server interface
  • the primary link interface includes a spoof packet generator capable of generating a pre-acknowledgement for the primary network server That is, the system has a "smart buffer" which gives the primary server a pre- acknowledgement or "spoof after mirrored data has been stored on a nonvolatile buffer in the primary link interface and before an acknowledgement arrives indicating that the mirrored data has been stored by the remote server
  • Off-SiteServer product includes technology to remotely mirror the disks of a Novell NetWare server to another server at a geographically remote location through a low bandwidth telecommunications link (NETWARE is a mark of Novell, Inc )
  • Remote mirroring of data from a primary network server to a remote replacement network server using data mirroring is a powerful and efficient method to back up data
  • Remote mirroring creates a copy of data at a safe distance from the original data and does so substantially concurrently with the storage of the original data
  • the remotely stored data can be available almost immediately after a disaster if it was copied to a "warm" remote network server, that is, a remote server which can be up and running as the new primary server within minutes of the actual or simulated disaster
  • a "warm" remote network server that is, a remote server which can be up and running as the new primary server within minutes of the actual or simulated disaster
  • use of the Off-SiteServer product involves a pair of Off- SiteServer boxes; one is a local box and the other is a remote box.
  • the Off-SiteServer boxes are configured with specialized hardware and with firmware and/or other software, generally as described in U.S.
  • a proprietary serial line connects the local NetWare server to one of these boxes.
  • the NetWare server itself uses a Vinca card (VINCA is a mark of Vinca Corporation). This card is driven by a NetWare Loadable Module (“NLM”) that intercepts disk-driver requests, and sends data down the serial line to the local Off-SiteServer box.
  • VINCA Vinca card
  • NLM NetWare Loadable Module
  • the local Off-SiteServer box has a 4 Gigabyte nonvolatile buffer, such as an IDE disk drive. Data is pre-acknowledged into this Off-SiteServer buffer. As far as the operating system of the local server is concerned a second '"mirrored" write has occurred locally. In reality, the Off-SiteServer product has received this data from the NLM and stored it on the local buffer. The local Off-SiteServer box stores sector and track (or block level) data changes until it can safely send them to the remote Off-SiteServer box at the remote location.
  • the buffer in the local Off-SiteServer box is also "smart" in that it stores any data above what the telecommunications link can handle locally.
  • This data is stored in the local Off-SiteServer box until the remote Off-SiteServer box has successfully written to the remote secondary server and sent back an acknowledgement to the local (primary) Off- SiteServer box.
  • the local Off-SiteServer box frees the space in the local nonvolatile buffer that is occupied by the successfully transmitted piece of sector/track/block data.
  • the Off- Site Server product uses a V.35 interface for data output at the local (primary) site.
  • V.35 is a serial telecommunications standard that connects to a Channel Service Unit/Data Service Unit ("CSU/DSU"), which in turn interfaces with the telecommunications link.
  • CSU/DSU Channel Service Unit/Data Service Unit
  • the remote (secondary) location has a second CSU/DSU that relays the sector/track/block information to the V.35 input interface of the remote secondary Off- Site Server box.
  • the secondary Off- Site Server box outputs this sector/track/block data through the proprietary serial connection using a serial cable connected to another Vinca card in the secondary (remote) server.
  • the remote server ' s data mirroring and system software then writes this sector/track/block information to the remote server ' s disk drive and the write is acknowledged back to the local Off-SiteServer box.
  • This system is capable of handling about 300 megabytes of change data in an hour.
  • the Off-SiteServer product is intelligent enough to sense if there is a decrease or increase in bandwidth and/or if the telecommunications link has gone down. During link downtime periods, the Off-SiteServer box can store data changes from the server in the local nonvolatile smart buffer. When the link is active again, the Off-SiteServer product starts transmitting automatically. The Off-SiteServer product can change its bandwidth output on the fly as bandwidth becomes more or less available. All of the transmissions described above also incorporate standard software checksum error detection and correction, and/or hardware error correcting code ("ECC”) error handling.
  • ECC hardware error correcting code
  • a secondary (remote) server attached to a remote (secondary) Off-SiteServer box in the manner just described has a complete mirrored disk copy of all the data on the local (primary) server.
  • This remote backup copy can be restored back to the local (primary) server.
  • This secondary remote server can also stand in for the local primary server in the event of disaster. Such a secondary restoration and/or stand-in can be executed relatively quickly with a simple set of command lines.
  • the Off-SiteServer product and other remote data mirroring technologies provide valuable fault-tolerance and disaster recovery capabilities, both to mission-critical data and in other contexts. Nonetheless, these existing approaches have unnecessarily limited flexibility.
  • the Off- Site Server product requires a specific version of hardware and software from Vinca Corporation. This required version of the Vinca product does not support any operating system/file system platform other than the Novell NetWare platform. The hardware component of the necessary Vinca package also does not work with newer, faster servers and larger disk volumes.
  • the original Off-SiteServer product was also designed to connect one local server to one remote server. Only a single server can mirror to a remote server at a given time. Multiple servers at different locations cannot readily mirror to a single remote site. Likewise, if an enterprise has multiple local servers running different operating systems and/or file systems, each server running a separate platform must be mirrored to a matching remote server.
  • the original Off-SiteServer product requires an NLM on the local server, and it was designed to use private dedicated telecommunications links.
  • Conventional mirroring also requires a remote server in order to keep mirrored information in a bootable format at the remote location.
  • the present invention provides tools and techniques for flexibly mirroring data.
  • the invention permits the use of various combinations of one or more external storage units and/or RAID units to hold mirrored data.
  • the invention provides many-to-one data mirroring, including mirroring from local servers running the same or different operating systems and/or file systems at two or more geographically dispersed locations.
  • the invention also provides one-to-many data mirroring, mirroring with or without a dedicated private telecommunications link, and mirroring with or without a server at the destination(s) to assist the remote mirroring unit(s). Spoofing, SCSI and other bus emulations, and other tools and techniques are used in various embodiments of the invention.
  • the invention does not require a secondary server at the remote location in order to mirror data.
  • a remote server may be used to test the integrity of mirrored data, or to replace a local server which becomes unavailable, but the remote server is not needed to maintain a complete copy of mirrored information in a bootable format at the remote location.
  • a flexible local mirroring unit mirrors a local disk volume to a remote location via a journey link, which may be part of a local area network, part of the Internet, a low bandwidth telecommunications link, or a high bandwidth dedicated telecommunications link such as a Tl link.
  • the local mirroring unit utilizes the bandwidth efficiently using an intelligent buffer with spoofing, as described in United States Patent No. 5,537,533, for instance.
  • the local mirroring unit is non-invasive of the host operating system. It is not necessary to install on the mirrored local host an NLM or other software designed specifically for remote data mirroring. In particular, larger host volumes than before can be mirrored without degrading performance of the mirrored host, because the load on the host CPU is not substantially increased by mirroring according to the present invention.
  • the local mirroring unit instead of in the host server also increases reliability and flexibility by making it possible to reconfigure or even reboot the local mirroring unit without interfering with host server processing.
  • the local mirroring unit appears to be simply some familiar type of disk subsystem. Accordingly, standard mirroring tools and techniques can be used within the local server to direct a copy of the data to the local mirroring unit, for subsequent forwarding (unbeknownst to the local server) to a remote mirroring unit that may be tens or hundreds of miles away.
  • Figure 1 is a diagram illustrating prior art mirroring in a network of computers which could also be adapted for use with the present invention.
  • Figure 2 is a diagram illustrating a computer system according to the invention, without a remote server, but including a remote mirroring unit having a large buffer.
  • Figure 3 is a diagram illustrating a computer system according to the invention, including a remote server with a hot-swappable RAID unit and a remote mirroring unit having a relatively small buffer.
  • Figure 4 is a diagram illustrating a computer system according to the invention, without a remote server, but including a remote mirroring unit having a relatively small buffer and a hot-swappable RAID unit.
  • Figure 5 is a diagram illustrating a computer system for many-to-one mirroring according to the invention, without a remote server, but including several local servers running a given platform with respective local mirroring units and a single remote mirroring unit having a relatively small buffer and several hot-swappable RAID units.
  • Figure 6 is a diagram illustrating another many-to-one computer system according to the invention, without a remote server, but including several local servers running a given platform with respective local mirroring units and a single remote mirroring unit having a relatively small buffer and several individual external storage volumes.
  • Figure 7 is a diagram illustrating another many-to-one computer system according to the invention, without a remote server, but including several local servers running a given platform with respective local mirroring units and a single remote mirroring unit having a relatively small buffer, an external storage volume having several partitions, and a hot- swappable RAID unit likewise having several partitions.
  • Figure 8 is a diagram illustrating another many-to-one computer system according to the invention, without a remote server, but including several local servers running different platforms with respective local mirroring units and a single remote mirroring unit having a relatively small buffer and several hot-swappable RAID units.
  • Figure 9 is a diagram illustrating another many-to-one computer system according to the invention, without a remote server, but including several local servers running different platforms with respective local mirroring units and a single remote mirroring unit having a relatively small buffer and several external storage volumes.
  • Figure 10 is a diagram illustrating another many- to-one computer system according to the invention, without a remote server, but including several local servers running different platforms with respective local mirroring units and a single remote mirroring unit having a relatively small buffer, an external storage volume having several partitions, and a hot-swappable RAID unit likewise having several partitions.
  • Figure 11 is a diagram illustrating a one-to-many mirroring computer system according to the invention, in which a local server is connected to several local mirroring units for data mirroring to several remote locations.
  • Figure 12 is a diagram illustrating an alternative one-to-many mirroring computer system according to the invention, in which a local server is connected to one multi-ported local mirroring unit for data mirroring to several remote locations.
  • Figure 13 is a flowchart illustrating methods of the present invention.
  • Figure 14 is a diagram illustrating a dual host configuration between a remote mirroring unit, a remote server, and a RAID unit, which may be used in performing a switchover according to the invention.
  • the present invention relates to computer systems, methods, and configured storage media for flexible data mirroring.
  • the invention provides non-invasive mirroring, mirroring with or without a dedicated private telecommunications link, and mirroring with or without a dedicated server or another server at the destination to assist the remote mirroring unit.
  • the invention also provides many-to-one data mirroring, including mirroring from local servers running the same or different operating systems and/or file systems at two or more geographically dispersed locations.
  • the invention provides flexibility by permitting the use of various combinations of one or more external storage units and/or RAID units to hold mirrored data.
  • the invention may be embodied in methods, systems, and/or configured storage media. Unless clearly indicated otherwise, discussions of any one of the embodiment types also apply to the other embodiment types. For instance, the discussions of inventive systems will also help one understand inventive methods for configuring such systems and/or methods for sending data through such systems to have the data mirrored. Computers and Networks Generally
  • Figure 1 illustrates a network 100 in which a local server 102 is mirrored over a conventional route 104 to a remote server 106.
  • the conventional route 104 is not limited to telecommunication links themselves, but also includes modems, data transfer units, and other conventional tools and techniques used to send data on such links and/or to receive data thus sent.
  • the conventional route 104 may include the server interfaces, link interfaces, and DTUs which are illustrated in Figure 1 of U.S. Patent No. 5,537,533 and discussed in that patent.
  • the conventional route 104 may include Small Computer System Interface (“SCSI”) performance extenders or standard Storage Access Network (“SAN”) connectors.
  • SCSI Small Computer System Interface
  • SAN Storage Access Network
  • Such devices require a very high bandwidth link and minimal latency. They tend to have distance limitations of perhaps ten or twenty miles because distance introduces latency. For instance, in a single mode fiber configuration the latency on a given SCSI extender might allow a distance of perhaps fifteen kilometers between the data source and destination. Using a multi-mode fiber would reduce the distance available to perhaps two- thirds of that because of latency.
  • Such connections have little or no tolerance for delays or interruptions longer than a few fractions of a second, or at best can only gracefully handle delays of a few seconds.
  • the network 100 as shown is configured for mirroring according to conventional tools and techniques, it is also one of the many possible networks suitable for adaptation and use according to the present invention.
  • adaptation would include various steps, depending on the particular embodiment of the present invention to be used. For instance, adaptation could involve disconnecting the remote server 106 if it is no longer needed, supplementing or replacing the conventional mirroring route 104 with mirroring units linked according to the present invention, unloading mirroring NLMs or other specialty software from the local server 102, adding more local servers which will also be mirrored, and/or adding remote storage in the form of external storage volumes and/or Redundant Array of Independent Disks ("RAID " ) units.
  • RAID Redundant Array of Independent Disks
  • the adaptation normally involves the addition of at least one local mirroring unit and at least one remote mirroring unit, with the remote mirroring units capable of being linked to each other for operation according to the present invention.
  • the network 100 may be connectable to other networks 108, including LANs or WANs or portions of the Internet or an intranet, through a gateway or similar mechanism, thereby forming a larger network.
  • the local server 102 is connected by communication links or network signal lines 110 to one or more network clients 112.
  • Other suitable networks include multi-server networks and peer-to-peer networks.
  • the server(s) 102 and client(s) 112 in a particular network may be uniprocessor, multiprocessor, or clustered processor machines.
  • the server(s) 102 and client(s) 112 each include an addressable storage medium such as random access memory.
  • Suitable network clients 112 include, without limitation, personal computers; laptops 114, personal digital assistants, and other mobile devices; and workstations 1 16.
  • the signal lines 1 10 may include twisted pair, coaxial, or optical fiber cables, telephone lines, satellites, microwave relays, modulated AC power lines, RF connections, a network link, a dial-up link, a portable link such as an infrared link, and/or other data transmission "wires " or communication links known to those of skill in the art.
  • the links 110 may embody conventional or novel signals, and in particular, may embody a novel series of commands and/or data structures for mirroring data as discussed herein.
  • the remote server 106 may store mirrored data it obtains over the conventional route 104 on an attached storage means such as an external hard disk and/or RAID subsystem 118. Examples of Flexible Mirroring Unit Systems
  • Figure 2 illustrates the present invention in systems according to the invention. Unlike previously discussed conventional approaches, systems according to this Figure do not require a remote server.
  • a local server 200 or some other host 200 communicates over a local link 202 with a local mirroring unit 204.
  • the local mirroring unit 204 communicates over a journey link 206 with a remote mirroring unit 208.
  • Local mirroring units may include a spoof packet generator for pre-acknowledging data to the local server 200. and a nonvolatile data buffer 210 for holding mirrored data before it has been stored at the remote location.
  • Remote mirroring units have a destination nonvolatile storage for mirrored data received from the local mirroring unit(s) 204 over the journey link(s) 206.
  • the remote mirroring unit may be physically separated from a local server 200 by various distances, such as under ten miles, at least ten miles, or at least one hundred miles. These distances are mere examples: because the present invention can take full advantage of journey links(s) 206, systems according to the invention have no inherent distance limitations. Individual mirroring units are discussed in greater detail below, both in connection with illustrations of their flexibility in example systems shown in Figures 2 through 12, and in connection with their components and operation in general.
  • local mirroring units 204 include SCSI emulation software and/or hardware, allowing the local link 202 to be a SCSI connection through which the local mirroring unit 204 appears to the local server 200 or other host 200 as a SCSI disk or other conventional SCSI device. This may be accomplished by using within the local mirroring unit 204 a SCSI host adapter that is running in target mode instead of the more usual initiator mode. Suitable SCSI host adapters having such a target mode include at least the Adaptec 2940UW adapter, and the QLogic QLA-1040 adapter.
  • the local link 202 can be a fibre channel connection, a Universal Serial Bus (“USB”) connection, a mainframe channel extender, a V.35 CSU/DSU connection, a FireWire (IEEE 1394) connection, a memory type (for instance, the AS/400 mirrors memory, not disk), an IDE bus.
  • a PCMCIA connection a serial connection, an Ethernet connection, a Fiber Distributed Data Interface (''FDDI") connection, or another standard bus for connecting a disk and/or a Redundant Array of Independent Disks (“RAID”) subsystem to a server.
  • conventional mirroring in the sense of copying to another local disk
  • hardware and/or software can be used within the local server 200. as if the mirrored data were simply being sent to another local disk instead of being sent across the journey link 206 to a remote location.
  • the journey link 206 need not be a dedicated private telecommunications link. Although such a link may still be used in some embodiments, the invention also provides mirroring units 204. 208 which communicate over a network, or a series of networks such as the Internet, using Ethernet. FDDI. V.35. or other data link protocols, the Internet Protocol (IP) or other network protocols, and/or the User Datagram Protocol (UDP). Transmission Control Protocol (TCP), or other transport protocols, without regard for the routability or non- routability of such protocols. Accordingly, the two mirroring units 204. 208 may be separated by many tens or hundreds of miles if so desired.
  • the journey link 206 can be fed through a conventional link 104 and a spoofing local mirroring unit 204 as the data acquisition point.
  • the journey link 206 does not necessarily impose requirements of high bandwidth and low latency, which are often imposed by conventional links 104.
  • a system using the journey link 206 can send mirrored data from a source to a destination which is an unlimited distance away.
  • the journey link 206 can also provide shared bandwidth, as it typically will when crossing the Internet or a wide area network.
  • the journey link 206 and/or the mirroring units provide inventive systems with the advantage of a relatively high tolerance for interruptions and disconnects.
  • the illustrated remote mirroring unit 208 has a large buffer 212.
  • the remote mirroring unit 208 can buffer a complete volume of the local server 200 or other host 200.
  • the local mirroring unit 204 also includes a large buffer.
  • the local server 200 volume and the large buffers (local and remote) can each hold up to one terabyte of data in nonvolatile storage.
  • This buffering may be accomplished, for instance, by using the QLogic QLA-1040 adapter within the local mirroring unit 204 or the remote mirroring unit 208 to control up to one terabyte of data with no substantial modifications needed.
  • the complete volume image of the local server 200 can therefore be stored on the buffer(s) within the mirroring unit(s).
  • an optional local mirror 230 may also be created: this is generally a "full " local mirror in the sense that it is consistent and available but not necessarily entirely up-to-date.
  • This local mirroring may be accomplished in various ways. These include, without limitation, using a second local mirroring unit 204 or a second port of a multi-ported local mirroring unit 204 to mirror data to a "remote" disk subsystem that is actually geographically close to the local host 200; forking the data within the local mirroring unit 204 below the disk emulation layer of that unit 204, thereby creating another copy which is sent to a local attached disk subsystem over a SCSI or similar bus (the first copy is sent over the journey link 206 to a remote mirroring unit); and using otherwise conventional tools and techniques with the local mirroring unit 204 to create and maintain a local mirror 230.
  • the mirror 230 includes a copy of the server 200 volume to permit recovery in the event of hardware or software errors.
  • the local mirror 230 is local rather than remote, it does not provide substantial protection against natural disasters, civil unrest, terrorist attacks, physical vandalism, and other geographically localized risks to the server 200. Accordingly, the local mirror 230 does not provide the same degree of data protection as remote mirroring even if the local mirror 230 includes another mirroring unit 204 or otherwise embodies the present invention.
  • the local mirror 230 is connected to the mirroring unit 204 by a path 232 which may include a conventional link such as the path 104, or a novel link according to the present invention.
  • local mirror 230 is not explicitly shown in the other Figures, one or more local mirrors may also be used with the systems illustrated in the other Figures and with other systems according to the invention.
  • one approach uses Nonstop Networks Limited's technology or other technology to mirror between two servers; the local mirroring unit is used as the sole
  • Figure 3 illustrates systems in which a local server 200 communicates over a local link 202 with a local mirroring unit 204.
  • the local mirroring unit 204 communicates over a journey link 206 with a remote mirroring unit 308.
  • the remote mirroring unit 308 has only a relatively small nonvolatile buffer 310, such as a buffer 310 holding only a few gigabytes, e.g., four gigabytes.
  • systems according to Figure 3 include a remote server 300 which has an associated nonvolatile internal or external storage.
  • Figure 3 shows a RAID unit 312 which can be controlled at some point by the remote server 300.
  • the RAID unit 312 is "hot-swappable," meaning that a failed drive in the RAID unit 312 can be taken out and replaced while the computer 300 is running; the file system structures and other data on the replacement drive will then be built automatically.
  • the RAID unit 312 can be viewed in some cases as part of the server 300 or connected thereto by conventional means such as means which include dedicated mirroring software on the server 300. as indicated by the arrow in Figure 3 from the RAID unit 312 to the server 300.
  • the RAID unit 312 may also be connected to the remote mirroring unit 308 and the server 300 by a dual host connection in a configuration 1400 as discussed later below and illustrated in Figure 14.
  • the dual host connection allows a switchover from a first "normal mirroring" state having a passive remote server 300, a remote RAID unit 312 or other remote disk subsystem used only for mirroring, and a local mirror and/or local host 200 disk actively used to service read requests, to a second "recovery" state having an active remote server 300 which services read requests from the mirrored data on the remote RAID unit 312 or other remote disk subsystem.
  • the remote mirroring unit 308 receives data from the local mirroring unit 204 using an Ethernet and/or TCP/IP connection 206, for instance.
  • the local link 202 can be a SCSI bus, USB, fibre channel, or similar connection.
  • the remote mirroring unit 308 transfers the data over a remote link 302 and remote mirroring unit 308 to the remote server 300 for subsequent storage on the hot-swappable RAID unit 312, or directly from the remote mirroring unit 308 to the RAID unit 312 if the dual host connection 1400 is being used.
  • the remote link 302 can be a SCSI bus connection, for instance, so the remote mirroring unit 308 appears to the remote server 300 to be a SCSI disk, for instance, which is to be mirrored by the remote server 300 to another "disk," the RAID unit 312.
  • the remote link 302 can also be a serial, Ethernet. FDDI, USB, fibre channel, or other nonproprietary connection.
  • the local mirroring unit 204 has a nonvolatile buffer which is similar or identical (except with respect to specific data stored in it) to the small buffer 310 of the remote mirroring unit. Data from the local server 200 is pre-acknowledged into the local mirroring unit 204 buffer. As far as the primary server 200 is concerned a second "mirrored" write has occurred locally. In reality, the local mirroring unit 204 has received this data and stored it on this local buffer. The local mirroring unit 204 stores this sector and track change data (or similar block level data) until the local mirroring unit 204 can safely send the data over the journey link 206 to the remote mirroring unit 308.
  • the smart buffer in the local mirroring unit 204 stores any data above what the journey link 206 can handle locally.
  • a remote mirroring unit 408 includes both a small nonvolatile buffer 310 and a large nonvolatile buffer; the large buffer is implemented as a hot-swappable RAID unit 312 which connects directly to the remote mirroring unit 408.
  • the small buffer 310 is used to buffer data received over the journey link 206, allowing the data to be acknowledged back to the local mirroring unit 204, and buffering the data until it can be stored by the remote mirroring unit 408 in the large buffer 312. No remote server is needed.
  • Figure 5 illustrates systems in which two or more local servers 200 write to a remote mirroring unit 508. In this Figure and elsewhere, references to the local server 200 should be understood to also generally include hosts 200 which are not servers.
  • the invention can be used to mirror any host computer system 200 that will connect to a mirroring unit 204.
  • Servers are a widely recognized example of suitable hosts 200, but other suitable hosts 200 include clusters, computers which are not servers, mainframes, and Storage Access Network (“SAN") or Networked Attached Storage (“NAS”) data sources.
  • the local servers 200 or other hosts 200 may be physically separated from one another by various distances, such as under ten miles, at least ten miles, or at least one hundred miles.
  • each local server 200 in a particular system relies on the same operating system and file system platform, but different systems according to Figure 5 may use different platforms.
  • each server 200 could be a Novell NetWare server in one such system
  • each server 200 could be a Microsoft Windows NT server using the NT File System (“NTFS”) in another such system.
  • NTFS NT File System
  • Each host 200 in the system is connected by a SCSI, fibre channel, USB, serial line, or other standard storage subsystem or other peripheral connection 202 to its own local mirroring unit 204.
  • the local mirroring units 204 are connected by journey links 206 to a single remote mirroring unit 508.
  • the remote mirroring unit 508 has a SCSI, fibre channel. USB, or similar controller card for each of the local mirroring units 204.
  • the data from each local mirroring unit 204 can be transferred directly (i.e., not through a remote server) to an individual hot-swappable RAID storage unit 312 in a group 512 of RAID units, by a SCSI, fibre channel, USB, or similar connection within the remote mirroring unit 508.
  • the RAID units 312 may be physically external to at least a portion of the remote mirroring unit 508, such as a portion containing an Ethernet card for connection to the journey link 206.
  • the remote mirroring unit 508 is defined by functionality rather than packaging.
  • the RAID units 312 are considered part of the remote mirroring unit 508 unless indicated otherwise (e.g.. in discussing Figure 14).
  • Each RAID storage unit 312 has a remote bootable volume, and the data is written in sector/track or block fashion.
  • the illustrated remote mirroring unit 508 also contains a small buffer 310 to allow acknowledgment and buffer of data received over the journey links 206.
  • Figure 6 illustrates systems similar to those shown in Figure 5, but a remote mirroring unit 608 writes to external bootable storage volumes 614 in a group 616 of such volumes.
  • Local servers 200 running on the same platform write to "disks" which are actually local mirroring units 204, which in turn write the data to the remote mirroring unit 608.
  • the remote mirroring unit 608 has a SCSI, fibre channel, USB, or similar controller card and a bootable storage volume 614 corresponding to each local mirroring unit 204.
  • each local mirroring unit 204 will be transferred from the remote mirroring unit 608 directly to the corresponding storage volume 614 using a SCSI bus or other data line.
  • Each volume 614 is a remote bootable volume, and the data is written in sector/track or block fashion.
  • separate partitions may be used to hold the mirrored data of respective local servers 200, instead of holding that mirrored data in corresponding separate disks 614 (e.g., as in Figure 6) or separate RAID units 312 (e.g., as in Figure 5).
  • Figure 7 illustrates systems in which a remote mirroring unit 708 includes both an individual external storage volume 614 and a RAID unit 312.
  • the mirrored data is stored by the remote mirroring unit 708 on both storage subsystems 312, 614, to provide extra assurance that the data will be available when needed.
  • Figure 7 also illustrates systems in which two or more local mirroring units 204 write to one remote mirroring unit 708 with all mirrored data for the several local servers 200 going to one large storage volume (312 or 614 or both, in various embodiments) which is mounted directly on the remote mirroring unit 708, instead of dividing the mirrored data among several remote storage units 312 or 614 as illustrated in Figures 5 and 6, respectively.
  • the volume used by the remote mirroring unit 708 has a partition for each local mirroring unit 204. Each partition provides a remote bootable "volume,” and the data is written in sector/track or block fashion as usual.
  • the mirrored data is divided between two or more storage units which are connected directly to the remote mirroring unit 708, with a given storage unit holding the mirrored data for a given local mirroring unit 204.
  • a mixture of external disks 614 and RAID units 312 is used, unlike the systems that use RAID units only ( Figure 5) or external disks only ( Figure 6).
  • an external disk 614 holds the data from a first local mirroring unit 204
  • a RAID unit 312 holds the data from a second local mirroring unit 204.
  • the remote mirroring unit 708 has a SCSI, fibre channel, USB, or similar controller card corresponding to each local mirroring unit 204, and the data from each local mirroring unit 204 will be transferred directly (without a server such as server 300) to an individual external hot-swappable RAID storage unit 312 or external bootable drive 614 via a SCSI, fibre channel, USB, or similar communications line.
  • Figure 8 illustrates systems like those discussed in connection with Figure 5.
  • the local servers 200 rely on different platforms, as indicated by the presence of several numbers 822, 824, 826.
  • systems according to this or other Figures do not necessarily have exactly three local servers 200 and corresponding local mirroring units 204; they merely have two or more pairs, with a server 200 and corresponding local mirroring unit 204 in each pair.
  • one system according to Figure 8 includes a Novell NetWare server 822 and a Microsoft Windows NT server 824
  • another system according to Figure 8 includes two Novell NetWare servers 822, 826 and a Microsoft Windows NT server 824.
  • Figure 9 illustrates systems like those discussed in connection with Figures 5 and 8. Unlike Figure 5, however, the local servers 200 rely on different platforms, and unlike Figure 8, the remote mirroring unit is a unit 608 which uses a group 616 of external disks 614 instead of a group 512 of RAID units 312.
  • Figure 10 illustrates systems like those discussed in connection with Figure 7.
  • the local servers 200 in systems according to Figure 10 rely on different platforms.
  • the local mirroring units 204 may be mapped in some systems to partitions or to storage units. When mapping to partitions, the local mirroring units 204 may be mapped to partitions within a RAID unit 312, to partitions within an external drive 614, or to partitions within a RAID unit 312 which are also mirrored to an external drive 614.
  • one or more local mirroring units 204 may send their data through the remote mirroring unit 708 to corresponding external drive(s) 614 while one or more other local mirroring units 204 send their data through the remote mirroring unit 708 to corresponding RAID unit(s) 312.
  • Figure 1 1 illustrates systems in which data is mirrored to two or more remote locations. Such systems are a counterpart of the systems illustrated in Figures 5-10, in the sense that Figures 5-10 illustrate "many-to-one" mirroring systems (more than one local server mirrored to one remote destination) while Figure 1 1 illustrates "one-to-many” mirroring systems (one local server mirrored to more than one remote destination).
  • the local mirroring units 204 will all be mirroring the same data, but using multiple local mirroring units 204 permits mirroring across at least one journey link 206 to continue uninterrupted despite the unavailability of a given local mirroring unit 204.
  • the local links 202 may all use the same type of connection, or different connections may be used.
  • one local link 202 may be a SCSI connection while another local link 202 is a USB connection.
  • the journey links 206 may also be uniform or varied.
  • the remote mirroring units may each have the same components (e.g., each may use a RAID unit 312), or they may use different components at the different locations.
  • Figure 12 illustrates systems which resemble those illustrated by Figure 1 1 in that data is again mirrored to two or more remote locations.
  • the local mirroring unit 204 of Figure 12 is a multi-port mirroring unit. That is, it can be connected simultaneously to more than one journey link 206 in a manner similar to the simultaneous connection of a conventional multi-port server.
  • the multi-port mirroring unit 204 sends mirror data from the host 200 over each of the active connections 206, thereby helping mirror the host 200 to several remote locations which may be miles apart from one another.
  • the multi-port local mirroring unit 204 needs only one local buffer, and like mirroring units 204 in other systems it optionally includes a full local mirror 230. More on Mirroring Units
  • mirroring units The components and operation of mirroring units are discussed above in connection with Figures 2 through 12. A given piece of additional information provided below does not necessarily pertain to every mirroring unit in every system according to the invention, but this additional information is helpful nonetheless in understanding how the mirroring units permit greater flexibility to the people and enterprises that are responsible for ensuring that data is properly mirrored. At least some of the mirroring units can reliably emulate disk drives connected by
  • SCSI, fibre channel, USB, or similar connections through standard server drivers running under Novell NetWare and/or Microsoft Windows NT platforms.
  • SCSI, fibre channel, USB. or similar emulation under other operating systems may also be provided.
  • Each of the local and remote mirroring units is preferably configured so that it supports I/O through a monitor, keyboard, and a mouse plugged into it.
  • Some mirroring units have a network address and otherwise allow a network administrator to access a specific mirroring unit on the adapted network 100, through a web browser on a remote workstation 1 16 or by other means.
  • the mirroring units are preferably Simple Network Management Protocol ("SNMP") capable.
  • the network administrator has remote access to both the local and remote mirroring units.
  • the mirroring unit 204 software provides an interface to monitoring utilities.
  • each local mirroring unit 204 acts like a network agent in that the unit 204 tracks the number of writes/reads to the local server 200, the status of each local server 200. number of restarts/warm starts of each local server 200, and so forth, and generates SNMP traps when necessary.
  • the following pieces of data may also be provided to administrators by the local mirroring unit 204: the number of blocks currently in the buffer 210: an alert when the buffer 210 fills up and/or fills beyond some specified threshold; the number of blocks sent since server 200 startup; and the number of blocks received since server 200 startup.
  • Some local mirroring units 204 also have incremental dial-up options. If a customer is using the mirroring unit 204 with a dial-up connection, and doesn't want to be connected at all times, the unit 204 provides an option to send data over the journey link 206 at specified times. Also, the local mirroring unit 204 may have a setting that does not allow data to be sent during periods of high traffic on the adapted network 100 or another portion of the journey link 206. The buffer 210 in the local mirroring unit 204 should be large enough to buffer data received from the local server 200 during these periods of non- transmittal. More generally, the local mirroring unit 204 preferably matches the performance of a high-speed RAID disk subsystem in terms of data transfer rates, reliability, and compatibility with existing platforms on servers 200.
  • the local mirroring unit 204 preferably comprises special-purpose hardware.
  • the design and construction of suitable software and hardware, including any necessary firmware, may be accomplished by those of skill in the art, with particular attention to the conventional mirroring path 104: the SCSI controllers identified herein or similar SCSI, fibre channel, USB, or similar controllers; individually known subsystems such as buffers 210, 212. 310, disks 614, and RAID units 312, and their interfaces; software such as FreeBSD drivers; Ethernet and other individually known Network Interface Cards ("NIC"); network protocols such as Ethernet and TCP/IP protocols; the descriptions and examples provided herein; and other tools and techniques now or subsequently available to such persons.
  • the SCSI controllers identified herein or similar SCSI, fibre channel, USB, or similar controllers
  • individually known subsystems such as buffers 210, 212. 310, disks 614, and RAID units 312, and their interfaces
  • software such as FreeBSD drivers
  • network protocols such as Ethernet
  • Writes to the local mirroring unit 204 should normally be acknowledged and written to the local buffer 210, and may also be written to a full local mirrored volume 230 over a conventional path 104 or another path, although such local mirroring is not explicitly shown in Figures 3 through 12.
  • an implementation may take advantage of an available hardware RAID unit 312 cache or other SCSI, fibre channel, USB, or similar cache. Reads from the local mirroring unit 204 should generally be serviced with the proper data from the local mirror 230.
  • the local mirroring unit 204 When the local mirroring unit 204 comes back on-line after a crash or a reboot or any other kind of service interruption, it will automatically begin sending data from its local buffer 210 to the remote mirroring unit 208, 308, 408, 508, 608 or 708.
  • the local mirroring unit 204 should not issue a SCSI, fibre channel, USB, or similar reset, as this may crash the host machine 200.
  • Data written to the local mirroring unit ' s buffer 210 should be sent over the network or other journey link 206 in a first in, first out fashion, to the remote mirroring unit. This may be done using the TCP/IP or another journey link protocol.
  • the remote mirroring unit preferably maintains a full, consistent, mirror so the remote volume is usable and mountable by an operating system at all times regardless of mirror synchronization status.
  • kernel panics should preferably not occur on the local mirroring unit 204 unless there is a failure of essential mirroring hardware or software. Misconfiguration of the local mirroring unit 204 software should preferably not result in a system shutdown, nor should any behavior of the host server 200. It is preferably possible to reconfigure the mirroring unit software without a reboot; a unique version number should accompany each software change. Accordingly, the software preferably reads all initialization information and configures itself accordingly through a system call which is available to an administrator without interrupting data processing by the mirroring unit. The host server 200 should not be interrupted.
  • the local mirroring unit 204 preferably accepts writes from the host system 200 whether or not the remote mirroring unit is on-line, and whether or not network or other journey link 206 bandwidth is available, unless the local buffer 210 is full. If the local buffer 210 fills, the local mirroring unit 204 preferably continues to maintain a local mirror 230 (if present), and preferably continues to dequeue a circular queue of data from the local buffer 210. However, the local mirroring unit 204 preferably stops adding to the queue until told to begin queuing again by a user (typically an administrator) process. A system call, rather than a reboot, preferably allows user-space processes to disable and re-enable local buffer 210 queuing.
  • the mirroring units preferably auto-detect the disappearance of and reconnection of network or other journey link 206 bandwidth. For instance, disconnecting the local mirroring unit's Ethernet cable and then reconnecting it the next day preferably results in zero data loss and requires no intervention on the network operator's part, provided there is enough space in the local buffer 210 to hold the data changes accumulated while the local mirroring unit 204 was disconnected.
  • the local mirroring unit 204 preferably loses as little data as possible in the event of power failure. Some mirroring units therefore contain an Uninterruptible Power Supply ("UPS"). It may be assumed there will be time to flush RAM-buffered writes to the local mirror (if present) and/or local buffer 210 in the event of power loss.
  • the mirroring unit operating system e.g.. FreeBSD
  • FreeBSD boots from the hard disk in a read-only mode to avoid filesystem problems with FreeBSD itself.
  • Configuration data is written to a smaller partition and can be restored either from the identical information on the mirroring unit peer, or by sending out a SNMP alert that the mirroring unit has lost configuration data and will be off line until it is restored.
  • the alert can be used if the peer mirroring unit is not reachable.
  • Some embodiments also avoid controller card initialization routines that disk drives would not perform on their own, to avoid bus resets for instance. Also, if the mirroring unit buffer fills up it may be better to simply acknowledge the write and mirror it locally while sending an alert that the buffer is full and the remote mirror is out of sync with the local mirror.
  • the local mirroring unit's buffer 210 retains the order of write requests and transmits them to the remote mirroring unit in the same order they were received by the local mirroring unit 204, to preserving data consistency at all times.
  • the remote mirroring unit receives TCP Protocol Data Units (also referred to herein as TCP packets), for instance, from the local mirroring unit 204 and writes them to a disk subsystem (such as an external drive 614 or a RAID unit 312) such that the drive is at least logically block-for-block the same as the local mirror 230, if any, and the host 200 volume at a previous time.
  • TCP Protocol Data Units also referred to herein as TCP packets
  • a disk subsystem such as an external drive 614 or a RAID unit 312
  • the mirrored data may be out of date, but it must be consistent.
  • the remote mirroring unit software preferably has an interface to user-space so that user-space programs can disable or re-enable reading, writing, and/or seeking of the remote mirror by the mirroring unit software, allowing the remote disk subsystem — and hence the mirrored data — to be accessed by a second SCSI host on the same chain.
  • the remote mirroring unit and a backup host server will be attached to the shared disk subsystem.
  • the remote mirroring unit may use SCSI ID 6 while the remote server used for restoration uses SCSI ID 7.
  • the remote mirroring unit is mirroring, the remote host will leave the shared drive unmounted.
  • the remote mirroring unit will stop accessing the shared drive and the backup host server can mount it.
  • the remote mirroring unit preferably reports to user-space programs the number of blocks received from the local mirroring unit 204.
  • the remote mirroring unit mirrors to the disk subsystem such that the volume can be mounted by a host system with the same operating system as the local server 200 that created the local volume. If the remote mirroring unit receives a request from the local mirroring unit 204 to write to logical block number N, then the data should be written to logical block number N on the remote mirroring unit's disk subsystem 312 or 614. Write requests from local mirroring unit 204 should be written to the remote mirroring unit's disk subsystem 312 or 614 in the order in which they were received by the local mirroring unit 204, to preserve data consistency.
  • the local mirroring unit 204 acts as a TCP server; the local mirroring unit 204 acts as the remote unit's client.
  • a loss of network bandwidth or connectivity preferably does not interrupt either the local mirroring unit 204 or the remote mirroring unit.
  • data recovery at the remote location preferably does not interrupt the local mirroring unit 204. If the connection between the local mirroring unit 204 and the remote mirroring unit times out or is otherwise broken, the local mirroring unit 204 preferably attempts to reconnect until a connection is re-established. Then the local mirroring unit 204 preferably continues sending mirror data where it left off and otherwise resumes normal operation.
  • the inventive mirroring units are more "intelligent" than the original Off-SiteServer product in that the inventive mirroring units run a modified operating system which is based on the FreeBSD UNIX operating system.
  • the boot process was also modified to show a mirroring unit configuration utility on the console in place of a login prompt, and the kernel was recompiled. At the source each mirroring unit 204 is running an operating system that allows it to run fully independently of the host server 200.
  • the mirroring unit 204 does not require initialization or connection software on the host server 200 (on the original Off-SiteServer product this software took the form of a Vinca NLM). Instead, the mirroring unit 204 operating system emulates a SCSI or other standard disk or data acquisition point. So the mirroring unit 204 can be mounted, for instance, as a mirrored SCSI disk under any operating system that supports SCSI, including at least the Microsoft Windows 95, Microsoft Windows 98, Microsoft Windows NT, Novell NetWare, FreeBSD, and Linux operating systems.
  • the disk emulation is preferably carried through to the point that any standard disk operation can be performed (at least from the server 200 perspective), including handling server 200 requests for disk formatting, disk partitioning, disk integrity checks such as scandisk, and so on, in addition to disk reads and disk writes.
  • a system according to the invention can also maintain a full mirrored volume 230 locally for fault tolerance. Because this mirroring operation occurs by forking the data (or doing two writes) below the emulation layer of the software in the mirroring unit 204, the mirroring unit 204 is able to maintain this local volume 230 along with a sequential data change buffer.
  • the mirroring unit 204 allows the mirroring unit 204 to service local reads by the server 200 without excessive latency, which in turn allows the system to run without a disk handicap and no split-seeks software, eliminating a potential software compatibility problem.
  • This also allows the inventive system to mirror data back to a local disk of the server 200 under local disk mirroring instead of going over the journey link 206.
  • the local mirroring unit 204 need not include a spoof generator to pre-acknowledge writes back to the host 200, because the local mirror 230 is not subject to the delays and risks associated with sending mirrored data over the journey link 206.
  • a mirroring unit normally includes operating system software. Accordingly, at least some mirroring units can run multiple "host” applications to manipulate the mirrored data they have acquired.
  • the system can also be scaled up or down to meet requirements in a particular environment, using drivers and/or other appropriate software and/or hardware. For example, processes could be spread across multiple processors, SCSI cards, and/or other "intelligent" devices to handle more activity and workload.
  • a system can be scaled down to reduce costs while still meeting the needs of lower performance environments.
  • the local mirroring unit 204 can run as an independent intelligent disk subsystem, or it can run an emulation of the host 200 operating system as a fail-over for local fault tolerance.
  • the local disk volume 230 can serve as a local mirrored replacement for local fault tolerance if the host 200 disk subsystem crashes.
  • the system maintains consistency and availability at the remote location in part by an intelligent buffer 210 that maintains and sends data on a first-in-first-out basis. In this way data blocks are transmitted to the remote location in the exact order they are received through the emulation layer at the local mirroring unit 204. Sequence numbers and/or timestamps may also be used, since packetized data does not necessarily arrive at the destination in the same order it was sent.
  • the local mirroring unit has two disk systems attached to it through a local SCSI disk controller.
  • One disk contains the host operating system (e.g., FreeBSD 3.1) on it, with associated utilities and mirroring unit administrative software.
  • This disk also serves as a buffer 210 disk.
  • the other disk system attached to the mirroring unit is at least as large as the host 200 disk being mirrored and serves as the local mirror 230 of the host 200 disk.
  • SCSI data is read off of the QLogic card and evaluated in the kernel as read or write requests.
  • Read requests that come from the QLogic card are preferably fulfilled using the local mirror disk 230 and not be sent across the network 206.
  • Write commands are copied directly to the local mirror disk 230 and acknowledged to the host system 200 as soon as possible (but not necessarily pre-acknowledged), as well as added to a circular queue on the buffer disk or in nonvolatile RAM. Every time a block is written to the circular queue two blocks are actually be written sequentially, one being the actual data block to be transmitted, and the other being a timestamp for the current tail pointer for the queue, possibly with other data such as LBN (logical block number). This second block is a so-called meta-data block.
  • Queue pointers may also be maintained by keeping a copy of at least them, and possibly the entire circular queue, in nonvolatile RAM if such RAM is available.
  • a way to save both space and time is to write to the circular buffer in larger chunks at a time, buffering blocks in memory until enough accumulate to perform a write. This allows the meta-data block to be used for many data blocks, lessening the number of disk write operations and saving on disk space.
  • the head of the queue is found by searching for the block with the most recent timestamp in its meta-data segment, and then using that meta-data segment to locate the tail pointer. This can be done, for instance, by performing a binary search. Since the buffer implementation is circular it is not necessary to remove transmitted blocks physically from the buffer (i.e.. by deleting or zeroing them); incrementing the tail pointer effectively does this. Buffer full conditions are detected when the head pointer is one less than the tail pointer. Pointers refer to positions in the circular buffer and not to the data in the buffer itself (i.e., it's an array not a linked list).
  • the queue buffer size changes, depending upon the end-user's data change rate and the length of time the customer needs to withstand a network 206 outage.
  • the queue buffer could be as small as a few hundred megabytes, or as large as the host volume being mirrored. There are no inherent restrictions on the minimum or maximum size of the buffer, and in cases where high data change rates and frequent lengthy interruptions of the journey link 206 are anticipated, the buffer may need to be larger than the host volume being mirrored.
  • the system may check writes against the local mirror and only add them to the circular buffer if they are indeed different, while avoiding any lazy write problems. This might be accomplished by maintaining a hash table of checksums for each LBN on the disk; one tradeoff would be processor time computing checksums and memory vs. additional disk operations.
  • Figure 13 illustrates methods of the present invention for remote data mirroring.
  • Some methods include steps for installing mirroring units; for convenience, these steps are collectively identified as steps within an installing step 1300.
  • system integrators, mirroring equipment vendors, and administrators may be licensed to perform some or all of the steps shown within step 1300 when installing systems such as those illustrated in any of Figures 2 through 12.
  • Other methods of the invention include steps for transmitting data to one or more mirroring units; for convenience, these steps are collectively identified as steps within a transmitting step 1302. These transmitting steps may be performed under license with test data by installers as part of the installing steps 1300, but they may also be routinely performed with mission-critical data at the behest of regular users of a system according to the invention.
  • At least one server 200 is connected to at least one local mirroring unit 204.
  • this connection may be in the form of a SCSI bus, a fibre channel connection, a USB connection, or some other standard disk subsystem bus.
  • the one local mirroring unit 204 emulates a disk subsystem, connecting it during step 1304 is basically the same as connecting a conventional disk subsystem to the server 200, at least from the point of view of the server 200. In particular, no special mirroring NLM or other mirroring software installation is required.
  • At least one local mirroring unit 204 is connected to at least one corresponding journey link 206.
  • this may involve various operations. For example, if the journey link 206 includes a local area network then the local mirroring unit 204 may be connected to that network like other network nodes; SNMP support may also be configured. If the journey link 206 includes a dial-up connection from the local mirroring unit 204, then the dial-up parameters are configured. Likewise, if the journey link 206 includes a dedicated private telecommunications line such as a Tl line, then familiar operations are performed to make the connection.
  • a connecting step 1308 at least one remote mirroring unit 208, 308, 408, 508, 608 or 708 is connected to at least one corresponding journey link 206. This may be accomplished in generally the same manner as the connection of the local mirroring unit 204 during step 1306. However, when the remote mirroring unit acts as a TCP server in a given embodiment, the local mirroring unit 204 acts as the remote mirroring unit ' s client. Thus, in such embodiments the connecting step 1306 connects a TCP client while the connecting step 1308 connects a TCP server.
  • a testing step 1310 tests are performed on the mirroring unit(s). These tests may include, for instance, comparing throughput performance of the local mirroring unit 204 with the performance of a RAID unit; re-mirroring data from the remote site back to the local site; putting incorrect configuration information into the local mirroring unit 204 and then correcting that information; rebooting the local mirroring unit 204; disconnecting the journey link 206; interrupting power to the local mirroring unit 204; interrupting power to the remote mirroring unit; overflowing the buffer 210 of the local mirroring unit 204; and other tests.
  • the testing step 1310 may involve performing one or more of the tests described in the "test suite" section of this document. Testing 1310 may also involve transmitting data as discussed below in connection with step 1302, but testing is shown as a separate step in Figure 13 for clarity of illustration.
  • the transmitting step 1302 may include a transmitting step 1312 which sends data from the server 200 over a standard bus to the local mirroring unit 204. This is possible because the present invention, unlike the conventional path 104, provides a mirroring unit which emulates a disk or RAID subsystem.
  • the data being mirrored is transmitted over the journey link 206.
  • this may be done with a dedicated link as was the case with the conventional path 104, but it may also be done using standard protocols such as Ethernet and/or TCP and/or other open standard protocols, including their associated conventional networking infrastructure such as local area networks and/or the Internet.
  • the mirrored data is time-stamped by the local mirroring unit 204 to maintain a record of the sequence in which blocks of data were mirrored and to also tie the data to a particular point in time.
  • This is coupled with remote and/or local data storage large enough to hold one or more snapshots of the mirrored volume plus incremental changes at the sector/track/block level to that volume, instead of simply holding a current copy of the mirrored volume.
  • only one snapshot is needed. The single snapshot provides a baseline, and subsequent changes are journaled so that the state of the volume at any desired point (subject to the journaling granularity) can be recovered.
  • the journal may be arbitrarily large with additional storage space added as needed to hold it, or it may be kept in a FIFO circular buffer of some fixed size, with older journal entries overwritten by new ones after the journal buffer is initially filled. More generally, suitable re-mirroring software plus the snapshot(s) and (if necessary) the incremental changes can be used at a later time to reconstruct the mirrored disk volume as it existed at a specified previous time.
  • the data being mirrored is transmitted to a serverless remote mirroring unit.
  • the remote mirroring unit is not a conventional server, although it has hardware and functional capabilities in common with such servers. Servers provide more general functionality than mirroring units; mirroring units are focused on effectively providing substantially continuous, nearly real-time remote data mirroring.
  • the remote mirroring unit behaves like a remote mirroring server with regard to acquisition of data over the journey link 206 but otherwise strongly resembles a mounted disk. In particular, the remote mirroring unit behaves like a disk or RAID unit with regard to a secondary server if one is attached.
  • the remote mirroring unit can do various things. For instance, the remote mirroring unit may simply convert the received data packet into data blocks that are written out to a single external disk 614. The remote mirroring unit may convert these data packets into disk blocks and write them to an internal disk subsystem and/or disk partition.
  • the remote mirroring unit may receive the packet data, convert it to disk data blocks, and write them to a RAID unit 312 in the form of an external data subsystem utilizing internal striping (RAID) software to stripe data across multiple disks on an "unintelligent" disk subsystem.
  • RAID internal striping
  • This same conversion from packets to disk block data to striped (RAID) data could also occur through a hardware controller and related driver, with storage to an external "unintelligent" disk subsystem.
  • the remote mirroring unit may also write to an external intelligent RAID subsystem 312, with disk blocks being written to the disk subsystem in a data stream and striped by the intelligent RAID subsystem.
  • the remote mirroring unit may write the data first to a remote buffer and then send an ACK with some type of "signature" of the data (such as a checksum or Cyclic Redundancy Check value) back to the local mirroring unit.
  • the local mirroring unit would then either ACK- ACK or NAK-ACK (based upon verification of the signature) the data; only upon receiving an ACK-ACK from the local mirroring unit would the remote mirroring unit commit the data from the remote buffer to the remote mirror.
  • the remote mirroring unit receives not only the data, but also an original signature from the local mirroring unit, it will NAK the original data transmission if the original signature does not verify correctly.
  • ACKs trickle up from the remote mirror disk itself (probably from its cache); on the local mirroring unit, ACKs also trickle up from the local mirror disk itself (probably from its cache); but on the local mirroring unit, ACKs would not be needed from the remote mirroring unit, only from the local end of the journey link, before ACKing the host. It would still be prudent on the local mirroring unit to wait for an ACK from the remote mirroring unit before deleting blocks from the local buffer, but this can be done long after ACKing the host.
  • the remote mirroring unit may relay data directly to a remote server 300 through the server ' s network operating system.
  • This operating system can be in an active or passive state. In either case data received through the connection 302 can be written to an internal local disk subsystem through the server 300 operating system.
  • This approach requires specific software for each operating system at the remote location.
  • the remote mirroring unit may also use an Internet-based data window to send and receive data between the remote mirroring unit and a secondary server 300. This data window could be through a plug-in extension to browser interfaces or though Internet component extensions to the core operating system, such as Microsoft ActiveX extensions.
  • the local mirroring unit may be "intelligent" enough to relay mirrored data to one remote mirroring unit or to many remote mirroring units; a one- to-many system like that shown in Figure 12 has three remote mirroring units connected by respective journey links 206 to a single multi-ported local mirroring unit 204 and multi-port mirroring units may likewise be used, alone or in combination with single-port mirroring units, in other systems according to the invention. There is no hard limitation on the number of remote mirroring units in a given system.
  • the remote mirroring unit can also relay mirrored data to a nearby mirroring unit and/or another more distant remote mirroring unit for further fault tolerance.
  • a remote mirroring unit can act as a head end to load balance between two or more following remote mirroring units to distribute loads and provide fault tolerance, with appropriate attention to continuous consistency and completeness of the data mirrors.
  • N remote mirroring units can be connected to each other and maintain the same network address or Domain Naming System ("DNS”) name to provide further fault tolerance. Combinations of these various approaches can also be used.
  • DNS Domain Naming System
  • the remote mirroring unit behaves as a SCSI master (for instance) and writes data out to the remote disk(s).
  • a secondary server 300 is present, this server 300 follows both the remote mirroring unit and the remote disk subsystems(s) in the SCSI chain.
  • the secondary server 300 is typically a slave and/or in a passive state.
  • the remote server 300 mounts the external volume(s) and becomes a SCSI master.
  • the remote mirroring unit dismounts its remote disk subsystem driver and goes into a passive (slave) state.
  • the remote mirroring unit 208, 308, 408, 508, 608, or 708 controls the RAID unit 312 or other remote disk subsystem until such time as it is commanded to stop so that a switchover can be performed.
  • the remote mirroring unit performs remote data mirroring and as SCSI master it sends data to the RAID unit 312 as discussed elsewhere herein.
  • the Novell or other secondary server 300 is in a passive (dismounted) state. This prevents damage that would otherwise occur by wiring together the server 300, remote mirroring unit, and RAID unit 312 or other remote disk subsystem in a two-to-one manner as shown in Figure 14.
  • the remote mirroring unit dismounts the RAID unit 312 driver and the server 300 mounts the RAID unit 312.
  • the server 300 then becomes the SCSI master. Since one cannot necessarily predict or enforce the secondary server SCSI card selection, the remote mirroring unit preferably has the secondary host position (LUN 6). As the two machines come up, the remote mirroring unit may experience a second reset as its driver powers up. This is normal, but the remote mirroring unit should be able to recover at the device driver level. Note that by utilizing the dual hosting (not dual channel) method, the cabling becomes a normally terminated SCSI chain; no additional hardware is required.
  • the switchover can be accomplished entirely by software, through storage subsystem and/or driver dismount, mount, and related operations.
  • the remote mirroring unit can be made to run a full network operating system. In the event of a disaster such a remote mirroring unit could go into an active state and become a fully functional server for the information on the disk subsystems to which it sent mirrored data.
  • the remote mirroring unit could also run an emulation program that would allow it to emulate a server under a specified host operating system at the local site.
  • the remote mirroring unit could also run a program to shut down the operating system it employed under mirroring, and any related programs, and then restart under a specified host operating system from a separate internal disk or a separate partition.
  • the remote mirroring unit could also be enhanced to run continuously as a secondary server rather than being normally dedicated to data mirroring only.
  • the remote mirroring unit can perform as a local mirroring unit 204. For instance, when mirroring from site A to site B to site C, a mirroring unit at site B is a remote mirroring unit with respect to site A and a local mirroring unit with respect to site C.
  • the remote mirroring unit can also perform as a local mirroring unit 204 in a recovery from the remote location back to the source. That is, when mirroring from site A to site B, the mirroring unit at site A is local and the mirroring unit at site B is remote, but in recovering data from site B back to site A.
  • the mirroring unit at site A is remote and the mirroring unit at site B is local.
  • some inventive systems can accommodate multiple user sessions; a user session being a mirrored data relay or storage session. Multiple combinations and instances of the above scenarios can thus occur concurrently or separately in the appropriate environment. It may be necessary to include more processors, disks, memory, and so on to accomplish particular combinations.
  • These various tools and techniques can also be used in a one-to-many mirroring system or a many-to-one mirroring system according to the invention.
  • discussion of tools and techniques which refer to packets refer to an IP, Ethernet, token ring, or other packetized data environment, and it will be understood that other supported environments may write in data streams instead of using packets.
  • step 1312 may be performed by performing transmission step 1314 (or by using a private dedicated link 206) if the transmission is to a serverless remote mirroring unit. Steps may also be omitted unless called for in issued claims, regard- less of whether they are expressly described as optional in this Detailed Description. Steps may also be repeated, combined, or named differently.
  • Articles of manufacture within the scope of the present invention include a computer-readable storage medium in combination with the specific physical configuration of a substrate of the computer-readable storage medium.
  • the substrate configuration represents data and instructions which cause the computers to operate in a specific and predefined manner as described herein.
  • Suitable storage devices include floppy disks, hard disks, tape, CD-ROMs, RAM, flash memory, and other media readable by one or more of the computers.
  • Each such medium tangibly embodies a program, functions, and/or instructions that are executable by the machines to perform flexible mirroring method steps substantially as described herein, including without limitation methods which perform some or all of the steps illustrated in Figure 13 and methods for installing and/or using the systems illustrated in Figures 2 through 12.
  • the invention also provides novel signals which are used in or by such programs.
  • the signals may be embodied in "wires", RAM, disk, or other storage media or data carriers. Additional Information
  • the present invention does not have any software that loads on the host computer (e.g., local server 200), thereby reducing or avoiding the aforementioned problems.
  • the host operating system freezes the mirroring units continue to operate and mirrored data is available because the mirroring units are running their own operating system(s).
  • the invention scales readily. If a faster processor comes out one simply uses this processor in the mirroring units as desired. If the disk size is bigger, one puts bigger disks in mirroring units. If the data change rate exceeds current ability to write to disk, one uses a caching controller and adds memory to the system.
  • Some other solutions require cooperation from the operating system manufacturer in order to integrate and operate properly without bugs. Because all operating systems will support SCSI and fibre channel (for instance) for the foreseeable future, such cooperation is not required for installation and use of the invention. When other solutions fail they can take the host 200 with them, because of the close interactions outlined above. Because the invention can operate independently of the host 200, if it fails it need not seriously affect the host computer. Conventional disk mirroring was originally designed for local fault tolerance. Two disks would be written to in parallel, and if one disk failed the computer would continue to operate. The disk that failed would be dismounted from the operating system in the background. The operating system and computer would often continue to run without missing a beat.
  • the inventive mirroring unit can look like a SCSI disk and be mounted as a mirrored disk, it provides a similar advantage. If a mirroring unit dies, it simply gets dismounted. For instance, if the operating system or other software on the mirroring unit fails then the mirroring unit stops emulating a disk. As a result, the operating system on the host 200 no longer recognizes the mirroring unit. In response, the operating system on the host 200 simply dismounts the mirroring unit 204 and continues to run.
  • At least some previous mirroring system implementations used a single disk IDE buffer. Even with spoofing, such a smart buffer has not been able to keep up with high speed SCSI RAID units with hardware striping. The most critical data that was being transmitted to the remote location was trusted to a single disk with no fault tolerance at the smart buffer level.
  • the local and remote mirroring units can both mirror a single disk buffer for fault tolerance, and they can perform hardware RAID striping across multiple disks. This provides an ability to keep up with new high speed storage subsystems on servers, and better fault tolerance. In the event of an individual disk failure in the server 200 volume or a mirroring unit disk 210, 310 this also reduces the risk of losing buffered data.
  • SAN Storage Access Network
  • NAS Network Attached Storage
  • Requiring a standard remote server such as the server 300 made it hard or impossible to provide backup and mirroring for the SAN and NAS disk subsystems that are becoming more prevalent.
  • all of these subsystems can perform a local mirror through Ethernet, fibre channel, and/or SCSI.
  • the inventive mirroring units can accept multiple input types, including SCSI, Ethernet, and fibre channel inputs.
  • the invention also provides support for larger storage subsystems. Many earlier fault tolerance solutions were designed for an environment in which a six Gigabyte storage volume was considered very large. With storage costs falling, disk subsystems are increasing in size at a very rapid rate. It is now common for servers to have volumes of 100 Gigabytes. The invention accommodates these larger volumes in part by handling synchronization for the host server 200 in the background, namely, on the mirroring unit. Offloading this task from the host server to the mirroring unit(s) allows a true mirror of the main host server 200 without a large performance decrease. By contrast, alternative "clustering" and/or mirroring solutions that require a local server to handle the synchronization required for a mirror may either severely slow or crash that primary server.
  • re-mirroring At least some previous implementations of re-mirroring have required the local server 200 to intervene if the local buffer could not support the entire local volume, although implementations have done much to avoid re-synchronization of mirrored disks (re- mirroring), over the telecommunications link.
  • Re-mirroring slowed the main/primary/host server 200 to a standstill, and could take several days. So the re-mirroring has generally been performed only on weekends when the network could run slower, as there would be fewer users. But as disk subsystems are getting larger this is no longer acceptable.
  • the invention supports nonvolatile storage, not only at the remote location but also in the local mirroring unit 204, which is large enough to hold the complete volume that is being mirrored to the remote location.
  • the local mirroring unit 204 pre- acknowledge the complete local disk storage volume into a localized smart buffer and perform the tasks related to a re-mirror in the "background" from the server 200 perspective.
  • the limitation of the maximum rate of a Tl output, from either the local or remote location slowed a re-mirror even if a frame relay network, ATM, and/or VSAT network was available.
  • the invention flexibly allows a larger I/O pipe capability, which can improve performance because re-mirrors will be quicker and data deployment will be more efficient. If mirrored data being stored remotely becomes unavailable, the data stored at the unavailable site can be moved at high speed to another facility using a high-speed private data network.
  • the present invention permits an open API so that adjustments can be made from user-space processes to address specific customers and/or emerging markets.
  • the present invention preferably has an API which provides one or more calls to reconfigure a mirroring unit without interrupting the server 200, and also provides a call to reboot the mirroring unit without interrupting the server 200.
  • Configuration Data System configuration data is preferably distributed, so that if one of the mirroring units loses configuration data, that configuration data can be recovered from one of the unit's peers.
  • Basic configuration data such as network information is preferably stored in nonvolatile storage (e.g., on disk, or in battery-backed semiconductor memory), so that even if the configuration data on the disk is lost, the configuration data can still be restored from the peer mirroring unit.
  • a world wide web interface preferably provides, at a minimum, the following configuration options or their equivalents: IP address (remote/local); gateway (remote/local); net mask (remote/local); administrator password (shared); buffer size (local); buffer high water mark (buffer filled beyond acceptable limits); volume size (configurable up to a factory-configured hard max); SCSI target Logical Unit Number ("LUN”); and SNMP configuration (remote/local).
  • the SNMP configuration itself preferably contains the following: add/delete SNMP monitoring hosts (remote/local); event polling intervals; buffer filled past acceptable limits; network connection failure; buffer full; remote out of synch; add/delete e-mail recipient.
  • the web interface preferably provides, at a minimum, the following status information: blocks in buffer; blocks sent; blocks received; mirroring unit version; mirroring unit serial number; volume size; whether this unit is remote or local.
  • the web interface preferably provides an unmount remote utility.
  • the web interface preferably also provides a log dump report. SNMP and SMTP traps are generally used for the following events: buffer filled past acceptable limits; buffer full; network connection failure; remote out of synch.
  • the administrative tools may provide notifications by e-mail, by paging, or other means. Notification may be real-time and/or in combination with automated logs or automatically generated reports. Notifications may be sent to system administrators and/or vendors.
  • a mirroring unit can notify users (and mirroring unit vendors) of problems or significant events on the mirroring unit via e-mail as well as through SNMP.
  • One can write custom scripts for this e-mail so that different users or groups of users are notified. Report outputs are not necessarily static.
  • a system according to the invention includes basic hardware such as a standard Pentium II, Pentium III, AMD K6-3 or AMD K7 class PC-compatible computer (marks of their respective owners).
  • the machines preferably have at least 64, 128, or 256 megabytes of RAM, and a rack-mounted case. They also preferably contain one 100Mb Ethernet card, FDDI card, or the like.
  • the machines preferably have a QLogic SCSI card for disk emulation and an Adaptec 2940UW adapter for buffer and mirror control, or a FreeBSD supported DPT brand RAID card.
  • Caching may be used, including RAID or SCSI controller caching, caching in volatile RAM in the mirroring unit(s), caching in nonvolatile RAM (e.g., static RAM or battery-back RAM) in the mirroring unit(s), and otherwise.
  • Caching tools and techniques familiar to those in the art may be readily adapted for use according to the present invention.
  • local mirroring units 204 which include a local mirror 230 have storage capacity of at least N for that local mirror.
  • a disk system which serves as the local buffer 210 (with or without a local mirror) has a capacity of at least six-fifths N, that is 1.2 times N.
  • the remote mirroring unit has at least one disk system, for the remote mirror, of size at least N.
  • the local mirroring unit buffer 210 may need to be equivalent in data capacity to its remote mirroring unit, including buffers and hot-swappable RAID subsystems, to permit a local re-mirror. Test Suite
  • Tests used to gauge performance of a system according to the invention preferably include analytical tests which can be used to gauge relative performance and Boolean (pass/fail) tests that cover critical functional specification conformance criteria.
  • a Boolean test is passed if the specified answer to all questions are correctly matched by test results.
  • the Boolean tests can be used to determine the suitability of deliverables.
  • Tests should preferably be passed both in a local network configuration (where the journey link 206 is within a single local area network) and in a local and remote configuration (where the local mirroring unit 204 and the remote mirroring unit are geographically distant from each other).
  • a remote network configuration could consist of two sites connected together with a Tl link 206 or an equivalent amount of public Internet bandwidth as the journey link 206.
  • Analytical tests preferably use a standard disk hardware test suite such as Bonie (for UNIX) or PCTools (for Windows NT and Novell clients). The tests compare performance of a native disk drive (for which the model, size, and characteristics are noted) with the performance of a flexible mirroring unit 204. The performance outputs are noted for later reference.
  • Bonie for UNIX
  • PCTools for Windows NT and Novell clients
  • a local mirroring unit 204 configured with a 100 megabyte host volume and a remote network configuration successfully mirror data to a remote mirroring unit with a data rate of at least 300 megabytes/hour and preferably higher if FDDI or other support is present? (Yes) Note that the 300 megabytes/hour rate is under the maximum carrying capacity for a Tl connection by about 50%; Tl capacity is about 617 megabytes/hour.
  • the host system 200 After mounting the remote mirror on the local host system 200 physically, is the host system 200 able to mount the mirror and are application programs on the host 200 and its clients able to use data on the mirror successfully? (Yes) In response to an input of improper information such as a wrong remote IP address, or an invalid SCSI ID (less than 0 or greater than 15), does the mirroring system crash or hang? (No) Can the user correct the information, re-initialize the software and have it perform normally without needing a mirroring unit reboot? (Yes) Does all software display a correct version number and copyright statement?
  • a computer system for remote mirroring of data includes one or more flexible mirroring characteristics.
  • Systems for local mirroring e.g., where the source and destination are less than ten miles apart
  • the system may be characterized by having a serverless destination.
  • one embodiment of the system mirrors data from the local server 200 as a source through the local mirroring unit 204 to the remote mirroring unit 208, 408, 508, 608, or 708 as a destination, without requiring the use of a remote server attached to the remote mirroring unit.
  • the system may also be characterized as non-invasive, in that no software designed specifically for remote data mirroring need be installed on the local server 200. Similarly, no such software need be installed on the secondary server 300 in systems that include a server 300. Instead, each mirroring unit runs an operating system and one or more remote data mirroring application programs (including threads, processes, tasks, etc.).
  • the mirroring units rather than the server(s) buffer data to be mirrored, create and monitor connections over the journey link 206, and transmit/receive mirrored data over the journey link 206, thereby relieving the server(s) of those tasks.
  • the system may be characterized by disk emulation, such that the system mirrors data from the local server 200 to the local mirroring unit 204 through a standard storage subsystem bus. Suitable standard storage subsystem buses include SCSI, fibre channel, USB, and other nonproprietary buses. Such buses are also referred to herein as "connections" to the local mirroring unit 204.
  • the system could be characterized by a TCP journey line characteristic and/or by an Ethernet journey line characteristic.
  • the system mirrors data from the local server 200 through the local mirroring unit 204, which operates as a TCP client over the journey line 206; the remote mirroring unit 208, 308, 408, 508, 608, or 708 operates as a TCP server.
  • a journey line characteristic indicates that the high-bandwidth low-latency requirements imposed by SCSI, original Off-SiteServer serial connections. SAN connections, and the like are not present in a connection 206 between a local mirroring unit 204 and a remote mirroring unit.
  • the system might also be characterized by a multiplicity characteristic. That is, the system may provide many-to-one mirroring from two or more local (primary) servers 200 to a single remote mirroring unit 208, 308, 408, 508, 608, or 708.
  • the data mirroring system of remote mirroring unit nonvolatile storage may then include one disk partition for each primary network server 200 with each disk partition holding mirrored data for the respective server 200, one external hard disk 614 for each server 200. one RAID unit 312 for each server 200, or some combination thereof.
  • the various primary (local) servers 200 may all use the same operating systems or they may use some combination of different operating systems. In some cases the destination nonvolatile storage is sufficiently large to hold the combined current nonvolatile data of all of the primary servers 200.
  • the system may provide one-to-many mirroring from a given local (primary) server 200 to two or more remote mirroring units 208, 308, 408, 508, 608, or 708.
  • the invention also provides methods, including methods for installing flexible mirroring units, methods for using such units, and methods for doing both.
  • a method for facilitating flexible data mirroring includes at least two steps from the group 1300 of installing steps.
  • Another method for flexible data mirroring includes one or more transmitting steps 1302.
  • One of the installing steps involves connecting 1304 the local server 200 to the local mirroring unit 204 with the standard storage subsystem bus 202, thereby permitting the local mirroring unit 204 to emulate a disk subsystem in communications over the link 202.
  • a step 1306 involves connecting the local mirroring unit 204 to the journey link 206 for transmission of data by at least one of an Ethernet connection and a TCP connection.
  • a step 1308 involves connecting the remote mirroring unit 208, 308, 408, 508, 608. or 708 to the journey link 206 for reception of data transmitted by at least one of an Ethernet connection and a TCP connection.
  • a testing step 1310 tests at least one mirroring unit 204, 208, 308, 408, 508, 608, or 708 after at least partial completion of at least one of the aforesaid connecting steps.
  • One of the transmitting steps 1302 is a step 1312 which transmits data from the local server 200 to the local mirroring unit 204 over the standard storage subsystem bus 202 while the local mirroring unit 204 emulates a disk subsystem.
  • a step 1314 transmits data from the local mirroring unit 204 over the journey link 206 to the remote mirroring unit 208, 308, 408. 508, 608, or 708.
  • a step 1316 (which may be performed with the same data transmission as step 1314) transmits data from the local mirroring unit 204 over the journey link 206 to the remote mirroring unit 208, 308, 408, 508. 608, or 708 when the remote mirroring unit is serverless, that is, when it is not attached to a secondary server 300.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Quality & Reliability (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

Methods, systems, and configured storage medium are provided for flexible data mirroring. In particular, the invention provides many-to-one data mirroring, including mirroring from local servers (200) running the same or different operating systems and/or file systems at two or more geographically dispersed locations. The invention also provides one-to-many data mirroring, mirroring with or without a dedicated private telecommunications link, and mirroring with or without a dedicated server or another server at the destination(s) to assist the remote mirroring unit(s) (208). In addition, the invention provides flexibility by permitting the use of various combinations of one or more external storage units and/or RAID units to hold mirrored data. Spoofing, SCSI and other bus emulations, and further tools and techniques are used in various embodiments of the invention.

Description

FLEXIBLE REMOTE DATA MIRRORING
FIELD OF THE INVENTION
The present invention relates to the remote mirroring of digital data from a server or other computer in order to provide better fault tolerance and/or disaster recovery, and relates more particularly to tools and techniques for increasing the flexibility of remote data mirroring by permitting its use in a wider variety of network configurations than those previously used
TECHNICAL BACKGROUND OF THE INVENTION
United States Patent No 5,537,533 describes tools and techniques for remote mirroring of digital data from a primary network server to a remote network server A system according to that patent includes a primary data transfer unit with a primary server interface and a primary link interface, and a remote data transfer unit with a remote link mterface and a remote server interface The primary link interface includes a spoof packet generator capable of generating a pre-acknowledgement for the primary network server That is, the system has a "smart buffer" which gives the primary server a pre- acknowledgement or "spoof after mirrored data has been stored on a nonvolatile buffer in the primary link interface and before an acknowledgement arrives indicating that the mirrored data has been stored by the remote server
MiraLink Corporation of Salt Lake City, Utah is the owner of U S Patent No 5,537,533 MiraLink has made commercially available for more than one year before the date of the present application an Off-SiteServer product (OFF-SITESERVER is a mark of MiraLink) The Off-SiteServer product includes technology to remotely mirror the disks of a Novell NetWare server to another server at a geographically remote location through a low bandwidth telecommunications link (NETWARE is a mark of Novell, Inc )
Remote mirroring of data from a primary network server to a remote replacement network server using data mirroring is a powerful and efficient method to back up data Remote mirroring creates a copy of data at a safe distance from the original data and does so substantially concurrently with the storage of the original data The remotely stored data can be available almost immediately after a disaster if it was copied to a "warm" remote network server, that is, a remote server which can be up and running as the new primary server within minutes of the actual or simulated disaster In a typical installation, use of the Off-SiteServer product involves a pair of Off- SiteServer boxes; one is a local box and the other is a remote box. The Off-SiteServer boxes are configured with specialized hardware and with firmware and/or other software, generally as described in U.S. Patent No. 5,537,533. A proprietary serial line connects the local NetWare server to one of these boxes. The NetWare server itself uses a Vinca card (VINCA is a mark of Vinca Corporation). This card is driven by a NetWare Loadable Module ("NLM") that intercepts disk-driver requests, and sends data down the serial line to the local Off-SiteServer box.
The local Off-SiteServer box has a 4 Gigabyte nonvolatile buffer, such as an IDE disk drive. Data is pre-acknowledged into this Off-SiteServer buffer. As far as the operating system of the local server is concerned a second '"mirrored" write has occurred locally. In reality, the Off-SiteServer product has received this data from the NLM and stored it on the local buffer. The local Off-SiteServer box stores sector and track (or block level) data changes until it can safely send them to the remote Off-SiteServer box at the remote location. The buffer in the local Off-SiteServer box is also "smart" in that it stores any data above what the telecommunications link can handle locally. This data is stored in the local Off-SiteServer box until the remote Off-SiteServer box has successfully written to the remote secondary server and sent back an acknowledgement to the local (primary) Off- SiteServer box. When this acknowledgement is received the local Off-SiteServer box frees the space in the local nonvolatile buffer that is occupied by the successfully transmitted piece of sector/track/block data.
The Off- Site Server product uses a V.35 interface for data output at the local (primary) site. V.35 is a serial telecommunications standard that connects to a Channel Service Unit/Data Service Unit ("CSU/DSU"), which in turn interfaces with the telecommunications link. The remote (secondary) location has a second CSU/DSU that relays the sector/track/block information to the V.35 input interface of the remote secondary Off- Site Server box. The secondary Off- Site Server box outputs this sector/track/block data through the proprietary serial connection using a serial cable connected to another Vinca card in the secondary (remote) server. The remote server's data mirroring and system software then writes this sector/track/block information to the remote server's disk drive and the write is acknowledged back to the local Off-SiteServer box. This system is capable of handling about 300 megabytes of change data in an hour. The Off-SiteServer product is intelligent enough to sense if there is a decrease or increase in bandwidth and/or if the telecommunications link has gone down. During link downtime periods, the Off-SiteServer box can store data changes from the server in the local nonvolatile smart buffer. When the link is active again, the Off-SiteServer product starts transmitting automatically. The Off-SiteServer product can change its bandwidth output on the fly as bandwidth becomes more or less available. All of the transmissions described above also incorporate standard software checksum error detection and correction, and/or hardware error correcting code ("ECC") error handling.
In case of a disk or server failure on the local (primary) NetWare server, a secondary (remote) server attached to a remote (secondary) Off-SiteServer box in the manner just described has a complete mirrored disk copy of all the data on the local (primary) server. This remote backup copy can be restored back to the local (primary) server. This secondary remote server can also stand in for the local primary server in the event of disaster. Such a secondary restoration and/or stand-in can be executed relatively quickly with a simple set of command lines.
In short, the Off-SiteServer product and other remote data mirroring technologies provide valuable fault-tolerance and disaster recovery capabilities, both to mission-critical data and in other contexts. Nonetheless, these existing approaches have unnecessarily limited flexibility. For instance, the Off- Site Server product requires a specific version of hardware and software from Vinca Corporation. This required version of the Vinca product does not support any operating system/file system platform other than the Novell NetWare platform. The hardware component of the necessary Vinca package also does not work with newer, faster servers and larger disk volumes. The original Off-SiteServer product was also designed to connect one local server to one remote server. Only a single server can mirror to a remote server at a given time. Multiple servers at different locations cannot readily mirror to a single remote site. Likewise, if an enterprise has multiple local servers running different operating systems and/or file systems, each server running a separate platform must be mirrored to a matching remote server.
As explained in greater detail in discussing the present invention, there are other flexibility limitations as well. For instance, the original Off-SiteServer product requires an NLM on the local server, and it was designed to use private dedicated telecommunications links. Conventional mirroring also requires a remote server in order to keep mirrored information in a bootable format at the remote location.
Thus, it would be an advancement in the art to provide more flexible tools and techniques for remote data mirroring, in order to take advantage of both existing and new technologies.
Such improved tools and techniques are disclosed and claimed herein.
BRIEF SUMMARY OF THE INVENTION
The present invention provides tools and techniques for flexibly mirroring data. For instance, the invention permits the use of various combinations of one or more external storage units and/or RAID units to hold mirrored data. In addition, the invention provides many-to-one data mirroring, including mirroring from local servers running the same or different operating systems and/or file systems at two or more geographically dispersed locations. The invention also provides one-to-many data mirroring, mirroring with or without a dedicated private telecommunications link, and mirroring with or without a server at the destination(s) to assist the remote mirroring unit(s). Spoofing, SCSI and other bus emulations, and other tools and techniques are used in various embodiments of the invention.
Unlike some conventional mirroring approaches, the invention does not require a secondary server at the remote location in order to mirror data. A remote server may be used to test the integrity of mirrored data, or to replace a local server which becomes unavailable, but the remote server is not needed to maintain a complete copy of mirrored information in a bootable format at the remote location.
A flexible local mirroring unit mirrors a local disk volume to a remote location via a journey link, which may be part of a local area network, part of the Internet, a low bandwidth telecommunications link, or a high bandwidth dedicated telecommunications link such as a Tl link. The local mirroring unit utilizes the bandwidth efficiently using an intelligent buffer with spoofing, as described in United States Patent No. 5,537,533, for instance. The local mirroring unit is non-invasive of the host operating system. It is not necessary to install on the mirrored local host an NLM or other software designed specifically for remote data mirroring. In particular, larger host volumes than before can be mirrored without degrading performance of the mirrored host, because the load on the host CPU is not substantially increased by mirroring according to the present invention. Putting the necessary processing in the local mirroring unit instead of in the host server also increases reliability and flexibility by making it possible to reconfigure or even reboot the local mirroring unit without interfering with host server processing. To the local host server whose data is being remotely mirrored, the local mirroring unit appears to be simply some familiar type of disk subsystem. Accordingly, standard mirroring tools and techniques can be used within the local server to direct a copy of the data to the local mirroring unit, for subsequent forwarding (unbeknownst to the local server) to a remote mirroring unit that may be tens or hundreds of miles away. Other features and advantages of the present invention will become more fully apparent through the following description.
BRIEF DESCRIPTION OF THE DRAWINGS
To illustrate the manner in which the advantages and features of the invention are obtained, a more particular description of the invention will be given with reference to the attached drawings. These drawings only illustrate selected aspects of the invention and thus do not limit the invention's scope. In the drawings:
Figure 1 is a diagram illustrating prior art mirroring in a network of computers which could also be adapted for use with the present invention. Figure 2 is a diagram illustrating a computer system according to the invention, without a remote server, but including a remote mirroring unit having a large buffer.
Figure 3 is a diagram illustrating a computer system according to the invention, including a remote server with a hot-swappable RAID unit and a remote mirroring unit having a relatively small buffer. Figure 4 is a diagram illustrating a computer system according to the invention, without a remote server, but including a remote mirroring unit having a relatively small buffer and a hot-swappable RAID unit.
Figure 5 is a diagram illustrating a computer system for many-to-one mirroring according to the invention, without a remote server, but including several local servers running a given platform with respective local mirroring units and a single remote mirroring unit having a relatively small buffer and several hot-swappable RAID units.
Figure 6 is a diagram illustrating another many-to-one computer system according to the invention, without a remote server, but including several local servers running a given platform with respective local mirroring units and a single remote mirroring unit having a relatively small buffer and several individual external storage volumes.
Figure 7 is a diagram illustrating another many-to-one computer system according to the invention, without a remote server, but including several local servers running a given platform with respective local mirroring units and a single remote mirroring unit having a relatively small buffer, an external storage volume having several partitions, and a hot- swappable RAID unit likewise having several partitions.
Figure 8 is a diagram illustrating another many-to-one computer system according to the invention, without a remote server, but including several local servers running different platforms with respective local mirroring units and a single remote mirroring unit having a relatively small buffer and several hot-swappable RAID units.
Figure 9 is a diagram illustrating another many-to-one computer system according to the invention, without a remote server, but including several local servers running different platforms with respective local mirroring units and a single remote mirroring unit having a relatively small buffer and several external storage volumes.
Figure 10 is a diagram illustrating another many- to-one computer system according to the invention, without a remote server, but including several local servers running different platforms with respective local mirroring units and a single remote mirroring unit having a relatively small buffer, an external storage volume having several partitions, and a hot-swappable RAID unit likewise having several partitions.
Figure 11 is a diagram illustrating a one-to-many mirroring computer system according to the invention, in which a local server is connected to several local mirroring units for data mirroring to several remote locations.
Figure 12 is a diagram illustrating an alternative one-to-many mirroring computer system according to the invention, in which a local server is connected to one multi-ported local mirroring unit for data mirroring to several remote locations.
Figure 13 is a flowchart illustrating methods of the present invention.
Figure 14 is a diagram illustrating a dual host configuration between a remote mirroring unit, a remote server, and a RAID unit, which may be used in performing a switchover according to the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
The present invention relates to computer systems, methods, and configured storage media for flexible data mirroring. In particular, the invention provides non-invasive mirroring, mirroring with or without a dedicated private telecommunications link, and mirroring with or without a dedicated server or another server at the destination to assist the remote mirroring unit. The invention also provides many-to-one data mirroring, including mirroring from local servers running the same or different operating systems and/or file systems at two or more geographically dispersed locations. In addition, the invention provides flexibility by permitting the use of various combinations of one or more external storage units and/or RAID units to hold mirrored data.
The invention may be embodied in methods, systems, and/or configured storage media. Unless clearly indicated otherwise, discussions of any one of the embodiment types also apply to the other embodiment types. For instance, the discussions of inventive systems will also help one understand inventive methods for configuring such systems and/or methods for sending data through such systems to have the data mirrored. Computers and Networks Generally
Figure 1 illustrates a network 100 in which a local server 102 is mirrored over a conventional route 104 to a remote server 106. The conventional route 104 is not limited to telecommunication links themselves, but also includes modems, data transfer units, and other conventional tools and techniques used to send data on such links and/or to receive data thus sent. In particular and without limitation, the conventional route 104 may include the server interfaces, link interfaces, and DTUs which are illustrated in Figure 1 of U.S. Patent No. 5,537,533 and discussed in that patent.
In addition, the conventional route 104 may include Small Computer System Interface ("SCSI") performance extenders or standard Storage Access Network ("SAN") connectors. Such devices require a very high bandwidth link and minimal latency. They tend to have distance limitations of perhaps ten or twenty miles because distance introduces latency. For instance, in a single mode fiber configuration the latency on a given SCSI extender might allow a distance of perhaps fifteen kilometers between the data source and destination. Using a multi-mode fiber would reduce the distance available to perhaps two- thirds of that because of latency. Such connections have little or no tolerance for delays or interruptions longer than a few fractions of a second, or at best can only gracefully handle delays of a few seconds. These same problems apply to mainframe channel extenders. Although the network 100 as shown is configured for mirroring according to conventional tools and techniques, it is also one of the many possible networks suitable for adaptation and use according to the present invention. Such adaptation would include various steps, depending on the particular embodiment of the present invention to be used. For instance, adaptation could involve disconnecting the remote server 106 if it is no longer needed, supplementing or replacing the conventional mirroring route 104 with mirroring units linked according to the present invention, unloading mirroring NLMs or other specialty software from the local server 102, adding more local servers which will also be mirrored, and/or adding remote storage in the form of external storage volumes and/or Redundant Array of Independent Disks ("RAID") units. At a minimum, however, the adaptation normally involves the addition of at least one local mirroring unit and at least one remote mirroring unit, with the remote mirroring units capable of being linked to each other for operation according to the present invention.
Before and/or after its adaptation, the network 100 may be connectable to other networks 108, including LANs or WANs or portions of the Internet or an intranet, through a gateway or similar mechanism, thereby forming a larger network. In the illustrated network 100 the local server 102 is connected by communication links or network signal lines 110 to one or more network clients 112. Other suitable networks include multi-server networks and peer-to-peer networks. The server(s) 102 and client(s) 112 in a particular network may be uniprocessor, multiprocessor, or clustered processor machines. The server(s) 102 and client(s) 112 each include an addressable storage medium such as random access memory.
Suitable network clients 112 include, without limitation, personal computers; laptops 114, personal digital assistants, and other mobile devices; and workstations 1 16. The signal lines 1 10 may include twisted pair, coaxial, or optical fiber cables, telephone lines, satellites, microwave relays, modulated AC power lines, RF connections, a network link, a dial-up link, a portable link such as an infrared link, and/or other data transmission "wires" or communication links known to those of skill in the art. The links 110 may embody conventional or novel signals, and in particular, may embody a novel series of commands and/or data structures for mirroring data as discussed herein. The remote server 106 may store mirrored data it obtains over the conventional route 104 on an attached storage means such as an external hard disk and/or RAID subsystem 118. Examples of Flexible Mirroring Unit Systems
Figure 2 illustrates the present invention in systems according to the invention. Unlike previously discussed conventional approaches, systems according to this Figure do not require a remote server. A local server 200 or some other host 200 communicates over a local link 202 with a local mirroring unit 204. The local mirroring unit 204 communicates over a journey link 206 with a remote mirroring unit 208. Local mirroring units may include a spoof packet generator for pre-acknowledging data to the local server 200. and a nonvolatile data buffer 210 for holding mirrored data before it has been stored at the remote location. Remote mirroring units have a destination nonvolatile storage for mirrored data received from the local mirroring unit(s) 204 over the journey link(s) 206. The remote mirroring unit may be physically separated from a local server 200 by various distances, such as under ten miles, at least ten miles, or at least one hundred miles. These distances are mere examples: because the present invention can take full advantage of journey links(s) 206, systems according to the invention have no inherent distance limitations. Individual mirroring units are discussed in greater detail below, both in connection with illustrations of their flexibility in example systems shown in Figures 2 through 12, and in connection with their components and operation in general.
However, it may be helpful to note here that some embodiments of local mirroring units 204 include SCSI emulation software and/or hardware, allowing the local link 202 to be a SCSI connection through which the local mirroring unit 204 appears to the local server 200 or other host 200 as a SCSI disk or other conventional SCSI device. This may be accomplished by using within the local mirroring unit 204 a SCSI host adapter that is running in target mode instead of the more usual initiator mode. Suitable SCSI host adapters having such a target mode include at least the Adaptec 2940UW adapter, and the QLogic QLA-1040 adapter. In a similar manner, the local link 202 can be a fibre channel connection, a Universal Serial Bus ("USB") connection, a mainframe channel extender, a V.35 CSU/DSU connection, a FireWire (IEEE 1394) connection, a memory type (for instance, the AS/400 mirrors memory, not disk), an IDE bus. a PCMCIA connection, a serial connection, an Ethernet connection, a Fiber Distributed Data Interface (''FDDI") connection, or another standard bus for connecting a disk and/or a Redundant Array of Independent Disks ("RAID") subsystem to a server. Thus, conventional mirroring (in the sense of copying to another local disk) hardware and/or software can be used within the local server 200. as if the mirrored data were simply being sent to another local disk instead of being sent across the journey link 206 to a remote location.
Unlike long distance links in previously discussed conventional approaches, the journey link 206 need not be a dedicated private telecommunications link. Although such a link may still be used in some embodiments, the invention also provides mirroring units 204. 208 which communicate over a network, or a series of networks such as the Internet, using Ethernet. FDDI. V.35. or other data link protocols, the Internet Protocol (IP) or other network protocols, and/or the User Datagram Protocol (UDP). Transmission Control Protocol (TCP), or other transport protocols, without regard for the routability or non- routability of such protocols. Accordingly, the two mirroring units 204. 208 may be separated by many tens or hundreds of miles if so desired.
The journey link 206 can be fed through a conventional link 104 and a spoofing local mirroring unit 204 as the data acquisition point. However, the journey link 206 does not necessarily impose requirements of high bandwidth and low latency, which are often imposed by conventional links 104. Unlike a SAN, for instance, a system using the journey link 206 can send mirrored data from a source to a destination which is an unlimited distance away. The journey link 206 can also provide shared bandwidth, as it typically will when crossing the Internet or a wide area network. Moreover, the journey link 206 and/or the mirroring units provide inventive systems with the advantage of a relatively high tolerance for interruptions and disconnects.
The illustrated remote mirroring unit 208 has a large buffer 212. As a result, the remote mirroring unit 208 can buffer a complete volume of the local server 200 or other host 200. In some embodiments the local mirroring unit 204 also includes a large buffer. In one embodiment, for instance, the local server 200 volume and the large buffers (local and remote) can each hold up to one terabyte of data in nonvolatile storage. This buffering may be accomplished, for instance, by using the QLogic QLA-1040 adapter within the local mirroring unit 204 or the remote mirroring unit 208 to control up to one terabyte of data with no substantial modifications needed. The complete volume image of the local server 200 can therefore be stored on the buffer(s) within the mirroring unit(s). For added data recovery ability, an optional local mirror 230 may also be created: this is generally a "full" local mirror in the sense that it is consistent and available but not necessarily entirely up-to-date. This local mirroring may be accomplished in various ways. These include, without limitation, using a second local mirroring unit 204 or a second port of a multi-ported local mirroring unit 204 to mirror data to a "remote" disk subsystem that is actually geographically close to the local host 200; forking the data within the local mirroring unit 204 below the disk emulation layer of that unit 204, thereby creating another copy which is sent to a local attached disk subsystem over a SCSI or similar bus (the first copy is sent over the journey link 206 to a remote mirroring unit); and using otherwise conventional tools and techniques with the local mirroring unit 204 to create and maintain a local mirror 230.
The mirror 230 includes a copy of the server 200 volume to permit recovery in the event of hardware or software errors. However, because the local mirror 230 is local rather than remote, it does not provide substantial protection against natural disasters, civil unrest, terrorist attacks, physical vandalism, and other geographically localized risks to the server 200. Accordingly, the local mirror 230 does not provide the same degree of data protection as remote mirroring even if the local mirror 230 includes another mirroring unit 204 or otherwise embodies the present invention. The local mirror 230 is connected to the mirroring unit 204 by a path 232 which may include a conventional link such as the path 104, or a novel link according to the present invention. Although the local mirror 230 is not explicitly shown in the other Figures, one or more local mirrors may also be used with the systems illustrated in the other Figures and with other systems according to the invention. For instance, one approach uses Nonstop Networks Limited's technology or other technology to mirror between two servers; the local mirroring unit is used as the sole
(primary) disk subsystem of the secondary server. Another approach makes all mirroring internal to the pair of mirroring units by using the local mirroring unit as the sole disk subsystem for the host 200; the local mirror 230 becomes the primary disk, and the remote mirror serves as the sole true mirror. This last is a lower assurance configuration, but it may also provide higher performance at a lower cost.
Figure 3 illustrates systems in which a local server 200 communicates over a local link 202 with a local mirroring unit 204. The local mirroring unit 204 communicates over a journey link 206 with a remote mirroring unit 308. Unlike the remote mirroring unit 208 which has a large nonvolatile buffer 212 capable of holding the data from an entire local server 200 volume, the remote mirroring unit 308 has only a relatively small nonvolatile buffer 310, such as a buffer 310 holding only a few gigabytes, e.g., four gigabytes.
However, systems according to Figure 3 include a remote server 300 which has an associated nonvolatile internal or external storage. To illustrate this, Figure 3 shows a RAID unit 312 which can be controlled at some point by the remote server 300. The RAID unit 312 is "hot-swappable," meaning that a failed drive in the RAID unit 312 can be taken out and replaced while the computer 300 is running; the file system structures and other data on the replacement drive will then be built automatically. The RAID unit 312 can be viewed in some cases as part of the server 300 or connected thereto by conventional means such as means which include dedicated mirroring software on the server 300. as indicated by the arrow in Figure 3 from the RAID unit 312 to the server 300.
But the RAID unit 312 may also be connected to the remote mirroring unit 308 and the server 300 by a dual host connection in a configuration 1400 as discussed later below and illustrated in Figure 14. The dual host connection allows a switchover from a first "normal mirroring" state having a passive remote server 300, a remote RAID unit 312 or other remote disk subsystem used only for mirroring, and a local mirror and/or local host 200 disk actively used to service read requests, to a second "recovery" state having an active remote server 300 which services read requests from the mirrored data on the remote RAID unit 312 or other remote disk subsystem.
In the first (normal mirroring) state, the remote mirroring unit 308 receives data from the local mirroring unit 204 using an Ethernet and/or TCP/IP connection 206, for instance. As noted in connection with Figure 2, the local link 202 can be a SCSI bus, USB, fibre channel, or similar connection. The remote mirroring unit 308 transfers the data over a remote link 302 and remote mirroring unit 308 to the remote server 300 for subsequent storage on the hot-swappable RAID unit 312, or directly from the remote mirroring unit 308 to the RAID unit 312 if the dual host connection 1400 is being used. The remote link 302 can be a SCSI bus connection, for instance, so the remote mirroring unit 308 appears to the remote server 300 to be a SCSI disk, for instance, which is to be mirrored by the remote server 300 to another "disk," the RAID unit 312. The remote link 302 can also be a serial, Ethernet. FDDI, USB, fibre channel, or other nonproprietary connection.
The local mirroring unit 204 has a nonvolatile buffer which is similar or identical (except with respect to specific data stored in it) to the small buffer 310 of the remote mirroring unit. Data from the local server 200 is pre-acknowledged into the local mirroring unit 204 buffer. As far as the primary server 200 is concerned a second "mirrored" write has occurred locally. In reality, the local mirroring unit 204 has received this data and stored it on this local buffer. The local mirroring unit 204 stores this sector and track change data (or similar block level data) until the local mirroring unit 204 can safely send the data over the journey link 206 to the remote mirroring unit 308. The smart buffer in the local mirroring unit 204 stores any data above what the journey link 206 can handle locally. Such data is stored in the local mirroring unit 204 until the remote mirroring unit 308 has successfully written to the remote server 300 and sent back an acknowledgement to the local mirroring unit 204. When this acknowledgement is received the local mirroring unit 204 eliminates the successfully transmitted piece of sector/track/block data from the local nonvolatile buffer. Unlike conventional systems, neither server 200, 300 necessarily requires an NLM or other software designed specifically for data mirroring, as opposed to standard file system and operating system software. Figure 4 illustrates systems having several components which are discussed above, as indicated by the use of the same identifying numbers in the Figures. However, in the systems of Figure 4 a remote mirroring unit 408 includes both a small nonvolatile buffer 310 and a large nonvolatile buffer; the large buffer is implemented as a hot-swappable RAID unit 312 which connects directly to the remote mirroring unit 408. The small buffer 310 is used to buffer data received over the journey link 206, allowing the data to be acknowledged back to the local mirroring unit 204, and buffering the data until it can be stored by the remote mirroring unit 408 in the large buffer 312. No remote server is needed. Figure 5 illustrates systems in which two or more local servers 200 write to a remote mirroring unit 508. In this Figure and elsewhere, references to the local server 200 should be understood to also generally include hosts 200 which are not servers. That is, the invention can be used to mirror any host computer system 200 that will connect to a mirroring unit 204. Servers are a widely recognized example of suitable hosts 200, but other suitable hosts 200 include clusters, computers which are not servers, mainframes, and Storage Access Network ("SAN") or Networked Attached Storage ("NAS") data sources. The local servers 200 or other hosts 200 may be physically separated from one another by various distances, such as under ten miles, at least ten miles, or at least one hundred miles. In the systems of interest for this Figure, each local server 200 in a particular system relies on the same operating system and file system platform, but different systems according to Figure 5 may use different platforms. For instance, each server 200 could be a Novell NetWare server in one such system, and each server 200 could be a Microsoft Windows NT server using the NT File System ("NTFS") in another such system.
Each host 200 in the system is connected by a SCSI, fibre channel, USB, serial line, or other standard storage subsystem or other peripheral connection 202 to its own local mirroring unit 204. The local mirroring units 204 are connected by journey links 206 to a single remote mirroring unit 508. The remote mirroring unit 508 has a SCSI, fibre channel. USB, or similar controller card for each of the local mirroring units 204.
The data from each local mirroring unit 204 can be transferred directly (i.e., not through a remote server) to an individual hot-swappable RAID storage unit 312 in a group 512 of RAID units, by a SCSI, fibre channel, USB, or similar connection within the remote mirroring unit 508. The RAID units 312 may be physically external to at least a portion of the remote mirroring unit 508, such as a portion containing an Ethernet card for connection to the journey link 206. However, the remote mirroring unit 508 is defined by functionality rather than packaging. In particular, the RAID units 312 are considered part of the remote mirroring unit 508 unless indicated otherwise (e.g.. in discussing Figure 14). Each RAID storage unit 312 has a remote bootable volume, and the data is written in sector/track or block fashion. The illustrated remote mirroring unit 508 also contains a small buffer 310 to allow acknowledgment and buffer of data received over the journey links 206. Figure 6 illustrates systems similar to those shown in Figure 5, but a remote mirroring unit 608 writes to external bootable storage volumes 614 in a group 616 of such volumes. Local servers 200 running on the same platform write to "disks" which are actually local mirroring units 204, which in turn write the data to the remote mirroring unit 608. The remote mirroring unit 608 has a SCSI, fibre channel, USB, or similar controller card and a bootable storage volume 614 corresponding to each local mirroring unit 204. The data from each local mirroring unit 204 will be transferred from the remote mirroring unit 608 directly to the corresponding storage volume 614 using a SCSI bus or other data line. Each volume 614 is a remote bootable volume, and the data is written in sector/track or block fashion. In alternative embodiments of a system generally according to Figure 6 and in other systems as well, separate partitions may be used to hold the mirrored data of respective local servers 200, instead of holding that mirrored data in corresponding separate disks 614 (e.g., as in Figure 6) or separate RAID units 312 (e.g., as in Figure 5). In various many-to-one systems it may be necessary to start a process which forks itself as new connections are made and locks volume mirrors from multiple mirror attempts using an IPC or other mechanism.
Figure 7 illustrates systems in which a remote mirroring unit 708 includes both an individual external storage volume 614 and a RAID unit 312. The mirrored data is stored by the remote mirroring unit 708 on both storage subsystems 312, 614, to provide extra assurance that the data will be available when needed.
Figure 7 also illustrates systems in which two or more local mirroring units 204 write to one remote mirroring unit 708 with all mirrored data for the several local servers 200 going to one large storage volume (312 or 614 or both, in various embodiments) which is mounted directly on the remote mirroring unit 708, instead of dividing the mirrored data among several remote storage units 312 or 614 as illustrated in Figures 5 and 6, respectively. The volume used by the remote mirroring unit 708 has a partition for each local mirroring unit 204. Each partition provides a remote bootable "volume," and the data is written in sector/track or block fashion as usual.
In an alternative system which is also illustrated by Figure 7, the mirrored data is divided between two or more storage units which are connected directly to the remote mirroring unit 708, with a given storage unit holding the mirrored data for a given local mirroring unit 204. However, a mixture of external disks 614 and RAID units 312 is used, unlike the systems that use RAID units only (Figure 5) or external disks only (Figure 6). For instance, an external disk 614 holds the data from a first local mirroring unit 204, while a RAID unit 312 holds the data from a second local mirroring unit 204. In such systems, the remote mirroring unit 708 has a SCSI, fibre channel, USB, or similar controller card corresponding to each local mirroring unit 204, and the data from each local mirroring unit 204 will be transferred directly (without a server such as server 300) to an individual external hot-swappable RAID storage unit 312 or external bootable drive 614 via a SCSI, fibre channel, USB, or similar communications line.
Figure 8 illustrates systems like those discussed in connection with Figure 5. However, in the systems of Figure 8, the local servers 200 rely on different platforms, as indicated by the presence of several numbers 822, 824, 826. Of course, systems according to this or other Figures do not necessarily have exactly three local servers 200 and corresponding local mirroring units 204; they merely have two or more pairs, with a server 200 and corresponding local mirroring unit 204 in each pair. For example, one system according to Figure 8 includes a Novell NetWare server 822 and a Microsoft Windows NT server 824, while another system according to Figure 8 includes two Novell NetWare servers 822, 826 and a Microsoft Windows NT server 824.
Figure 9 illustrates systems like those discussed in connection with Figures 5 and 8. Unlike Figure 5, however, the local servers 200 rely on different platforms, and unlike Figure 8, the remote mirroring unit is a unit 608 which uses a group 616 of external disks 614 instead of a group 512 of RAID units 312.
Figure 10 illustrates systems like those discussed in connection with Figure 7. However, the local servers 200 in systems according to Figure 10 rely on different platforms. As with Figure 7, the local mirroring units 204 may be mapped in some systems to partitions or to storage units. When mapping to partitions, the local mirroring units 204 may be mapped to partitions within a RAID unit 312, to partitions within an external drive 614, or to partitions within a RAID unit 312 which are also mirrored to an external drive 614. When mapping local mirroring units 204 to storage units, one or more local mirroring units 204 may send their data through the remote mirroring unit 708 to corresponding external drive(s) 614 while one or more other local mirroring units 204 send their data through the remote mirroring unit 708 to corresponding RAID unit(s) 312.
Figure 1 1 illustrates systems in which data is mirrored to two or more remote locations. Such systems are a counterpart of the systems illustrated in Figures 5-10, in the sense that Figures 5-10 illustrate "many-to-one" mirroring systems (more than one local server mirrored to one remote destination) while Figure 1 1 illustrates "one-to-many" mirroring systems (one local server mirrored to more than one remote destination). In general, the local mirroring units 204 will all be mirroring the same data, but using multiple local mirroring units 204 permits mirroring across at least one journey link 206 to continue uninterrupted despite the unavailability of a given local mirroring unit 204. The local links 202 may all use the same type of connection, or different connections may be used. For instance, one local link 202 may be a SCSI connection while another local link 202 is a USB connection. The journey links 206 may also be uniform or varied. Likewise, the remote mirroring units may each have the same components (e.g., each may use a RAID unit 312), or they may use different components at the different locations.
Figure 12 illustrates systems which resemble those illustrated by Figure 1 1 in that data is again mirrored to two or more remote locations. However, the local mirroring unit 204 of Figure 12 is a multi-port mirroring unit. That is, it can be connected simultaneously to more than one journey link 206 in a manner similar to the simultaneous connection of a conventional multi-port server. The multi-port mirroring unit 204 sends mirror data from the host 200 over each of the active connections 206, thereby helping mirror the host 200 to several remote locations which may be miles apart from one another. The multi-port local mirroring unit 204 needs only one local buffer, and like mirroring units 204 in other systems it optionally includes a full local mirror 230. More on Mirroring Units
The components and operation of mirroring units are discussed above in connection with Figures 2 through 12. A given piece of additional information provided below does not necessarily pertain to every mirroring unit in every system according to the invention, but this additional information is helpful nonetheless in understanding how the mirroring units permit greater flexibility to the people and enterprises that are responsible for ensuring that data is properly mirrored. At least some of the mirroring units can reliably emulate disk drives connected by
SCSI, fibre channel, USB, or similar connections through standard server drivers running under Novell NetWare and/or Microsoft Windows NT platforms. SCSI, fibre channel, USB. or similar emulation under other operating systems may also be provided.
Each of the local and remote mirroring units is preferably configured so that it supports I/O through a monitor, keyboard, and a mouse plugged into it. Some mirroring units have a network address and otherwise allow a network administrator to access a specific mirroring unit on the adapted network 100, through a web browser on a remote workstation 1 16 or by other means.
The mirroring units are preferably Simple Network Management Protocol ("SNMP") capable. The network administrator has remote access to both the local and remote mirroring units. The mirroring unit 204 software provides an interface to monitoring utilities. In particular, each local mirroring unit 204 acts like a network agent in that the unit 204 tracks the number of writes/reads to the local server 200, the status of each local server 200. number of restarts/warm starts of each local server 200, and so forth, and generates SNMP traps when necessary. The following pieces of data may also be provided to administrators by the local mirroring unit 204: the number of blocks currently in the buffer 210: an alert when the buffer 210 fills up and/or fills beyond some specified threshold; the number of blocks sent since server 200 startup; and the number of blocks received since server 200 startup. Some local mirroring units 204 also have incremental dial-up options. If a customer is using the mirroring unit 204 with a dial-up connection, and doesn't want to be connected at all times, the unit 204 provides an option to send data over the journey link 206 at specified times. Also, the local mirroring unit 204 may have a setting that does not allow data to be sent during periods of high traffic on the adapted network 100 or another portion of the journey link 206. The buffer 210 in the local mirroring unit 204 should be large enough to buffer data received from the local server 200 during these periods of non- transmittal. More generally, the local mirroring unit 204 preferably matches the performance of a high-speed RAID disk subsystem in terms of data transfer rates, reliability, and compatibility with existing platforms on servers 200. Because an implementation which is primarily in software is unlikely to meet these performance goals, the local mirroring unit 204 preferably comprises special-purpose hardware. The design and construction of suitable software and hardware, including any necessary firmware, may be accomplished by those of skill in the art, with particular attention to the conventional mirroring path 104: the SCSI controllers identified herein or similar SCSI, fibre channel, USB, or similar controllers; individually known subsystems such as buffers 210, 212. 310, disks 614, and RAID units 312, and their interfaces; software such as FreeBSD drivers; Ethernet and other individually known Network Interface Cards ("NIC"); network protocols such as Ethernet and TCP/IP protocols; the descriptions and examples provided herein; and other tools and techniques now or subsequently available to such persons.
Writes to the local mirroring unit 204 should normally be acknowledged and written to the local buffer 210, and may also be written to a full local mirrored volume 230 over a conventional path 104 or another path, although such local mirroring is not explicitly shown in Figures 3 through 12. For performance, it is generally acceptable to buffer the writes through a RAM cache in the local mirroring unit 204 or the local server 200 or both. In particular, an implementation may take advantage of an available hardware RAID unit 312 cache or other SCSI, fibre channel, USB, or similar cache. Reads from the local mirroring unit 204 should generally be serviced with the proper data from the local mirror 230.
When the local mirroring unit 204 comes back on-line after a crash or a reboot or any other kind of service interruption, it will automatically begin sending data from its local buffer 210 to the remote mirroring unit 208, 308, 408, 508, 608 or 708. The local mirroring unit 204 should not issue a SCSI, fibre channel, USB, or similar reset, as this may crash the host machine 200. Data written to the local mirroring unit's buffer 210 should be sent over the network or other journey link 206 in a first in, first out fashion, to the remote mirroring unit. This may be done using the TCP/IP or another journey link protocol. The remote mirroring unit preferably maintains a full, consistent, mirror so the remote volume is usable and mountable by an operating system at all times regardless of mirror synchronization status.
At least in embodiments utilizing FreeBSD-based software, kernel panics should preferably not occur on the local mirroring unit 204 unless there is a failure of essential mirroring hardware or software. Misconfiguration of the local mirroring unit 204 software should preferably not result in a system shutdown, nor should any behavior of the host server 200. It is preferably possible to reconfigure the mirroring unit software without a reboot; a unique version number should accompany each software change. Accordingly, the software preferably reads all initialization information and configures itself accordingly through a system call which is available to an administrator without interrupting data processing by the mirroring unit. The host server 200 should not be interrupted. The local mirroring unit 204 preferably accepts writes from the host system 200 whether or not the remote mirroring unit is on-line, and whether or not network or other journey link 206 bandwidth is available, unless the local buffer 210 is full. If the local buffer 210 fills, the local mirroring unit 204 preferably continues to maintain a local mirror 230 (if present), and preferably continues to dequeue a circular queue of data from the local buffer 210. However, the local mirroring unit 204 preferably stops adding to the queue until told to begin queuing again by a user (typically an administrator) process. A system call, rather than a reboot, preferably allows user-space processes to disable and re-enable local buffer 210 queuing.
The mirroring units preferably auto-detect the disappearance of and reconnection of network or other journey link 206 bandwidth. For instance, disconnecting the local mirroring unit's Ethernet cable and then reconnecting it the next day preferably results in zero data loss and requires no intervention on the network operator's part, provided there is enough space in the local buffer 210 to hold the data changes accumulated while the local mirroring unit 204 was disconnected.
Monitoring software in the mirroring unit, or in connection with those units, preferably determines whether the system was shut down cleanly after the previous boot so that the monitoring software can determine the likelihood that the remote mirror is out of synch. The local mirroring unit 204 preferably loses as little data as possible in the event of power failure. Some mirroring units therefore contain an Uninterruptible Power Supply ("UPS"). It may be assumed there will be time to flush RAM-buffered writes to the local mirror (if present) and/or local buffer 210 in the event of power loss. In one embodiment, the mirroring unit operating system (e.g.. FreeBSD) boots from the hard disk in a read-only mode to avoid filesystem problems with FreeBSD itself. Configuration data is written to a smaller partition and can be restored either from the identical information on the mirroring unit peer, or by sending out a SNMP alert that the mirroring unit has lost configuration data and will be off line until it is restored. The alert can be used if the peer mirroring unit is not reachable. Some embodiments also avoid controller card initialization routines that disk drives would not perform on their own, to avoid bus resets for instance. Also, if the mirroring unit buffer fills up it may be better to simply acknowledge the write and mirror it locally while sending an alert that the buffer is full and the remote mirror is out of sync with the local mirror.
As noted, it is preferably possible to cold-reboot the local mirroring unit 204 without disturbing the host system 200, especially with regard to SCSI, fibre channel, USB, or similar handshaking. The local mirroring unit's buffer 210 retains the order of write requests and transmits them to the remote mirroring unit in the same order they were received by the local mirroring unit 204, to preserving data consistency at all times.
The remote mirroring unit receives TCP Protocol Data Units (also referred to herein as TCP packets), for instance, from the local mirroring unit 204 and writes them to a disk subsystem (such as an external drive 614 or a RAID unit 312) such that the drive is at least logically block-for-block the same as the local mirror 230, if any, and the host 200 volume at a previous time. The mirrored data may be out of date, but it must be consistent.
For data recovery purposes, the remote mirroring unit software preferably has an interface to user-space so that user-space programs can disable or re-enable reading, writing, and/or seeking of the remote mirror by the mirroring unit software, allowing the remote disk subsystem — and hence the mirrored data — to be accessed by a second SCSI host on the same chain. At the remote site, the remote mirroring unit and a backup host server will be attached to the shared disk subsystem. For instance, the remote mirroring unit may use SCSI ID 6 while the remote server used for restoration uses SCSI ID 7. While the remote mirroring unit is mirroring, the remote host will leave the shared drive unmounted. For data recovery, as part of a switchover the remote mirroring unit will stop accessing the shared drive and the backup host server can mount it.
The remote mirroring unit preferably reports to user-space programs the number of blocks received from the local mirroring unit 204. The remote mirroring unit mirrors to the disk subsystem such that the volume can be mounted by a host system with the same operating system as the local server 200 that created the local volume. If the remote mirroring unit receives a request from the local mirroring unit 204 to write to logical block number N, then the data should be written to logical block number N on the remote mirroring unit's disk subsystem 312 or 614. Write requests from local mirroring unit 204 should be written to the remote mirroring unit's disk subsystem 312 or 614 in the order in which they were received by the local mirroring unit 204, to preserve data consistency.
In the journey link 206, communication between the local mirroring unit 204 and the remote mirroring unit can use the TCP protocol, since it features error recovery and transmission guarantee. The remote mirroring unit software acts as a TCP server; the local mirroring unit 204 acts as the remote unit's client. A loss of network bandwidth or connectivity preferably does not interrupt either the local mirroring unit 204 or the remote mirroring unit. Likewise, data recovery at the remote location preferably does not interrupt the local mirroring unit 204. If the connection between the local mirroring unit 204 and the remote mirroring unit times out or is otherwise broken, the local mirroring unit 204 preferably attempts to reconnect until a connection is re-established. Then the local mirroring unit 204 preferably continues sending mirror data where it left off and otherwise resumes normal operation.
The inventive mirroring units are more "intelligent" than the original Off-SiteServer product in that the inventive mirroring units run a modified operating system which is based on the FreeBSD UNIX operating system. One modification included altering the driver for the QLogic SCSI controller to make the card act as a SCSI target rather than a host, so it emulates a disk drive; other controllers could also be used, with suitable drivers. The boot process was also modified to show a mirroring unit configuration utility on the console in place of a login prompt, and the kernel was recompiled. At the source each mirroring unit 204 is running an operating system that allows it to run fully independently of the host server 200. As a result one of the flexible mirroring characteristics provided is that the mirroring unit 204 does not require initialization or connection software on the host server 200 (on the original Off-SiteServer product this software took the form of a Vinca NLM). Instead, the mirroring unit 204 operating system emulates a SCSI or other standard disk or data acquisition point. So the mirroring unit 204 can be mounted, for instance, as a mirrored SCSI disk under any operating system that supports SCSI, including at least the Microsoft Windows 95, Microsoft Windows 98, Microsoft Windows NT, Novell NetWare, FreeBSD, and Linux operating systems. The disk emulation is preferably carried through to the point that any standard disk operation can be performed (at least from the server 200 perspective), including handling server 200 requests for disk formatting, disk partitioning, disk integrity checks such as scandisk, and so on, in addition to disk reads and disk writes. A system according to the invention can also maintain a full mirrored volume 230 locally for fault tolerance. Because this mirroring operation occurs by forking the data (or doing two writes) below the emulation layer of the software in the mirroring unit 204, the mirroring unit 204 is able to maintain this local volume 230 along with a sequential data change buffer. This allows the mirroring unit 204 to service local reads by the server 200 without excessive latency, which in turn allows the system to run without a disk handicap and no split-seeks software, eliminating a potential software compatibility problem. This also allows the inventive system to mirror data back to a local disk of the server 200 under local disk mirroring instead of going over the journey link 206. In addition, if a local mirror 230 is maintained then the local mirroring unit 204 need not include a spoof generator to pre-acknowledge writes back to the host 200, because the local mirror 230 is not subject to the delays and risks associated with sending mirrored data over the journey link 206.
A mirroring unit according to the invention normally includes operating system software. Accordingly, at least some mirroring units can run multiple "host" applications to manipulate the mirrored data they have acquired. The system can also be scaled up or down to meet requirements in a particular environment, using drivers and/or other appropriate software and/or hardware. For example, processes could be spread across multiple processors, SCSI cards, and/or other "intelligent" devices to handle more activity and workload. Likewise, a system can be scaled down to reduce costs while still meeting the needs of lower performance environments. With appropriate software the local mirroring unit 204 can run as an independent intelligent disk subsystem, or it can run an emulation of the host 200 operating system as a fail-over for local fault tolerance. The local disk volume 230 can serve as a local mirrored replacement for local fault tolerance if the host 200 disk subsystem crashes.
The system maintains consistency and availability at the remote location in part by an intelligent buffer 210 that maintains and sends data on a first-in-first-out basis. In this way data blocks are transmitted to the remote location in the exact order they are received through the emulation layer at the local mirroring unit 204. Sequence numbers and/or timestamps may also be used, since packetized data does not necessarily arrive at the destination in the same order it was sent.
?7 Some embodiments use the following approach with a circular buffer and other means for protecting data in the event of a shutdown. In addition to the QLogic card used as the disk target emulator, the local mirroring unit has two disk systems attached to it through a local SCSI disk controller. One disk contains the host operating system (e.g., FreeBSD 3.1) on it, with associated utilities and mirroring unit administrative software. This disk also serves as a buffer 210 disk. The other disk system attached to the mirroring unit is at least as large as the host 200 disk being mirrored and serves as the local mirror 230 of the host 200 disk.
SCSI data is read off of the QLogic card and evaluated in the kernel as read or write requests. Read requests that come from the QLogic card are preferably fulfilled using the local mirror disk 230 and not be sent across the network 206. Write commands are copied directly to the local mirror disk 230 and acknowledged to the host system 200 as soon as possible (but not necessarily pre-acknowledged), as well as added to a circular queue on the buffer disk or in nonvolatile RAM. Every time a block is written to the circular queue two blocks are actually be written sequentially, one being the actual data block to be transmitted, and the other being a timestamp for the current tail pointer for the queue, possibly with other data such as LBN (logical block number). This second block is a so-called meta-data block. This approach is not space efficient, but it reduces the number of disk writes required to maintain the queue pointers. Queue pointers may also be maintained by keeping a copy of at least them, and possibly the entire circular queue, in nonvolatile RAM if such RAM is available. A way to save both space and time is to write to the circular buffer in larger chunks at a time, buffering blocks in memory until enough accumulate to perform a write. This allows the meta-data block to be used for many data blocks, lessening the number of disk write operations and saving on disk space.
In the event of a system shutdown and restart, the head of the queue is found by searching for the block with the most recent timestamp in its meta-data segment, and then using that meta-data segment to locate the tail pointer. This can be done, for instance, by performing a binary search. Since the buffer implementation is circular it is not necessary to remove transmitted blocks physically from the buffer (i.e.. by deleting or zeroing them); incrementing the tail pointer effectively does this. Buffer full conditions are detected when the head pointer is one less than the tail pointer. Pointers refer to positions in the circular buffer and not to the data in the buffer itself (i.e., it's an array not a linked list). It may not be necessary to keep a 64 bit timestamp, since having the most recent second may be enough to determine the last block written before the system shutdown. For example, assume four blocks were written in the same second and have the same timestamp. Then the last block according to the timestamps is the one last written, since this is a ordered queue. If timestamps are too computationally expensive a simple incrementing counter may suffice, though it could roll-over sooner than the year 2038. The queue buffer size changes, depending upon the end-user's data change rate and the length of time the customer needs to withstand a network 206 outage. The queue buffer could be as small as a few hundred megabytes, or as large as the host volume being mirrored. There are no inherent restrictions on the minimum or maximum size of the buffer, and in cases where high data change rates and frequent lengthy interruptions of the journey link 206 are anticipated, the buffer may need to be larger than the host volume being mirrored.
A separate process, which may run in user-space or system-space, reads blocks out of the circular queue and sends them across the network 206 to the remote mirroring unit. This transmitting process can inform the queuing process from time to time as to the transmitting process's current pointer position and can watch the timestamps to determine when the queue is empty. It may be fine if the tail pointer being saved in the meta-data is a little out of date, because in the worse case the system will end up resending a number of blocks it has already sent, provided the resend number does not grow to an excessive size in the event of a system restart. Preferably, the transmitting process can also determine the number of blocks since server startup. In some cases it can be presumed that the buffer will be able to buffer the entire host volume. Under a "do no harm" philosophy it would be better to not take a risk of slowing the SCSI bus down and simply dump data that will not fit into an already full queue, and inform user-space monitoring processes of this event. To attempt to reduce the number of resent blocks, the system may check writes against the local mirror and only add them to the circular buffer if they are indeed different, while avoiding any lazy write problems. This might be accomplished by maintaining a hash table of checksums for each LBN on the disk; one tradeoff would be processor time computing checksums and memory vs. additional disk operations. Methods Generally
Figure 13 illustrates methods of the present invention for remote data mirroring. Some methods include steps for installing mirroring units; for convenience, these steps are collectively identified as steps within an installing step 1300. For instance, system integrators, mirroring equipment vendors, and administrators may be licensed to perform some or all of the steps shown within step 1300 when installing systems such as those illustrated in any of Figures 2 through 12. Other methods of the invention include steps for transmitting data to one or more mirroring units; for convenience, these steps are collectively identified as steps within a transmitting step 1302. These transmitting steps may be performed under license with test data by installers as part of the installing steps 1300, but they may also be routinely performed with mission-critical data at the behest of regular users of a system according to the invention.
During a connecting step 1304, at least one server 200 is connected to at least one local mirroring unit 204. As discussed above, this connection may be in the form of a SCSI bus, a fibre channel connection, a USB connection, or some other standard disk subsystem bus. Because the one local mirroring unit 204 emulates a disk subsystem, connecting it during step 1304 is basically the same as connecting a conventional disk subsystem to the server 200, at least from the point of view of the server 200. In particular, no special mirroring NLM or other mirroring software installation is required.
During a connecting step 1306, at least one local mirroring unit 204 is connected to at least one corresponding journey link 206. Depending on the situation, this may involve various operations. For example, if the journey link 206 includes a local area network then the local mirroring unit 204 may be connected to that network like other network nodes; SNMP support may also be configured. If the journey link 206 includes a dial-up connection from the local mirroring unit 204, then the dial-up parameters are configured. Likewise, if the journey link 206 includes a dedicated private telecommunications line such as a Tl line, then familiar operations are performed to make the connection.
During a connecting step 1308, at least one remote mirroring unit 208, 308, 408, 508, 608 or 708 is connected to at least one corresponding journey link 206. This may be accomplished in generally the same manner as the connection of the local mirroring unit 204 during step 1306. However, when the remote mirroring unit acts as a TCP server in a given embodiment, the local mirroring unit 204 acts as the remote mirroring unit's client. Thus, in such embodiments the connecting step 1306 connects a TCP client while the connecting step 1308 connects a TCP server.
During a testing step 1310, tests are performed on the mirroring unit(s). These tests may include, for instance, comparing throughput performance of the local mirroring unit 204 with the performance of a RAID unit; re-mirroring data from the remote site back to the local site; putting incorrect configuration information into the local mirroring unit 204 and then correcting that information; rebooting the local mirroring unit 204; disconnecting the journey link 206; interrupting power to the local mirroring unit 204; interrupting power to the remote mirroring unit; overflowing the buffer 210 of the local mirroring unit 204; and other tests. In particular and without limitation, the testing step 1310 may involve performing one or more of the tests described in the "test suite" section of this document. Testing 1310 may also involve transmitting data as discussed below in connection with step 1302, but testing is shown as a separate step in Figure 13 for clarity of illustration.
The transmitting step 1302 may include a transmitting step 1312 which sends data from the server 200 over a standard bus to the local mirroring unit 204. This is possible because the present invention, unlike the conventional path 104, provides a mirroring unit which emulates a disk or RAID subsystem.
During a transmitting step 1314, the data being mirrored is transmitted over the journey link 206. As noted, this may be done with a dedicated link as was the case with the conventional path 104, but it may also be done using standard protocols such as Ethernet and/or TCP and/or other open standard protocols, including their associated conventional networking infrastructure such as local area networks and/or the Internet.
In some embodiments, the mirrored data is time-stamped by the local mirroring unit 204 to maintain a record of the sequence in which blocks of data were mirrored and to also tie the data to a particular point in time. This is coupled with remote and/or local data storage large enough to hold one or more snapshots of the mirrored volume plus incremental changes at the sector/track/block level to that volume, instead of simply holding a current copy of the mirrored volume. In a preferred embodiment only one snapshot is needed. The single snapshot provides a baseline, and subsequent changes are journaled so that the state of the volume at any desired point (subject to the journaling granularity) can be recovered. The journal may be arbitrarily large with additional storage space added as needed to hold it, or it may be kept in a FIFO circular buffer of some fixed size, with older journal entries overwritten by new ones after the journal buffer is initially filled. More generally, suitable re-mirroring software plus the snapshot(s) and (if necessary) the incremental changes can be used at a later time to reconstruct the mirrored disk volume as it existed at a specified previous time.
During a transmitting step 1316, the data being mirrored is transmitted to a serverless remote mirroring unit. This configuration is illustrated by Figure 2, for instance. The remote mirroring unit is not a conventional server, although it has hardware and functional capabilities in common with such servers. Servers provide more general functionality than mirroring units; mirroring units are focused on effectively providing substantially continuous, nearly real-time remote data mirroring. The remote mirroring unit behaves like a remote mirroring server with regard to acquisition of data over the journey link 206 but otherwise strongly resembles a mounted disk. In particular, the remote mirroring unit behaves like a disk or RAID unit with regard to a secondary server if one is attached. No secondary server is needed for the remote mirroring unit to re-mirror all the data back over the journey link 206 toward the local server 200 if that becomes necessary. After data is transmitted from the local mirroring unit 204 to a remote mirroring unit at the destination, the remote mirroring unit can do various things. For instance, the remote mirroring unit may simply convert the received data packet into data blocks that are written out to a single external disk 614. The remote mirroring unit may convert these data packets into disk blocks and write them to an internal disk subsystem and/or disk partition. The remote mirroring unit may receive the packet data, convert it to disk data blocks, and write them to a RAID unit 312 in the form of an external data subsystem utilizing internal striping (RAID) software to stripe data across multiple disks on an "unintelligent" disk subsystem. This same conversion from packets to disk block data to striped (RAID) data could also occur through a hardware controller and related driver, with storage to an external "unintelligent" disk subsystem. The remote mirroring unit may also write to an external intelligent RAID subsystem 312, with disk blocks being written to the disk subsystem in a data stream and striped by the intelligent RAID subsystem.
Rather than write the received data immediately to the remote mirror 312 or 614, the remote mirroring unit may write the data first to a remote buffer and then send an ACK with some type of "signature" of the data (such as a checksum or Cyclic Redundancy Check value) back to the local mirroring unit. The local mirroring unit would then either ACK- ACK or NAK-ACK (based upon verification of the signature) the data; only upon receiving an ACK-ACK from the local mirroring unit would the remote mirroring unit commit the data from the remote buffer to the remote mirror. In such embodiments, if the remote mirroring unit receives not only the data, but also an original signature from the local mirroring unit, it will NAK the original data transmission if the original signature does not verify correctly. More generally, various approaches to ACKing data are possible. For instance, one may view the remote mirroring unit and the local mirroring unit as being peers, rather than either being a subsystem of the other. In this case, on the remote mirroring unit, ACKs trickle up from the remote mirror disk itself (probably from its cache); on the local mirroring unit, ACKs also trickle up from the local mirror disk itself (probably from its cache); but on the local mirroring unit, ACKs would not be needed from the remote mirroring unit, only from the local end of the journey link, before ACKing the host. It would still be prudent on the local mirroring unit to wait for an ACK from the remote mirroring unit before deleting blocks from the local buffer, but this can be done long after ACKing the host.
Additional steps are possible if at least one secondary server 300 is present in the system. For instance, the remote mirroring unit may relay data directly to a remote server 300 through the server's network operating system. This operating system can be in an active or passive state. In either case data received through the connection 302 can be written to an internal local disk subsystem through the server 300 operating system. This approach requires specific software for each operating system at the remote location. The remote mirroring unit may also use an Internet-based data window to send and receive data between the remote mirroring unit and a secondary server 300. This data window could be through a plug-in extension to browser interfaces or though Internet component extensions to the core operating system, such as Microsoft ActiveX extensions.
In any of the scenarios above, the local mirroring unit may be "intelligent" enough to relay mirrored data to one remote mirroring unit or to many remote mirroring units; a one- to-many system like that shown in Figure 12 has three remote mirroring units connected by respective journey links 206 to a single multi-ported local mirroring unit 204 and multi-port mirroring units may likewise be used, alone or in combination with single-port mirroring units, in other systems according to the invention. There is no hard limitation on the number of remote mirroring units in a given system.
The remote mirroring unit can also relay mirrored data to a nearby mirroring unit and/or another more distant remote mirroring unit for further fault tolerance. A remote mirroring unit can act as a head end to load balance between two or more following remote mirroring units to distribute loads and provide fault tolerance, with appropriate attention to continuous consistency and completeness of the data mirrors. N remote mirroring units can be connected to each other and maintain the same network address or Domain Naming System ("DNS") name to provide further fault tolerance. Combinations of these various approaches can also be used.
In embodiments having one or more separate fully independent remote disk subsystem(s) connected to the remote mirroring unit, the remote mirroring unit behaves as a SCSI master (for instance) and writes data out to the remote disk(s). If a secondary server 300 is present, this server 300 follows both the remote mirroring unit and the remote disk subsystems(s) in the SCSI chain. During data mirroring, the secondary server 300 is typically a slave and/or in a passive state. In the event of failure of the mirrored local server 200, the remote server 300 mounts the external volume(s) and becomes a SCSI master. At the same time the remote mirroring unit dismounts its remote disk subsystem driver and goes into a passive (slave) state.
In particular, this can be accomplished using a configuration like that shown in Figure 14, which includes a "dual host" connection 1400. Under many conventional approaches, only one host adapter lives on a SCSI chain, typically as LUN 7. During power up or reset, the host cycles through all the other LUNs to determine what is connected. If a system uses a dual host capable adapter then the second host typically lives at LUN 6, and it will only reset or interrogate LUNs 0-5. Thus LUN 7 might be considered the primary and LUN 6 a secondary. In any event both hosts have "access" to the lower ordered targets when connected as shown in Figure 14. Dual host connections themselves are not new. In particular, a dual host connection with BusLogic EISA cards and a Novell NetWare server is known. However, the inability of that Novell server to refresh its file allocation table on a demand basis rendered moot the capabilities provided by the dual host connection in that case. General information about dual host connections is publicly available from sources wh.ch include an online SCSI FAQ. If a dual host connection is not used, then the remote server 300 requires a driver, NLM, and/or other software dedicated to mirroring so the remote server 300 can receive mirrored data directly from the remote mirroring unit and store it for possible later use.
In embodiments according to the present invention which use a dual host configuration 1400, the remote mirroring unit 208, 308, 408, 508, 608, or 708 controls the RAID unit 312 or other remote disk subsystem until such time as it is commanded to stop so that a switchover can be performed. During this time the remote mirroring unit performs remote data mirroring and as SCSI master it sends data to the RAID unit 312 as discussed elsewhere herein. During this time the Novell or other secondary server 300 is in a passive (dismounted) state. This prevents damage that would otherwise occur by wiring together the server 300, remote mirroring unit, and RAID unit 312 or other remote disk subsystem in a two-to-one manner as shown in Figure 14.
To perform a switchover, the remote mirroring unit dismounts the RAID unit 312 driver and the server 300 mounts the RAID unit 312. The server 300 then becomes the SCSI master. Since one cannot necessarily predict or enforce the secondary server SCSI card selection, the remote mirroring unit preferably has the secondary host position (LUN 6). As the two machines come up, the remote mirroring unit may experience a second reset as its driver powers up. This is normal, but the remote mirroring unit should be able to recover at the device driver level. Note that by utilizing the dual hosting (not dual channel) method, the cabling becomes a normally terminated SCSI chain; no additional hardware is required. The switchover can be accomplished entirely by software, through storage subsystem and/or driver dismount, mount, and related operations.
The previous discussion may be viewed as implicitly assuming a one-to-one relationship between a remote mirroring unit and a secondary server 300. However, a software or mechanical SCSI switch (for instance) could be employed to allow connection of a remote mirroring unit to several potential host servers 300. In protocols such as fibre channel and/or in SAN architectures there is not a traditional SCSI master/slave relationship. There is instead an address relationship that occurs through DNS and/or numeric addresses. In such systems, the switch-over would occur though an address change, with the remote mirroring unit still going into a passive state.
The remote mirroring unit can be made to run a full network operating system. In the event of a disaster such a remote mirroring unit could go into an active state and become a fully functional server for the information on the disk subsystems to which it sent mirrored data. The remote mirroring unit could also run an emulation program that would allow it to emulate a server under a specified host operating system at the local site. The remote mirroring unit could also run a program to shut down the operating system it employed under mirroring, and any related programs, and then restart under a specified host operating system from a separate internal disk or a separate partition. The remote mirroring unit could also be enhanced to run continuously as a secondary server rather than being normally dedicated to data mirroring only. However, doing so could severely reduce mirroring performance, as well as increase the risk that mirroring fails outright. If the remote mirroring unit has essentially the same software as the local mirroring unit 204, then the remote mirroring unit can perform as a local mirroring unit 204. For instance, when mirroring from site A to site B to site C, a mirroring unit at site B is a remote mirroring unit with respect to site A and a local mirroring unit with respect to site C. The remote mirroring unit can also perform as a local mirroring unit 204 in a recovery from the remote location back to the source. That is, when mirroring from site A to site B, the mirroring unit at site A is local and the mirroring unit at site B is remote, but in recovering data from site B back to site A. the mirroring unit at site A is remote and the mirroring unit at site B is local. Finally, some inventive systems can accommodate multiple user sessions; a user session being a mirrored data relay or storage session. Multiple combinations and instances of the above scenarios can thus occur concurrently or separately in the appropriate environment. It may be necessary to include more processors, disks, memory, and so on to accomplish particular combinations. These various tools and techniques can also be used in a one-to-many mirroring system or a many-to-one mirroring system according to the invention. Likewise, discussion of tools and techniques which refer to packets, refer to an IP, Ethernet, token ring, or other packetized data environment, and it will be understood that other supported environments may write in data streams instead of using packets. The method steps discussed above and elsewhere herein may be performed in various orders and/or concurrently, except in those cases in which the results of one step are required as input to another step. For instance, connecting steps 1304, 1306. and 1308 may be done in various orders and/or concurrently, but many operations in the testing step 1310 will assume that some or all of the indicated connections are present, at least nominally. Transmitting data to a local mirroring unit during step 1312 necessarily precedes transmitting that data over the journey link 206 or to a local mirror 230 during step 1314. On the other hand, transmission step 1316 may be performed by performing transmission step 1314 (or by using a private dedicated link 206) if the transmission is to a serverless remote mirroring unit. Steps may also be omitted unless called for in issued claims, regard- less of whether they are expressly described as optional in this Detailed Description. Steps may also be repeated, combined, or named differently. Configured Storage Media, Signals
Articles of manufacture within the scope of the present invention include a computer-readable storage medium in combination with the specific physical configuration of a substrate of the computer-readable storage medium. The substrate configuration represents data and instructions which cause the computers to operate in a specific and predefined manner as described herein. Suitable storage devices include floppy disks, hard disks, tape, CD-ROMs, RAM, flash memory, and other media readable by one or more of the computers. Each such medium tangibly embodies a program, functions, and/or instructions that are executable by the machines to perform flexible mirroring method steps substantially as described herein, including without limitation methods which perform some or all of the steps illustrated in Figure 13 and methods for installing and/or using the systems illustrated in Figures 2 through 12. The invention also provides novel signals which are used in or by such programs. The signals may be embodied in "wires", RAM, disk, or other storage media or data carriers. Additional Information
To further assist people and enterprises in understanding and properly practicing the invention, additional insights and details are provided below. These comments are given with the continued assumption that discussions of any one of the embodiment types (methods, systems, configured storage media) also apply to the other embodiment types unless clearly indicated otherwise.
Specific Examples of the Invention's Improvements
Many other solutions to the problem of data protection (tape backup, local clustering, replication, shadowing, remote mainframe channel extension, and so on) are in some way directly connected to and dependent upon the host 200 operating system. This dependence creates problems to the customer, which may be avoided by using the present invention. For instance, the reliance on dependent dedicated software can cause compatibility problems and bugs when that software does not fully work with the current host operating system or with an upgrade to the operating system. Software solutions that depend on dedicated host mirroring software can also present a performance problem because they impose additional work on the host. Dependent software solutions can also be a point of instability. As disk volumes become larger and software and operating systems become more complicated these problems increase for approaches that require dependent
JZ software. In addition, if the host 200 operating system freezes up then solutions which depend on that operating system also stop working.
By contrast, in at least some embodiments the present invention does not have any software that loads on the host computer (e.g., local server 200), thereby reducing or avoiding the aforementioned problems. If the host operating system freezes the mirroring units continue to operate and mirrored data is available because the mirroring units are running their own operating system(s). Unlike solutions that need to be substantially modified at their core as disk volumes increase and software gets more complicated, the invention scales readily. If a faster processor comes out one simply uses this processor in the mirroring units as desired. If the disk size is bigger, one puts bigger disks in mirroring units. If the data change rate exceeds current ability to write to disk, one uses a caching controller and adds memory to the system. Some other solutions require cooperation from the operating system manufacturer in order to integrate and operate properly without bugs. Because all operating systems will support SCSI and fibre channel (for instance) for the foreseeable future, such cooperation is not required for installation and use of the invention. When other solutions fail they can take the host 200 with them, because of the close interactions outlined above. Because the invention can operate independently of the host 200, if it fails it need not seriously affect the host computer. Conventional disk mirroring was originally designed for local fault tolerance. Two disks would be written to in parallel, and if one disk failed the computer would continue to operate. The disk that failed would be dismounted from the operating system in the background. The operating system and computer would often continue to run without missing a beat. Because the inventive mirroring unit can look like a SCSI disk and be mounted as a mirrored disk, it provides a similar advantage. If a mirroring unit dies, it simply gets dismounted. For instance, if the operating system or other software on the mirroring unit fails then the mirroring unit stops emulating a disk. As a result, the operating system on the host 200 no longer recognizes the mirroring unit. In response, the operating system on the host 200 simply dismounts the mirroring unit 204 and continues to run.
At least some previous mirroring system implementations used a single disk IDE buffer. Even with spoofing, such a smart buffer has not been able to keep up with high speed SCSI RAID units with hardware striping. The most critical data that was being transmitted to the remote location was trusted to a single disk with no fault tolerance at the smart buffer level. With the present invention, by contrast, the local and remote mirroring units can both mirror a single disk buffer for fault tolerance, and they can perform hardware RAID striping across multiple disks. This provides an ability to keep up with new high speed storage subsystems on servers, and better fault tolerance. In the event of an individual disk failure in the server 200 volume or a mirroring unit disk 210, 310 this also reduces the risk of losing buffered data.
The limited data input capabilities of the prior approaches made it very difficult to address new technologies that are gaining market acceptance. For example, under at least some prior approaches there is no Storage Access Network ("SAN") or Network Attached Storage ("NAS") support. Requiring a standard remote server such as the server 300 made it hard or impossible to provide backup and mirroring for the SAN and NAS disk subsystems that are becoming more prevalent. However, all of these subsystems can perform a local mirror through Ethernet, fibre channel, and/or SCSI. The inventive mirroring units can accept multiple input types, including SCSI, Ethernet, and fibre channel inputs.
The invention also provides support for larger storage subsystems. Many earlier fault tolerance solutions were designed for an environment in which a six Gigabyte storage volume was considered very large. With storage costs falling, disk subsystems are increasing in size at a very rapid rate. It is now common for servers to have volumes of 100 Gigabytes. The invention accommodates these larger volumes in part by handling synchronization for the host server 200 in the background, namely, on the mirroring unit. Offloading this task from the host server to the mirroring unit(s) allows a true mirror of the main host server 200 without a large performance decrease. By contrast, alternative "clustering" and/or mirroring solutions that require a local server to handle the synchronization required for a mirror may either severely slow or crash that primary server. At least some previous implementations of re-mirroring have required the local server 200 to intervene if the local buffer could not support the entire local volume, although implementations have done much to avoid re-synchronization of mirrored disks (re- mirroring), over the telecommunications link. Re-mirroring slowed the main/primary/host server 200 to a standstill, and could take several days. So the re-mirroring has generally been performed only on weekends when the network could run slower, as there would be fewer users. But as disk subsystems are getting larger this is no longer acceptable. The invention supports nonvolatile storage, not only at the remote location but also in the local mirroring unit 204, which is large enough to hold the complete volume that is being mirrored to the remote location. This allows the local mirroring unit 204 to pre- acknowledge the complete local disk storage volume into a localized smart buffer and perform the tasks related to a re-mirror in the "background" from the server 200 perspective. In at least some prior approaches, the limitation of the maximum rate of a Tl output, from either the local or remote location, slowed a re-mirror even if a frame relay network, ATM, and/or VSAT network was available. By contrast, the invention flexibly allows a larger I/O pipe capability, which can improve performance because re-mirrors will be quicker and data deployment will be more efficient. If mirrored data being stored remotely becomes unavailable, the data stored at the unavailable site can be moved at high speed to another facility using a high-speed private data network. These data networks usually support bandwidths up to an OC48 rate (2.488-Gigabits per second). An example of this might be a customer that normally mirrors their data to Chicago and now needs to use the facility in New York for recovery. This type of need is much more common than originally realized.
The original Off- Site Server product failed to provide an open Application Programmer Interface ("API"). It was written instead solely to closed proprietary hardware (MiraLink's) and closed proprietary software (Vinca's). If a corporate customer had needs that exceeded the scope of that product there was generally no easy way to make custom modifications or adjustments. By contrast, the present invention permits an open API so that adjustments can be made from user-space processes to address specific customers and/or emerging markets. In particular, and without limitation, the present invention preferably has an API which provides one or more calls to reconfigure a mirroring unit without interrupting the server 200, and also provides a call to reboot the mirroring unit without interrupting the server 200. Configuration Data System configuration data is preferably distributed, so that if one of the mirroring units loses configuration data, that configuration data can be recovered from one of the unit's peers. Basic configuration data such as network information is preferably stored in nonvolatile storage (e.g., on disk, or in battery-backed semiconductor memory), so that even if the configuration data on the disk is lost, the configuration data can still be restored from the peer mirroring unit.
A world wide web interface preferably provides, at a minimum, the following configuration options or their equivalents: IP address (remote/local); gateway (remote/local); net mask (remote/local); administrator password (shared); buffer size (local); buffer high water mark (buffer filled beyond acceptable limits); volume size (configurable up to a factory-configured hard max); SCSI target Logical Unit Number ("LUN"); and SNMP configuration (remote/local).
The SNMP configuration itself preferably contains the following: add/delete SNMP monitoring hosts (remote/local); event polling intervals; buffer filled past acceptable limits; network connection failure; buffer full; remote out of synch; add/delete e-mail recipient.
The web interface preferably provides, at a minimum, the following status information: blocks in buffer; blocks sent; blocks received; mirroring unit version; mirroring unit serial number; volume size; whether this unit is remote or local. The web interface preferably provides an unmount remote utility. The web interface preferably also provides a log dump report. SNMP and SMTP traps are generally used for the following events: buffer filled past acceptable limits; buffer full; network connection failure; remote out of synch. The administrative tools may provide notifications by e-mail, by paging, or other means. Notification may be real-time and/or in combination with automated logs or automatically generated reports. Notifications may be sent to system administrators and/or vendors. In embodiments which run a web server/mail server package as an interface many of the characteristics of a web server are available. For instance, users can access and mange the mirroring unit either locally or remotely. Depending on permissions, users can access the mirroring unit internally to the company and/or from anywhere in the world. A mirroring unit can notify users (and mirroring unit vendors) of problems or significant events on the mirroring unit via e-mail as well as through SNMP. One can write custom scripts for this e-mail so that different users or groups of users are notified. Report outputs are not necessarily static. If a customer requires custom reports for their management instead of copying the required information each month and writing the report over and over again, the customer or a certified developer can use HTML, JAVA, and/or other familiar tools and techniques to have the mirroring unit generate and e-mail the report as needed in the desired format. Basic Hardware
In general, a system according to the invention includes basic hardware such as a standard Pentium II, Pentium III, AMD K6-3 or AMD K7 class PC-compatible computer (marks of their respective owners). In various configurations the machines preferably have at least 64, 128, or 256 megabytes of RAM, and a rack-mounted case. They also preferably contain one 100Mb Ethernet card, FDDI card, or the like. For disk interfaces, the machines preferably have a QLogic SCSI card for disk emulation and an Adaptec 2940UW adapter for buffer and mirror control, or a FreeBSD supported DPT brand RAID card. Caching may be used, including RAID or SCSI controller caching, caching in volatile RAM in the mirroring unit(s), caching in nonvolatile RAM (e.g., static RAM or battery-back RAM) in the mirroring unit(s), and otherwise. Caching tools and techniques familiar to those in the art may be readily adapted for use according to the present invention.
In some embodiments, if N is the size of the volume to be mirrored, then local mirroring units 204 which include a local mirror 230 have storage capacity of at least N for that local mirror. In some embodiments, a disk system, which serves as the local buffer 210 (with or without a local mirror) has a capacity of at least six-fifths N, that is 1.2 times N.
The remote mirroring unit has at least one disk system, for the remote mirror, of size at least N. In all scenarios, the local mirroring unit buffer 210 may need to be equivalent in data capacity to its remote mirroring unit, including buffers and hot-swappable RAID subsystems, to permit a local re-mirror. Test Suite
Tests used to gauge performance of a system according to the invention preferably include analytical tests which can be used to gauge relative performance and Boolean (pass/fail) tests that cover critical functional specification conformance criteria. A Boolean test is passed if the specified answer to all questions are correctly matched by test results. The Boolean tests can be used to determine the suitability of deliverables.
Tests should preferably be passed both in a local network configuration (where the journey link 206 is within a single local area network) and in a local and remote configuration (where the local mirroring unit 204 and the remote mirroring unit are geographically distant from each other). For instance, a remote network configuration could consist of two sites connected together with a Tl link 206 or an equivalent amount of public Internet bandwidth as the journey link 206.
Analytical tests preferably use a standard disk hardware test suite such as Bonie (for UNIX) or PCTools (for Windows NT and Novell clients). The tests compare performance of a native disk drive (for which the model, size, and characteristics are noted) with the performance of a flexible mirroring unit 204. The performance outputs are noted for later reference.
The following questions are preferably asked, and any necessary correction is made until they are answered as indicated. Is the mirroring unit 204 recognized by host 200 operating system as a disk with the correct configured size? (Yes)
Can data be read and written to the mirroring unit 204 without loss? (Yes)
Can the host system 200 perform arbitrary file operations with data on the mirroring unit 204 for forty-eight hours without error? (Yes)
Can a local mirroring unit 204 configured with a 100 megabyte host volume and a remote network configuration successfully mirror data to a remote mirroring unit with a data rate of at least 300 megabytes/hour and preferably higher if FDDI or other support is present? (Yes) Note that the 300 megabytes/hour rate is under the maximum carrying capacity for a Tl connection by about 50%; Tl capacity is about 617 megabytes/hour.
Can the local mirroring unit 204 be rebooted fully without the attached host system 200 failing to operate in a normal manner, namely, does the host 200 continue fulfilling its intended purpose without significant performance degradation? (Yes)
When the local mirroring unit 204 comes back on line does it automatically start to transfer across the network or other journey link 206 (e.g., using TCP sockets) data that was left on the local mirroring unit 204 queue, sending that data to the remote mirroring unit without loss of data? (Yes) Note that this should be confirmed by mounting the remote mirroring unit's drive on the host system 200 before and after rebooting the local mirroring unit 204 while it is attached to the host system 200. The remote mirror should be mountable after such an event without significant need for file system repair. Data should not be lost and should make sense to the application programs that created it. After mounting the remote mirror on the local host system 200 physically, is the host system 200 able to mount the mirror and are application programs on the host 200 and its clients able to use data on the mirror successfully? (Yes) In response to an input of improper information such as a wrong remote IP address, or an invalid SCSI ID (less than 0 or greater than 15), does the mirroring system crash or hang? (No) Can the user correct the information, re-initialize the software and have it perform normally without needing a mirroring unit reboot? (Yes) Does all software display a correct version number and copyright statement? (Yes) In response to a disconnection of the network cable 206 for a period of 30 minutes and preferably for longer periods while a mirroring operation or other disk I/O intensive operation is being conducted by the host system 200, does the local mirroring unit 204 continue to work? (Yes) Is it recognized by the host operating system as a disk with the correct configured size? (Yes) Can data be read and written to the local mirroring unit 204 without loss? (Yes)
After an initial mirror has been established, disconnect the network cable for twenty- four hours and perform periodic re-runs of the tests. Is the local mirroring unit 204 still recognized by the host 200 operating system as a disk with the correct configured size? (Yes) Can data still be read and written to the local mirroring unit 204 without loss? (Yes) Likewise, after forcing the host system 200 to overflow the buffer 210 (e.g., by re- mirroring multiple times), verify that the local mirroring unit 204 still operates properly to the extent possible. Is the local mirroring unit 204 still recognized by the host 200 operating system as a disk with the correct configured size? (Yes) Can data still be read and written to the local mirroring unit 204 without loss? (Yes) Can a user stop the en-queuing process and restart it without requiring a local mirroring unit 204 reboot? (Yes) Can a user can stop the de-queuing process and restart it without requiring a local mirroring unit 204 reboot? (Yes) Can a user selectively flush specified portion(s) of the buffer, e.g., flush an aborted mirror without flushing a full mirror if the data is at least partially remirrored more than once? (Yes)
While a mirroring operation or other disk I/O intensive operation is being conducted by the host system 200, disconnect the network cable or other journey link 206 for a period of thirty minutes. Can the local mirroring unit 204 start sending data from the queue to the remote mirroring unit after re-establishing a physical network connection? (Yes) Are valid statistics available from the local mirroring unit 204 as to the status of the buffer (e.g., full or not full, number of blocks in the buffer, and the number of blocks transferred from the buffer and received on the remote side)? (Yes)
Unplug the local mirroring unit 204 UPS, shut down the host system 200, and wait for the power to fail on the local mirroring unit 204. Restore power to the local mirroring unit 204 and then to the host system 200. Does the host system operate properly? (Yes) Can the local mirroring unit 204 be rebooted fully without the attached host system 200 failing to operate in a normal manner? (Yes) When the local mirroring unit 204 comes back on line does it automatically start to transfer across the network or other journey link 206 data left in the local mirroring unit 204 buffer 210, without loss of data? (Yes) Note that the last two of these remote mirror mounting tests should be performed both before and after this simulated power failure. Do they pass? (Yes) In addition, do all previous tests succeed with a host volume size of 200 gigabytes? (Yes)
Can the remote mirroring unit be disabled and the remote mirror mounted by a standby server running the same operating system as the primary host system 200? (Yes) Will the remote host then operate normally and without adverse impact on its performance? (Yes) Note that the operation of the previous two tests is supported by having the remote backup host attached on the same SCSI chain as the remote mirroring unit and its remote mirror disk subsystem 312 or 614. Summary The present invention provides tools and techniques for data mirroring, locally and/or remotely. In particular, a computer system for remote mirroring of data according to the invention includes one or more flexible mirroring characteristics. Systems for local mirroring (e.g., where the source and destination are less than ten miles apart) may also have such flexible mirroring characteristics. For instance, the system may be characterized by having a serverless destination.
That is, one embodiment of the system mirrors data from the local server 200 as a source through the local mirroring unit 204 to the remote mirroring unit 208, 408, 508, 608, or 708 as a destination, without requiring the use of a remote server attached to the remote mirroring unit. The system may also be characterized as non-invasive, in that no software designed specifically for remote data mirroring need be installed on the local server 200. Similarly, no such software need be installed on the secondary server 300 in systems that include a server 300. Instead, each mirroring unit runs an operating system and one or more remote data mirroring application programs (including threads, processes, tasks, etc.). For instance, the mirroring units rather than the server(s) buffer data to be mirrored, create and monitor connections over the journey link 206, and transmit/receive mirrored data over the journey link 206, thereby relieving the server(s) of those tasks. Likewise, the system may be characterized by disk emulation, such that the system mirrors data from the local server 200 to the local mirroring unit 204 through a standard storage subsystem bus. Suitable standard storage subsystem buses include SCSI, fibre channel, USB, and other nonproprietary buses. Such buses are also referred to herein as "connections" to the local mirroring unit 204.
The system could be characterized by a TCP journey line characteristic and/or by an Ethernet journey line characteristic. In one case, for instance, the system mirrors data from the local server 200 through the local mirroring unit 204, which operates as a TCP client over the journey line 206; the remote mirroring unit 208, 308, 408, 508, 608, or 708 operates as a TCP server. More generally, a journey line characteristic indicates that the high-bandwidth low-latency requirements imposed by SCSI, original Off-SiteServer serial connections. SAN connections, and the like are not present in a connection 206 between a local mirroring unit 204 and a remote mirroring unit.
The system might also be characterized by a multiplicity characteristic. That is, the system may provide many-to-one mirroring from two or more local (primary) servers 200 to a single remote mirroring unit 208, 308, 408, 508, 608, or 708. The data mirroring system of remote mirroring unit nonvolatile storage may then include one disk partition for each primary network server 200 with each disk partition holding mirrored data for the respective server 200, one external hard disk 614 for each server 200. one RAID unit 312 for each server 200, or some combination thereof. The various primary (local) servers 200 may all use the same operating systems or they may use some combination of different operating systems. In some cases the destination nonvolatile storage is sufficiently large to hold the combined current nonvolatile data of all of the primary servers 200. As another multiplicity characteristic, the system may provide one-to-many mirroring from a given local (primary) server 200 to two or more remote mirroring units 208, 308, 408, 508, 608, or 708.
The invention also provides methods, including methods for installing flexible mirroring units, methods for using such units, and methods for doing both. For example, a method for facilitating flexible data mirroring includes at least two steps from the group 1300 of installing steps. Another method for flexible data mirroring includes one or more transmitting steps 1302.
One of the installing steps involves connecting 1304 the local server 200 to the local mirroring unit 204 with the standard storage subsystem bus 202, thereby permitting the local mirroring unit 204 to emulate a disk subsystem in communications over the link 202. A step 1306 involves connecting the local mirroring unit 204 to the journey link 206 for transmission of data by at least one of an Ethernet connection and a TCP connection. A step 1308 involves connecting the remote mirroring unit 208, 308, 408, 508, 608. or 708 to the journey link 206 for reception of data transmitted by at least one of an Ethernet connection and a TCP connection. A testing step 1310 tests at least one mirroring unit 204, 208, 308, 408, 508, 608, or 708 after at least partial completion of at least one of the aforesaid connecting steps. One of the transmitting steps 1302 is a step 1312 which transmits data from the local server 200 to the local mirroring unit 204 over the standard storage subsystem bus 202 while the local mirroring unit 204 emulates a disk subsystem. A step 1314 transmits data from the local mirroring unit 204 over the journey link 206 to the remote mirroring unit 208, 308, 408. 508, 608, or 708. A step 1316 (which may be performed with the same data transmission as step 1314) transmits data from the local mirroring unit 204 over the journey link 206 to the remote mirroring unit 208, 308, 408, 508. 608, or 708 when the remote mirroring unit is serverless, that is, when it is not attached to a secondary server 300.
Particular embodiments (methods, configured storage media, and systems) of the present invention are expressly illustrated and described herein. To avoid unnecessary repetition, concepts and details applicable to one embodiment are not always stated expressly with regard to other embodiments. Unless otherwise expressly indicted, however, the descriptions herein of particular embodiments of the present invention extend to other embodiments. For instance, discussions of the invention's systems also pertain to its methods and vice versa, and the descriptions of inventive methods also pertain to corresponding configured storage media and vice versa.
As used herein, terms such as "a" and "the" and item designations such as "mirroring unit" are generally inclusive of one or more of the indicated item. In particular, in the claims a reference to an item means at least one such item is required, unless otherwise indicated.
The invention may be embodied in other specific forms without departing from its essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. Headings are for convenience only. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.
What is claimed and desired to be secured by patent is:

Claims

1. In a computer system for mirroring data, an improvement comprising at least two flexible mirroring characteristics from a group of such characteristics, the system not necessarily having every characteristic in the group, wherein the group consists of: a serverless destination characteristic by which the system mirrors data to a remote mirroring unit without requiring the use of a remote server attached to the remote mirroring unit; a non-invasive characteristic by which the system mirrors data from a host through a local mirroring unit toward a remote mirroring unit without requiring on the host any software that is designed specifically for remote data mirroring; a disk emulation characteristic by which the system mirrors data from a host to a local mirroring unit through a standard storage subsystem bus; a TCP journey line characteristic by which the system mirrors data from a host through a local mirroring unit which operates as a TCP client over a journey line to a remote mirroring unit which operates as a TCP server; and a multiplicity characteristic by which the system provides at least one of many-to-one mirroring and one-to-many mirroring.
2. The improvement of claim 1, wherein the improvement comprises at least three flexible mirroring characteristics from the group.
3. A computer system comprising the improvement of claim 2 in combination with a local server linked by a local link to at least one local mirroring unit, which is linked in turn by a journey link to a remote mirroring unit.
4. The improvement of claim 1, wherein the improvement comprises at least four flexible mirroring characteristics from the group.
5. A computer system comprising the improvement of claim 4 in combination with a local server linked by a local link to at least one local mirroring unit, which is linked in turn by a journey link to a remote mirroring unit.
6. The improvement of claim 1. wherein the improvement comprises all flexible mirroring characteristics of the group.
7. The improvement of claim 1 , wherein the improvement characteristics comprise the serverless destination characteristic.
8. The improvement of claim 7, wherein the improvement characteristics further comprise the disk emulation characteristic.
9. The improvement of claim 7. wherein the improvement characteristics further comprise the non-invasive characteristic.
10. The improvement of claim 1 , wherein the improvement characteristics comprise the disk emulation characteristic.
1 1. The improvement of claim 10, wherein the standard storage subsystem bus includes a SCSI connection.
12. The improvement of claim 10, wherein the standard storage subsystem bus includes a fibre channel connection.
13. The improvement of claim 10, wherein the standard storage subsystem bus includes a USB connection.
14. A computer system comprising the improvement of claim 10 in combination with a local server linked by a local link to at least one local mirroring unit, which is linked in turn by a journey link to a remote mirroring unit.
15. The improvement of claim 1 , wherein the improvement characteristics comprise the TCP journey line characteristic.
16. The improvement of claim 15, wherein the improvement characteristics further comprise the serverless destination characteristic.
17. The improvement of claim 15, wherein the improvement characteristics further comprise the non-invasive characteristic.
18. The improvement of claim 15, wherein the improvement characteristics further comprise the disk emulation characteristic.
19. The improvement of claim 18, wherein the improvement characteristics further comprise the serverless destination characteristic.
20. A computer system comprising the improvement of claim 15 in combination with a local server linked by a local link to at least one local mirroring unit, which is linked in turn by a journey link to a remote mirroring unit.
21. The improvement of claim 1. wherein the improvement characteristics comprise the non-invasive characteristic.
22. The improvement of claim 21 , wherein the improvement characteristics further comprise the disk emulation characteristic.
23. A computer system comprising the improvement of claim 21 in combination with a local server linked by a local link to at least one local mirroring unit, which is linked in turn by a journey link to a remote mirroring unit.
24. A computer system comprising the improvement of claim 1 in combination with a local server linked by a local link to at least one local mirroring unit.
25. The system of claim 24, wherein the system further comprises at least one remote mirroring unit.
26. The system of claim 25. wherein the system further comprises at least one remote server connected to the remote mirroring unit.
27. A data mirroring system which is characterized by at least a disk emulation flexible mirroring characteristic and a many-to-one multiplicity flexible mirroring characteristic, the system for mirroring data from primary network servers having nonvolatile data stores, over journey links, to a remote nonvolatile data storage area, wherein the system comprises: a source including at least two primary network servers, each primary server being linked through a respective local link to a respective local mirroring unit for sending mirrored data from the primary server to the local mirroring unit, each of the local mirroring units having a spoof packet generator and a nonvolatile data buffer for mirrored data, the local link including a standard storage subsystem bus and the local mirroring unit emulating a disk subsystem in communications over that bus; and a destination including a remote mirroring unit having a destination nonvolatile storage for mirrored data received from the local mirroring units over the journey links.
28. The data mirroring system of claim 27. wherein the destination nonvolatile storage comprises one disk partition for each primary network server, each disk partition holding mirrored data for the respective primary network server.
29. The data mirroring system of claim 27, wherein the destination nonvolatile storage comprises one external hard disk for each primary network server, each external hard disk holding mirrored data for the respective primary network server.
30. The data mirroring system of claim 29, wherein each external hard disk is bootable.
31. The data mirroring system of claim 27. wherein the destination nonvolatile storage comprises one RAID unit for each primary network server, each RAID unit holding mirrored data for the respective primary network server.
32. The data mirroring system of claim 31 , wherein each RAID unit is hot- swappable.
33. The data mirroring system of claim 27, wherein the primary servers comprise a first primary server and a second primary server, and the first primary server has a different operating system than the second primary server.
34. The data mirroring system of claim 27, wherein the destination nonvolatile storage is sufficiently large to hold the combined current nonvolatile data of all of the primary servers.
35. The data mirroring system of claim 27, wherein the standard storage subsystem bus comprises a SCSI bus.
36. The data mirroring system of claim 27, wherein the standard storage subsystem bus comprises a fibre channel connection.
37. The data mirroring system of claim 27, wherein the standard storage subsystem bus comprises a Universal Serial Bus connection.
38. The data mirroring system of claim 27, wherein the journey link comprises an Ethernet connection.
39. The data mirroring system of claim 27, wherein the journey link comprises a TCP connection.
40. The data mirroring system of claim 27, wherein the destination does not include a server.
41. The data mirroring system of claim 27, wherein at least one of the primary network servers does not include software designed specifically for remote data mirroring.
42. The data mirroring system of claim 27, wherein the remote mirroring unit is physically separated from at least one of the primary servers by a distance of at least ten miles.
43. The data mirroring system of claim 42, wherein the remote mirroring unit is physically separated from at least one of the primary servers by a distance of at least one hundred miles.
44. The data mirroring system of claim 27. wherein at least two of the primary servers are physically separated from each other by a distance of at least ten miles.
45. The data mirroring system of claim 27, wherein at least two of the primary servers are physically separated from each other by a distance of at least one hundred miles.
46. A data mirroring system which is characterized by at least a disk emulation flexible mirroring characteristic and a journey line flexible mirroring characteristic, the system for mirroring data from a host having a nonvolatile data store, over at least one journey link, to at least one remote nonvolatile data storage area, wherein the system comprises: a source including a host linked through at least one local link for sending mirrored data to at least one local mirroring unit, each local mirroring unit generating pre-acknowledgements indicating storage of mirrored data in a nonvolatile data buffer of the local mirroring unit, each local link including a standard storage subsystem bus and each local mirroring unit emulating a disk subsystem in communications over that bus: and a destination including at least one remote mirroring unit having a destination nonvolatile storage for mirrored data received from a local mirroring unit over a journey link which operates with relatively low bandwidth and relatively high latency in comparison to a storage area network link.
47. The data mirroring system of claim 46, wherein the destination nonvolatile storage comprises an external hard disk for holding mirrored data for the host.
48. The data mirroring system of claim 47, wherein the external hard disk is bootable.
49. The data mirroring system of claim 46, wherein the destination nonvolatile storage comprises a RAID unit for holding mirrored data for the host.
50. The data mirroring system of claim 49. wherein the RAID unit is hot- swappable.
51. The data mirroring system of claim 49, wherein the RAID unit includes internal striping software.
52. The data mirroring system of claim 49, wherein the RAID unit includes a hardware controller for striping.
53. The data mirroring system of claim 46. wherein the destination nonvolatile storage is sufficiently large to hold the current nonvolatile data of the host.
54. The data mirroring system of claim 46, wherein the host does not include software designed specifically for remote data mirroring.
55. The data mirroring system of claim 46, wherein the host does not include software for buffering mirrored data.
56. The data mirroring system of claim 46. wherein the host does not include software for transmitting mirrored data over the journey link.
57. The data mirroring system of claim 46, wherein the standard storage subsystem bus comprises a SCSI bus.
58. The data mirroring system of claim 46, wherein the journey link comprises a packet transmission connection.
59. The data mirroring system of claim 46, wherein the journey link comprises a data stream transmission connection.
60. The data mirroring system of claim 46, wherein the local mirroring unit emulates a disk subsystem in a communication which requests that disk formatting be performed.
61. The data mirroring system of claim 46, wherein the local mirroring unit emulates a disk subsystem in a communication which requests that disk partitioning be performed.
62. The data mirroring system of claim 46, wherein the local mirroring unit emulates a disk subsystem in a communication which requests that a disk integrity check be performed.
63. The data mirroring system of claim 46, wherein the journey link comprises an Ethernet connection.
64. The data mirroring system of claim 46, wherein the journey link comprises a
TCP connection.
65. The data mirroring system of claim 64. wherein the standard storage subsystem bus comprises a SCSI bus.
66. The data mirroring system of claim 64, wherein the standard storage subsystem bus comprises a fibre channel connection.
67. The data mirroring system of claim 64, wherein the standard storage subsystem bus comprises a Universal Serial Bus connection.
68. The data mirroring system of claim 46. wherein the destination does not include a server.
69. The data mirroring system of claim 46, wherein the remote mirroring unit is physically separated from the host by a distance of at least ten miles.
70. The data mirroring system of claim 69, wherein the remote mirroring unit is physically separated from the host by a distance of at least one hundred miles.
71. The data mirroring system of claim 46, wherein the source comprises at least two hosts linked to at least two respective local mirroring units, and the destination comprises at least two remote mirroring units, each having a destination nonvolatile storage for mirrored data received from a respective local mirroring unit over a respective journey link.
72. The data mirroring system of claim 71 , wherein at least two of the hosts are physically separated from each other by a distance of at least ten miles.
73. The data mirroring system of claim 72, wherein at least two of the hosts are physically separated from each other by a distance of at least one hundred miles.
74. The data mirroring system of claim 46, further comprising a means for reconstructing a disk volume as it existed at a specified previous time.
75. The data mirroring system of claim 46, further comprising a full local mirrored volume.
76. A data mirroring system comprising at least one host and a means for mirroring data, the means for mirroring data including at least one mirroring unit, the system being further characterized by the presence of at least one flexible mirroring characteristic in the system.
77. The data mirroring system of claim 76, wherein the system is characterized by the presence of at least two flexible mirroring characteristics in the system.
78. The data mirroring system of claim 76, wherein the system comprises at least one local server and at least two mirroring units, and the system is characterized by the presence of at least three flexible mirroring characteristics in the system.
79. The data mirroring system of claim 76, wherein the system comprises at least one local server, at least one remote server, and at least two mirroring units, and the system is characterized by the presence of at least three flexible mirroring characteristics in the system.
80. The data mirroring system of claim 76, wherein the system comprises at least one local server and at least two mirroring units, and the system is characterized by the presence of at least four flexible mirroring characteristics in the system.
81. The data mirroring system of claim 76, wherein the host comprises a storage area network.
82. The data mirroring system of claim 76, wherein the host comprises a network attached storage subsystem.
83. The data mirroring system of claim 76, wherein the host comprises a computer which is not a server.
84. The data mirroring system of claim 76, wherein the system comprises a full local mirror.
85. The data mirroring system of claim 84, wherein the local mirroring unit does not generate pre-acknowledgments.
86. The data mirroring system of claim 84, wherein the full local mirror receives data by way of a data fork done below a disk emulation layer of at least one mirroring unit.
87. The data mirroring system of claim 76, wherein the system comprises a circular buffer with head and tail pointers.
88. The data mirroring system of claim 76, wherein the system comprises a dual host connection between a remote mirroring unit, a remote server, and a remote disk subsystem.
89. A method for facilitating flexible data mirroring, the method comprising at least two steps from a group of installing steps, the method not necessarily including every step in the group of installing steps, wherein the group of installing steps consists of the following steps: connecting a host to a local mirroring unit with a standard storage subsystem bus permitting the local mirroring unit to emulate a disk subsystem in communications over that bus; connecting a local mirroring unit to a journey link for transmission of data; connecting a remote mirroring unit to a journey link for reception of data; and testing at least one mirroring unit after at least partial completion of at least one of the aforesaid connecting steps.
90. The method of claim 89, wherein the method comprises at least three steps from the group of installing steps.
91. The method of claim 90, wherein the method comprises all steps from the group of installing steps.
92. The method of claim 89, wherein the method further comprises at least one step from a group of transmitting steps, the method not necessarily including every step in the group of transmitting steps, wherein the group of transmitting steps consists of the following steps: transmitting data from a host to a local mirroring unit over a standard storage subsystem bus while the local mirroring unit emulates a disk subsystem; and transmitting data from a local mirroring unit over a journey link to a remote mirroring unit.
93. The method of claim 92, wherein the step of transmitting data from a host to a local mirroring unit transmits data over a SCSI bus while the local mirroring unit emulates a SCSI disk subsystem.
94. The method of claim 92, wherein the step of transmitting data from a host to a local mirroring unit transmits data over a fibre channel bus while the local mirroring unit emulates a fibre channel disk subsystem.
95. The method of claim 92. wherein the step of transmitting data from a host to a local mirroring unit transmits data over a Universal Serial Bus while the local mirroring unit emulates a USB disk subsystem.
96. The method of claim 92, wherein the step of transmitting data from a local mirroring unit over a journey link to a remote mirroring unit comprises transmitting packets over the journey link.
97. The method of claim 96, wherein the step of transmitting data from a local mirroring unit over a journey link to a remote mirroring unit comprises transmitting Ethernet packets over the journey link.
98. The method of claim 96, wherein the step of transmitting data from a local mirroring unit over a journey link to a remote mirroring unit comprises transmitting packets over the journey link using the TCP protocol.
99. The method of claim 92. wherein the step of transmitting data from a local mirroring unit over a journey link to a remote mirroring unit comprises transmitting a data stream over the journey link.
100. The method of claim 92. wherein the step of transmitting data from a local mirroring unit over a journey link to a remote mirroring unit comprises transmitting data over the journey link to a serverless remote mirroring unit.
101. A configured computer program storage medium which contains software to perform method steps for flexible data mirroring, the method comprising at least two steps from a group of transmitting steps, the method not necessarily including every step in the group of transmitting steps, wherein the group of transmitting steps consists of the following steps: transmitting data from a host to a local mirroring unit over a standard storage subsystem bus while the local mirroring unit emulates a disk subsystem; transmitting data from a local mirroring unit over a journey link to a remote mirroring unit using at least one of Ethernet packets and TCP packets; and transmitting data from a local mirroring unit over a journey link to a serverless remote mirroring unit.
102. The configured program storage medium of claim 101 , wherein the method comprises every step in the group of transmitting steps.
103. The configured program storage medium of claim 101 , wherein the method comprises transmitting data from a host to a local mirroring unit over a SCSI bus while the local mirroring unit emulates a SCSI disk subsystem.
104. The configured program storage medium of claim 101 , wherein the method comprises transmitting TCP packets over a journey link to a remote mirroring unit.
105. The configured program storage medium of claim 104, wherein the method further comprises transmitting TCP packets over the journey link to the remote mirroring unit while no remote server is connected to the remote mirroring unit.
106. The configured program storage medium of claim 101 , wherein the method comprises transmitting Ethernet packets over a journey link to a remote mirroring unit.
107. The configured program storage medium of claim 106, wherein the method further comprises transmitting Ethernet packets over the journey link to the remote mirroring unit while no remote server is connected to the remote mirroring unit.
108. The configured program storage medium of claim 101 , wherein the method further comprises transmitting data from the remote mirroring unit over another journey link to another remote mirroring unit.
109. The configured program storage medium of claim 101 , wherein the method further comprises performing a switchover from a normal mirroring state to a recovery state.
PCT/US2000/005443 1999-11-11 2000-03-03 Flexible remote data mirroring WO2001035244A1 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
CA 2427920 CA2427920A1 (en) 1999-11-11 2000-03-03 Flexible remote data mirroring
EP00923070A EP1240591A4 (en) 1999-11-11 2000-03-03 Flexible remote data mirroring
AU43252/00A AU4325200A (en) 1999-11-11 2000-03-03 Flexible remote data mirroring
HK03102020.7A HK1051417A1 (en) 1999-11-11 2003-03-18 Flexible remote data mirroring

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US43818499A 1999-11-11 1999-11-11
US09/438,184 1999-11-11

Publications (1)

Publication Number Publication Date
WO2001035244A1 true WO2001035244A1 (en) 2001-05-17

Family

ID=23739596

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2000/005443 WO2001035244A1 (en) 1999-11-11 2000-03-03 Flexible remote data mirroring

Country Status (7)

Country Link
US (5) US20050027892A1 (en)
EP (1) EP1240591A4 (en)
AU (1) AU4325200A (en)
CA (1) CA2427920A1 (en)
HK (1) HK1051417A1 (en)
TW (1) TW454120B (en)
WO (1) WO2001035244A1 (en)

Cited By (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2003005625A2 (en) * 2001-07-06 2003-01-16 Computer Associates Think, Inc. Systems and methods of information backup
WO2003025760A1 (en) * 2001-09-20 2003-03-27 Cebridge Pty. Ltd. Data protection and retrival
EP1298530A1 (en) * 2001-09-27 2003-04-02 Emc Corporation Remote data facility over an IP network
WO2003061222A1 (en) * 2001-12-21 2003-07-24 Cisco Technology, Inc. Methods and apparatus for implementing a high availability fibre channel switch
EP1402420A2 (en) * 2001-06-26 2004-03-31 EMC Corporation Mirroring network data to establish virtual storage area network
WO2004111849A2 (en) * 2003-06-17 2004-12-23 International Business Machines Corporation Method, system and article of manufacture for remote copying of data
EP1498816A1 (en) * 2003-07-18 2005-01-19 Network Appliance, Inc. System and method for reliable peer communication in a clustered storage system
WO2005048124A2 (en) * 2003-11-04 2005-05-26 Constant Data, Inc. Hybrid real-time data replication
WO2005071544A1 (en) * 2003-06-18 2005-08-04 International Business Machines Corporation Method. system. and program for andling a failover to a remote storage location
EP1562117A2 (en) 2004-02-03 2005-08-10 Hitachi, Ltd. Data restoration method using a log and corresponding storage subsystem
EP1647891A2 (en) * 2004-10-14 2006-04-19 Hitachi Ltd. Computer system performing remote copying via an intermediate site
US7171452B1 (en) 2002-10-31 2007-01-30 Network Appliance, Inc. System and method for monitoring cluster partner boot status over a cluster interconnect
US7293195B1 (en) 2004-01-29 2007-11-06 Network Appliance, Inc. System and method for coordinated bringup of a storage appliance in a cluster configuration
EP1914633A3 (en) * 2006-10-12 2008-08-06 Hitachi, Ltd. Storage apparatus
US7593996B2 (en) 2003-07-18 2009-09-22 Netapp, Inc. System and method for establishing a peer connection using reliable RDMA primitives
US7685358B1 (en) 2003-03-03 2010-03-23 Netapp, Inc. System and method for coordinating cluster state information
US7730153B1 (en) 2001-12-04 2010-06-01 Netapp, Inc. Efficient use of NVRAM during takeover in a node cluster
US7870354B2 (en) 2003-11-04 2011-01-11 Bakbone Software, Inc. Data replication from one-to-one or one-to-many heterogeneous devices
GB2473079A (en) * 2009-08-28 2011-03-02 Bizconline Ltd Remote backup using journals relayed via Stagers
EP2307975A1 (en) * 2008-07-02 2011-04-13 Hewlett-Packard Development Company, L.P. Verification of remote copies of data
US8688798B1 (en) 2009-04-03 2014-04-01 Netapp, Inc. System and method for a shared write address protocol over a remote direct memory access connection
US8762328B2 (en) 1999-11-11 2014-06-24 Intellectual Ventures Ii Llc Flexible remote data mirroring

Families Citing this family (157)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020055972A1 (en) * 2000-05-08 2002-05-09 Weinman Joseph Bernard Dynamic content distribution and data continuity architecture
US6671820B1 (en) * 2000-08-10 2003-12-30 Dell Products, L.P. System and method for the prevention of corruption of networked storage devices during backup data recovery
US6662282B2 (en) * 2001-04-17 2003-12-09 Hewlett-Packard Development Company, L.P. Unified data sets distributed over multiple I/O-device arrays
US7542993B2 (en) * 2001-05-10 2009-06-02 Equifax, Inc. Systems and methods for notifying a consumer of changes made to a credit report
US7139817B1 (en) * 2001-06-12 2006-11-21 Network Appliance, Inc. Managing configuration information for multiple devices
US20030105565A1 (en) * 2001-12-03 2003-06-05 Loda David C. Integrated internet portal and deployed product microserver management system
US7180865B1 (en) * 2002-06-28 2007-02-20 Bellsouth Intellectual Property Corporation System and method for analyzing frame relay communications
US7391739B1 (en) 2002-06-28 2008-06-24 At&T Delaware Intellectual Property, Inc. System and method for creating a frame relay port mirror
US7636320B1 (en) * 2002-06-28 2009-12-22 At&T Intellectual Property I, L.P. System and method for creating an asynchronous transfer mode port mirror
US7200148B1 (en) * 2002-06-28 2007-04-03 Bellsouth Intellectual Property Corp. System and method for analyzing asynchronous transfer mode communications
JP3974538B2 (en) * 2003-02-20 2007-09-12 株式会社日立製作所 Information processing system
JP4165747B2 (en) * 2003-03-20 2008-10-15 株式会社日立製作所 Storage system, control device, and control device program
JP2004355083A (en) * 2003-05-27 2004-12-16 Nec Corp Backup system and backup program
US7111136B2 (en) * 2003-06-26 2006-09-19 Hitachi, Ltd. Method and apparatus for backup and recovery system using storage based journaling
US20050022213A1 (en) 2003-07-25 2005-01-27 Hitachi, Ltd. Method and apparatus for synchronizing applications for data recovery using storage based journaling
US7398422B2 (en) * 2003-06-26 2008-07-08 Hitachi, Ltd. Method and apparatus for data recovery system using storage based journaling
US20050015416A1 (en) 2003-07-16 2005-01-20 Hitachi, Ltd. Method and apparatus for data recovery using storage based journaling
JP4124348B2 (en) * 2003-06-27 2008-07-23 株式会社日立製作所 Storage system
US7568025B2 (en) 2003-06-27 2009-07-28 Bank Of America Corporation System and method to monitor performance of different domains associated with a computer system or network
US7694086B1 (en) * 2003-06-30 2010-04-06 Symantec Operating Corporation Method and system for incremental backup of data volumes
US7529782B2 (en) * 2003-11-13 2009-05-05 Commvault Systems, Inc. System and method for performing a snapshot and for restoring data
US7917711B2 (en) 2003-11-14 2011-03-29 International Business Machines Corporation System, apparatus, and method for automatic copy function selection
US7613785B2 (en) * 2003-11-20 2009-11-03 International Business Machines Corporation Decreased response time for peer-to-peer remote copy write operation
US7342555B2 (en) * 2004-04-30 2008-03-11 Microsoft Corporation Detecting low fidelity sync data
US7707372B1 (en) * 2004-06-30 2010-04-27 Symantec Operating Corporation Updating a change track map based on a mirror recovery map
US7844444B1 (en) * 2004-11-23 2010-11-30 Sanblaze Technology, Inc. Fibre channel disk emulator system and method
US9195397B2 (en) 2005-04-20 2015-11-24 Axxana (Israel) Ltd. Disaster-proof data recovery
EP2328089B1 (en) * 2005-04-20 2014-07-09 Axxana (Israel) Ltd. Remote data mirroring system
US20070061379A1 (en) * 2005-09-09 2007-03-15 Frankie Wong Method and apparatus for sequencing transactions globally in a distributed database cluster
US8856091B2 (en) * 2005-09-09 2014-10-07 Open Invention Network, Llc Method and apparatus for sequencing transactions globally in distributed database cluster
US7428602B2 (en) * 2005-11-29 2008-09-23 International Business Machines Corporation Method for executing initialization code to configure connected devices and executing segments of configuration code from a failed segment
US7617262B2 (en) * 2005-12-19 2009-11-10 Commvault Systems, Inc. Systems and methods for monitoring application data in a data replication system
US7962709B2 (en) * 2005-12-19 2011-06-14 Commvault Systems, Inc. Network redirector systems and methods for performing data replication
US7606844B2 (en) 2005-12-19 2009-10-20 Commvault Systems, Inc. System and method for performing replication copy storage operations
US7651593B2 (en) 2005-12-19 2010-01-26 Commvault Systems, Inc. Systems and methods for performing data replication
EP1974296B8 (en) 2005-12-19 2016-09-21 Commvault Systems, Inc. Systems and methods for performing data replication
US7636743B2 (en) * 2005-12-19 2009-12-22 Commvault Systems, Inc. Pathname translation in a data replication system
US8655850B2 (en) * 2005-12-19 2014-02-18 Commvault Systems, Inc. Systems and methods for resynchronizing information
US8726242B2 (en) 2006-07-27 2014-05-13 Commvault Systems, Inc. Systems and methods for continuous data replication
JP4369471B2 (en) * 2006-12-27 2009-11-18 富士通株式会社 Mirroring program, mirroring method, information storage device
US8290808B2 (en) * 2007-03-09 2012-10-16 Commvault Systems, Inc. System and method for automating customer-validated statement of work for a data storage environment
WO2009002545A1 (en) * 2007-06-26 2008-12-31 Leader Technologies Incorporated System and method for providing alerting services
US8856633B2 (en) * 2007-10-03 2014-10-07 Qualcomm Incorporated Millimeter-wave communications for peripheral devices
EP2201456A4 (en) * 2007-10-08 2012-02-15 Axxana Israel Ltd Fast data recovery system
JP2009223442A (en) * 2008-03-13 2009-10-01 Hitachi Ltd Storage system
EP2286343A4 (en) * 2008-05-19 2012-02-15 Axxana Israel Ltd Resilient data storage in the presence of replication faults and rolling disasters
US8516121B1 (en) * 2008-06-30 2013-08-20 Symantec Corporation Method and apparatus for optimizing computer network usage to prevent congestion
US8291403B2 (en) * 2008-06-30 2012-10-16 Symantec Operating Corporation Install-unit upgrade using dynamic configuration data manipulation and merging
US8677018B2 (en) 2008-08-25 2014-03-18 Google Inc. Parallel, side-effect based DNS pre-caching
US9197486B2 (en) 2008-08-29 2015-11-24 Google Inc. Adaptive accelerated application startup
US9495382B2 (en) * 2008-12-10 2016-11-15 Commvault Systems, Inc. Systems and methods for performing discrete data replication
US8204859B2 (en) 2008-12-10 2012-06-19 Commvault Systems, Inc. Systems and methods for managing replicated database data
WO2010076755A2 (en) * 2009-01-05 2010-07-08 Axxana (Israel) Ltd Disaster-proof storage unit having transmission capabilities
US9225794B2 (en) 2009-03-31 2015-12-29 Google Inc. Adaptive DNS pre-resolution
US20100299447A1 (en) * 2009-05-25 2010-11-25 Nilesh Anant Salvi Data Replication
US9454325B2 (en) * 2009-11-04 2016-09-27 Broadcom Corporation Method and system for offline data access on computer systems
WO2011067702A1 (en) 2009-12-02 2011-06-09 Axxana (Israel) Ltd. Distributed intelligent network
US8943379B2 (en) * 2009-12-26 2015-01-27 Intel Corporation Retry based protocol with source/receiver FIFO recovery and anti-starvation mechanism to support dynamic pipeline lengthening for ECC error correction
US8627138B1 (en) * 2010-03-26 2014-01-07 Emc Corporation Data protection system and method
US8504517B2 (en) * 2010-03-29 2013-08-06 Commvault Systems, Inc. Systems and methods for selective data replication
US8725698B2 (en) 2010-03-30 2014-05-13 Commvault Systems, Inc. Stub file prioritization in a data replication system
US8504515B2 (en) 2010-03-30 2013-08-06 Commvault Systems, Inc. Stubbing systems and methods in a data replication environment
US8352422B2 (en) 2010-03-30 2013-01-08 Commvault Systems, Inc. Data restore systems and methods in a replication environment
WO2011150391A1 (en) 2010-05-28 2011-12-01 Commvault Systems, Inc. Systems and methods for performing data replication
US8478845B2 (en) * 2010-08-16 2013-07-02 International Business Machines Corporation End-to-end provisioning of storage clouds
US8589732B2 (en) * 2010-10-25 2013-11-19 Microsoft Corporation Consistent messaging with replication
US9971656B2 (en) 2010-12-13 2018-05-15 International Business Machines Corporation Instant data restoration
US9020895B1 (en) * 2010-12-27 2015-04-28 Netapp, Inc. Disaster recovery for virtual machines across primary and secondary sites
US8510267B2 (en) * 2011-03-08 2013-08-13 Rackspace Us, Inc. Synchronization of structured information repositories
US20130054870A1 (en) * 2011-08-22 2013-02-28 Byungcheol Cho Network-capable raid controller for a semiconductor storage device
US9298715B2 (en) 2012-03-07 2016-03-29 Commvault Systems, Inc. Data storage system utilizing proxy device for storage operations
US9471578B2 (en) 2012-03-07 2016-10-18 Commvault Systems, Inc. Data storage system utilizing proxy device for storage operations
US10417027B1 (en) 2012-03-30 2019-09-17 EMC IP Holding Company LLC Virtual machine proxy server for hyper-V image backup and recovery
IN2014KN02343A (en) * 2012-04-13 2015-05-01 Citrix Systems Inc
US9342537B2 (en) 2012-04-23 2016-05-17 Commvault Systems, Inc. Integrated snapshot interface for a data storage system
US9825932B2 (en) * 2013-01-09 2017-11-21 Qatar Foundation Storage system and method of storing and managing data
US9336226B2 (en) 2013-01-11 2016-05-10 Commvault Systems, Inc. Criteria-based data synchronization management
US9886346B2 (en) 2013-01-11 2018-02-06 Commvault Systems, Inc. Single snapshot for multiple agents
US9307011B2 (en) * 2013-04-01 2016-04-05 Netapp, Inc. Synchronous mirroring of NVLog to multiple destinations (architecture level)
US20140300983A1 (en) * 2013-04-05 2014-10-09 Front Porch Digital, Inc. Media storage and management system
US20140372583A1 (en) * 2013-06-13 2014-12-18 Johnny Tseng Expandable departmental cloud server systems
WO2015056169A1 (en) 2013-10-16 2015-04-23 Axxana (Israel) Ltd. Zero-transaction-loss recovery for database systems
US9575773B2 (en) * 2013-10-23 2017-02-21 Vmware, Inc. Monitoring multiple remote desktops on a wireless device
US20150160864A1 (en) * 2013-12-09 2015-06-11 Netapp, Inc. Systems and methods for high availability in multi-node storage networks
US9632874B2 (en) 2014-01-24 2017-04-25 Commvault Systems, Inc. Database application backup in single snapshot for multiple applications
US9639426B2 (en) 2014-01-24 2017-05-02 Commvault Systems, Inc. Single snapshot for multiple applications
US9495251B2 (en) 2014-01-24 2016-11-15 Commvault Systems, Inc. Snapshot readiness checking and reporting
US9753812B2 (en) 2014-01-24 2017-09-05 Commvault Systems, Inc. Generating mapping information for single snapshot for multiple applications
US9660930B2 (en) 2014-03-17 2017-05-23 Splunk Inc. Dynamic data server nodes
US9838346B2 (en) 2014-03-17 2017-12-05 Splunk Inc. Alerting on dual-queue systems
US9753818B2 (en) 2014-09-19 2017-09-05 Splunk Inc. Data forwarding using multiple data pipelines
US9836358B2 (en) * 2014-03-17 2017-12-05 Splunk Inc. Ephemeral remote data store for dual-queue systems
US9838467B2 (en) * 2014-03-17 2017-12-05 Splunk Inc. Dynamically instantiating dual-queue systems
JP6264116B2 (en) * 2014-03-18 2018-01-24 富士通株式会社 Information processing apparatus, control method, and control program
CA2847683A1 (en) * 2014-03-28 2015-09-28 Kapsch Trafficcom Ag An electronic commerce transaction system using electronic toll collection transponders
US9672165B1 (en) * 2014-05-21 2017-06-06 Veritas Technologies Llc Data management tier coupling primary storage and secondary storage
US10671500B2 (en) * 2014-06-09 2020-06-02 Oracle International Corporation Sequential resilvering of storage devices with reduced restarts
US9774672B2 (en) 2014-09-03 2017-09-26 Commvault Systems, Inc. Consolidated processing of storage-array commands by a snapshot-control media agent
US10042716B2 (en) 2014-09-03 2018-08-07 Commvault Systems, Inc. Consolidated processing of storage-array commands using a forwarder media agent in conjunction with a snapshot-control media agent
KR20170065575A (en) * 2014-09-24 2017-06-13 브이5 시스템즈, 인크. Dynamic data management
CN105653345A (en) * 2014-10-17 2016-06-08 伊姆西公司 Method and device supporting data nonvolatile random access
US10230670B1 (en) 2014-11-10 2019-03-12 Google Llc Watermark-based message queue
US9648105B2 (en) 2014-11-14 2017-05-09 Commvault Systems, Inc. Unified snapshot storage management, using an enhanced storage manager and enhanced media agents
US9448731B2 (en) 2014-11-14 2016-09-20 Commvault Systems, Inc. Unified snapshot storage management
CN105700819B (en) * 2014-11-28 2019-04-12 国际商业机器公司 Method and system for network data storage
US10296422B1 (en) 2015-01-31 2019-05-21 Veritas Technologies Llc Low cost, heterogeneous method of transforming replicated data for consumption in the cloud
US10418762B2 (en) 2015-03-09 2019-09-17 ZPE Systems, Inc. High serial port count infrastructure management device
US11337323B2 (en) 2015-03-09 2022-05-17 ZPE Systems, Inc. Modular infrastructure management device
US9760458B2 (en) * 2015-06-02 2017-09-12 Netapp, Inc. Dynamic mirroring
US10379958B2 (en) 2015-06-03 2019-08-13 Axxana (Israel) Ltd. Fast archiving for database systems
US9407585B1 (en) 2015-08-07 2016-08-02 Machine Zone, Inc. Scalable, real-time messaging system
US10333879B2 (en) 2015-08-07 2019-06-25 Satori Worldwide, Llc Scalable, real-time messaging system
US9319363B1 (en) 2015-08-07 2016-04-19 Machine Zone, Inc. Scalable, real-time messaging system
US9602455B2 (en) 2015-08-07 2017-03-21 Machine Zone, Inc. Scalable, real-time messaging system
US9385976B1 (en) 2015-10-09 2016-07-05 Machine Zone, Inc. Systems and methods for storing message data
US9319365B1 (en) 2015-10-09 2016-04-19 Machine Zone, Inc. Systems and methods for storing and transferring message data
US9397973B1 (en) 2015-10-16 2016-07-19 Machine Zone, Inc. Systems and methods for transferring message data
US10630802B2 (en) * 2015-12-07 2020-04-21 International Business Machines Corporation Read caching in PPRC environments
US9430163B1 (en) 2015-12-15 2016-08-30 International Business Machines Corporation Implementing synchronization for remote disk mirroring
US10592128B1 (en) * 2015-12-30 2020-03-17 EMC IP Holding Company LLC Abstraction layer
US10721120B2 (en) * 2016-03-08 2020-07-21 ZPE Systems, Inc. Infrastructure management device
US10503753B2 (en) 2016-03-10 2019-12-10 Commvault Systems, Inc. Snapshot replication operations based on incremental block change tracking
US9602450B1 (en) 2016-05-16 2017-03-21 Machine Zone, Inc. Maintaining persistence of a messaging system
US9990151B2 (en) * 2016-05-27 2018-06-05 Netapp, Inc. Methods for flexible data-mirroring to improve storage performance during mobility events and devices thereof
US10404647B2 (en) 2016-06-07 2019-09-03 Satori Worldwide, Llc Message compression in scalable messaging system
US9608928B1 (en) 2016-07-06 2017-03-28 Machine Zone, Inc. Multiple-speed message channel of messaging system
US9967203B2 (en) 2016-08-08 2018-05-08 Satori Worldwide, Llc Access control for message channels in a messaging system
CN107707583B (en) * 2016-08-08 2020-11-17 环旭电子股份有限公司 Cloud data transmission system and dynamic distribution method thereof
US10374986B2 (en) 2016-08-23 2019-08-06 Satori Worldwide, Llc Scalable, real-time messaging system
US10305981B2 (en) 2016-08-31 2019-05-28 Satori Worldwide, Llc Data replication in scalable messaging system
US10101941B2 (en) 2016-09-20 2018-10-16 International Business Machines Corporation Data mirror invalid timestamped write handling
US10839852B2 (en) 2016-09-21 2020-11-17 International Business Machines Corporation Log snapshot control on an automated data storage library
US10782890B2 (en) * 2016-09-21 2020-09-22 International Business Machines Corporation Log snapshot procedure control on an automated data storage library
US9667681B1 (en) 2016-09-23 2017-05-30 Machine Zone, Inc. Systems and methods for providing messages to multiple subscribers
US10089198B1 (en) * 2016-09-29 2018-10-02 EMC IP Holding Company LLC Data storage system with active-standby system management
US10447623B2 (en) 2017-02-24 2019-10-15 Satori Worldwide, Llc Data storage systems and methods using a real-time messaging system
US10270726B2 (en) 2017-02-24 2019-04-23 Satori Worldwide, Llc Selective distribution of messages in a scalable, real-time messaging system
US10187278B2 (en) 2017-02-24 2019-01-22 Satori Worldwide, Llc Channel management in scalable messaging system
US10592326B2 (en) 2017-03-08 2020-03-17 Axxana (Israel) Ltd. Method and apparatus for data loss assessment
US10296375B1 (en) * 2017-08-26 2019-05-21 Plesk International Gmbh Method of fast file transfer across network
US10999323B2 (en) * 2017-09-22 2021-05-04 Nec Corporation Network gateway spoofing detection and mitigation
US10740022B2 (en) 2018-02-14 2020-08-11 Commvault Systems, Inc. Block-level live browsing and private writable backup copies using an ISCSI server
TWI675368B (en) * 2018-02-21 2019-10-21 神雲科技股份有限公司 Control method of starting multiple hard disk
CN110333904B (en) * 2018-03-29 2022-07-12 佛山市顺德区顺达电脑厂有限公司 Multi-hard-disk starting control method
WO2019237010A1 (en) * 2018-06-08 2019-12-12 Fungible, Inc. Early acknowledgment for write operations
CN109117310A (en) * 2018-08-08 2019-01-01 中国建设银行股份有限公司 Realize disaster tolerance system, the method and device of data backup
US11275764B2 (en) * 2018-10-11 2022-03-15 EMC IP Holding Company LLC Highly resilient synchronous replication with automatic recovery
US10936451B2 (en) * 2018-10-24 2021-03-02 EMC IP Holding Company LLC Concurrent remote IO processing for synchronous replication
CN112117572B (en) * 2019-06-21 2022-03-25 默升科技集团有限公司 Debug arrangement for active ethernet cable
US11042318B2 (en) 2019-07-29 2021-06-22 Commvault Systems, Inc. Block-level data replication
US11513899B2 (en) 2020-11-30 2022-11-29 International Business Machines Corporation Selectively capturing snapshots of volumes based on modified capacity
US11575745B2 (en) * 2021-01-05 2023-02-07 Nutanix, Inc. Dynamic feedback technique for improving disaster recovery replication performance
TWI758175B (en) * 2021-04-30 2022-03-11 科智企業股份有限公司 A data processing system and method
US11550955B1 (en) 2021-07-20 2023-01-10 Red Hat, Inc. Automatically anonymizing data in a distributed storage system
US11809285B2 (en) 2022-02-09 2023-11-07 Commvault Systems, Inc. Protecting a management database of a data storage management system to meet a recovery point objective (RPO)
US11922026B2 (en) * 2022-02-16 2024-03-05 T-Mobile Usa, Inc. Preventing data loss in a filesystem by creating duplicates of data in parallel, such as charging data in a wireless telecommunications network
US12056018B2 (en) 2022-06-17 2024-08-06 Commvault Systems, Inc. Systems and methods for enforcing a recovery point objective (RPO) for a production database without generating secondary copies of the production database

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4982324A (en) * 1988-12-19 1991-01-01 International Business Machines Corporation Method of and system for using device drivers to couple the communication and data storage of remote computer systems
US5537533A (en) * 1994-08-11 1996-07-16 Miralink Corporation System and method for remote mirroring of digital data from a primary network server to a remote network server
US5745748A (en) * 1994-12-09 1998-04-28 Sprint Communication Co. L.P. System and method for direct accessing of remote data
US5901327A (en) * 1996-05-28 1999-05-04 Emc Corporation Bundling of write data from channel commands in a command chain for transmission over a data link between data storage systems for remote data mirroring

Family Cites Families (98)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US527660A (en) * 1894-10-16 Neck-yoke
US4045781A (en) 1976-02-13 1977-08-30 Digital Equipment Corporation Memory module with selectable byte addressing for digital data processing system
US4347563A (en) 1980-06-16 1982-08-31 Forney Engineering Company Industrial control system
US4351023A (en) 1980-04-11 1982-09-21 The Foxboro Company Process control system with improved system security features
US4507751A (en) 1982-06-21 1985-03-26 International Business Machines Corporation Method and apparatus for logging journal data using a log write ahead data set
US4755928A (en) 1984-03-05 1988-07-05 Storage Technology Corporation Outboard back-up and recovery system with transfer of randomly accessible data sets between cache and host and cache and tape simultaneously
US4827349A (en) * 1985-04-30 1989-05-02 Canon Kabushiki Kaisha Communication terminal device
US4710870A (en) 1985-07-10 1987-12-01 Bell Communications Research, Inc. Central computer backup system utilizing localized data bases
US4765928A (en) 1985-08-19 1988-08-23 Mrinal Thakur Intrinsically conductive doped polymers of enhanced stability
US4751702A (en) 1986-02-10 1988-06-14 International Business Machines Corporation Improving availability of a restartable staged storage data base system that uses logging facilities
US4819159A (en) 1986-08-29 1989-04-04 Tolerant Systems, Inc. Distributed multiprocess transaction processing system and method
US5051887A (en) 1987-08-25 1991-09-24 International Business Machines Corporation Maintaining duplex-paired storage devices during gap processing using of a dual copy function
US5146605A (en) 1987-11-12 1992-09-08 International Business Machines Corporation Direct control facility for multiprocessor network
EP0323013B1 (en) 1987-11-30 1995-08-30 International Business Machines Corporation Method of operating a multiprocessor system employing a shared virtual memory
JP2924905B2 (en) 1988-03-25 1999-07-26 エヌシーアール インターナショナル インコーポレイテッド File backup system
US5136707A (en) 1988-10-28 1992-08-04 At&T Bell Laboratories Reliable database administration arrangement
US5222217A (en) 1989-01-18 1993-06-22 International Business Machines Corporation System and method for implementing operating system message queues with recoverable shared virtual storage
US4958270A (en) 1989-01-23 1990-09-18 Honeywell Inc. Method for control data base updating of a redundant processor in a process control system
US4959768A (en) 1989-01-23 1990-09-25 Honeywell Inc. Apparatus for tracking predetermined data for updating a secondary data base
US5454099A (en) 1989-07-25 1995-09-26 International Business Machines Corporation CPU implemented method for backing up modified data sets in non-volatile store for recovery in the event of CPU failure
US5133065A (en) 1989-07-27 1992-07-21 Personal Computer Peripherals Corporation Backup computer program for networks
US5276860A (en) 1989-12-19 1994-01-04 Epoch Systems, Inc. Digital data processor with improved backup storage
US5155845A (en) 1990-06-15 1992-10-13 Storage Technology Corporation Data storage system for providing redundant copies of data on different disk drives
US5157663A (en) * 1990-09-24 1992-10-20 Novell, Inc. Fault tolerant computer system
US5544347A (en) * 1990-09-24 1996-08-06 Emc Corporation Data storage system controlled remote data mirroring with respectively maintained data indices
GB2251502B (en) * 1990-11-07 1995-06-14 Nonstop Networks Limited Data-loss prevention products
US5633999A (en) * 1990-11-07 1997-05-27 Nonstop Networks Limited Workstation-implemented data storage re-routing for server fault-tolerance on computer networks
US5257384A (en) * 1991-09-09 1993-10-26 Compaq Computer Corporation Asynchronous protocol for computer system manager
US5668986A (en) 1991-10-02 1997-09-16 International Business Machines Corporation Method and apparatus for handling data storage requests in a distributed data base environment
EP0541281B1 (en) * 1991-11-04 1998-04-29 Commvault Systems, Inc. Incremental-computer-file backup using signatures
WO1994000816A1 (en) 1992-06-18 1994-01-06 Andor Systems, Inc. Remote dual copy of data in computer systems
US5446871A (en) * 1993-03-23 1995-08-29 International Business Machines Corporation Method and arrangement for multi-system remote data duplexing and recovery
US5689678A (en) 1993-03-11 1997-11-18 Emc Corporation Distributed storage array system having a plurality of modular control units
US6604118B2 (en) * 1998-07-31 2003-08-05 Network Appliance, Inc. File system image transfer
US5812748A (en) * 1993-06-23 1998-09-22 Vinca Corporation Method for improving recovery performance from hardware and software errors in a fault-tolerant computer system
US5604487A (en) * 1993-07-30 1997-02-18 Lockheed Martin Tactical Systems, Inc. Apparatus and method for user-selective data communication with verification
KR0128271B1 (en) * 1994-02-22 1998-04-15 윌리암 티. 엘리스 Remote data duplexing
US5574950A (en) * 1994-03-01 1996-11-12 International Business Machines Corporation Remote data shadowing using a multimode interface to dynamically reconfigure control link-level and communication link-level
US5592618A (en) * 1994-10-03 1997-01-07 International Business Machines Corporation Remote copy secondary data copy validation-audit function
US5835953A (en) * 1994-10-13 1998-11-10 Vinca Corporation Backup system that takes a snapshot of the locations in a mass storage device that has been identified for updating prior to updating
US5574906A (en) * 1994-10-24 1996-11-12 International Business Machines Corporation System and method for reducing storage requirement in backup subsystems utilizing segmented compression and differencing
CA2168087A1 (en) * 1995-02-13 1996-08-14 James S. Coman Operating system based remote communication system
US6453325B1 (en) 1995-05-24 2002-09-17 International Business Machines Corporation Method and means for backup and restoration of a database system linked to a system for filing data
US5799141A (en) * 1995-06-09 1998-08-25 Qualix Group, Inc. Real-time data protection system and method
US5680640A (en) * 1995-09-01 1997-10-21 Emc Corporation System for migrating data by selecting a first or second transfer means based on the status of a data element map initialized to a predetermined state
US5987544A (en) 1995-09-08 1999-11-16 Digital Equipment Corporation System interface protocol with optional module cache
US5907672A (en) * 1995-10-04 1999-05-25 Stac, Inc. System for backing up computer disk volumes with error remapping of flawed memory addresses
JP3856855B2 (en) * 1995-10-06 2006-12-13 三菱電機株式会社 Differential backup method
US5778395A (en) * 1995-10-23 1998-07-07 Stac, Inc. System for backing up files from disk volumes on multiple nodes of a computer network
ATE221710T1 (en) * 1995-10-23 2002-08-15 Nokia Corp METHOD FOR PACKET DATA TRANSMISSION USING HYBRID FEC/ARQ TYPE II METHOD
US5870537A (en) * 1996-03-13 1999-02-09 International Business Machines Corporation Concurrent switch to shadowed device for storage controller and device errors
US5852715A (en) 1996-03-19 1998-12-22 Emc Corporation System for currently updating database by one host and reading the database by different host for the purpose of implementing decision support functions
US6366930B1 (en) * 1996-04-12 2002-04-02 Computer Associates Think, Inc. Intelligent data inventory & asset management systems method and apparatus
US6052797A (en) * 1996-05-28 2000-04-18 Emc Corporation Remotely mirrored data storage system with a count indicative of data consistency
US6101497A (en) 1996-05-31 2000-08-08 Emc Corporation Method and apparatus for independent and simultaneous access to a common data set
US5857208A (en) 1996-05-31 1999-01-05 Emc Corporation Method and apparatus for performing point in time backup operation in a computer system
US6477627B1 (en) * 1996-05-31 2002-11-05 Emc Corporation Method and apparatus for mirroring data in a remote data storage system
US5933653A (en) 1996-05-31 1999-08-03 Emc Corporation Method and apparatus for mirroring data in a remote data storage system
US5845295A (en) 1996-08-27 1998-12-01 Unisys Corporation System for providing instantaneous access to a snapshot Op data stored on a storage medium for offline analysis
US6158019A (en) * 1996-12-15 2000-12-05 Delta-Tek Research, Inc. System and apparatus for merging a write event journal and an original storage to produce an updated storage using an event map
US6073209A (en) * 1997-03-31 2000-06-06 Ark Research Corporation Data storage controller providing multiple hosts with access to multiple storage subsystems
US6578120B1 (en) * 1997-06-24 2003-06-10 International Business Machines Corporation Synchronization and resynchronization of loosely-coupled copy operations between a primary and a remote secondary DASD volume under concurrent updating
US6366988B1 (en) * 1997-07-18 2002-04-02 Storactive, Inc. Systems and methods for electronic data storage management
US5993653C1 (en) * 1997-08-11 2001-11-06 Phenomenex Composition and column used in hplc
JP4128641B2 (en) 1997-10-13 2008-07-30 株式会社東芝 Data backup method
US6058106A (en) * 1997-10-20 2000-05-02 Motorola, Inc. Network protocol method, access point device and peripheral devices for providing for an efficient centrally coordinated peer-to-peer wireless communications network
US6636327B2 (en) * 1997-12-04 2003-10-21 Canon Kabushiki Kaisha Image processing apparatus and method
US6065018A (en) * 1998-03-04 2000-05-16 International Business Machines Corporation Synchronizing recovery log having time stamp to a remote site for disaster recovery of a primary database having related hierarchial and relational databases
US6324654B1 (en) * 1998-03-30 2001-11-27 Legato Systems, Inc. Computer network remote data mirroring system
US6167494A (en) * 1998-04-28 2000-12-26 International Business Machine Corporation Method and system for recovering from operating system failure
US6144999A (en) * 1998-05-29 2000-11-07 Sun Microsystems, Incorporated Method and apparatus for file system disaster recovery
US6604236B1 (en) * 1998-06-30 2003-08-05 Iora, Ltd. System and method for generating file updates for files stored on read-only media
US6282602B1 (en) * 1998-06-30 2001-08-28 Emc Corporation Method and apparatus for manipulating logical objects in a data storage system
US6237008B1 (en) 1998-07-20 2001-05-22 International Business Machines Corporation System and method for enabling pair-pair remote copy storage volumes to mirror data in another storage volume
DE69938378T2 (en) * 1998-08-20 2009-04-30 Hitachi, Ltd. Copy data to storage systems
US6308284B1 (en) * 1998-08-28 2001-10-23 Emc Corporation Method and apparatus for maintaining data coherency
US6327671B1 (en) * 1998-11-18 2001-12-04 International Business Machines Corporation Delta compressed asynchronous remote copy
US6397308B1 (en) * 1998-12-31 2002-05-28 Emc Corporation Apparatus and method for differential backup and restoration of data in a computer storage system
US7107395B1 (en) * 1998-12-31 2006-09-12 Emc Corporation Apparatus and methods for operating a computer storage system
US6487561B1 (en) * 1998-12-31 2002-11-26 Emc Corporation Apparatus and methods for copying, backing up, and restoring data using a backup segment size larger than the storage block size
US6718347B1 (en) 1999-01-05 2004-04-06 Emc Corporation Method and apparatus for maintaining coherence among copies of a database shared by multiple computers
US6209002B1 (en) 1999-02-17 2001-03-27 Emc Corporation Method and apparatus for cascading data through redundant data storage units
US6571312B1 (en) 1999-02-19 2003-05-27 Mitsubishi Denki Kabushiki Kaisha Data storage method and data processing device using an erasure block buffer and write buffer for writing and erasing data in memory
US6397307B2 (en) * 1999-02-23 2002-05-28 Legato Systems, Inc. Method and system for mirroring and archiving mass storage
US7000069B2 (en) * 1999-04-05 2006-02-14 Hewlett-Packard Development Company, L.P. Apparatus and method for providing very large virtual storage volumes using redundant arrays of disks
US6370626B1 (en) * 1999-04-30 2002-04-09 Emc Corporation Method and apparatus for independent and simultaneous access to a common data set
US6717943B1 (en) * 1999-06-03 2004-04-06 David Schwering System and method for routing and processing data packets
US6405337B1 (en) * 1999-06-21 2002-06-11 Ericsson Inc. Systems, methods and computer program products for adjusting a timeout for message retransmission based on measured round-trip communications delays
US6611901B1 (en) * 1999-07-02 2003-08-26 International Business Machines Corporation Method, system, and program for maintaining electronic data as of a point-in-time
US6912629B1 (en) * 1999-07-28 2005-06-28 Storage Technology Corporation System and method for restoring data from secondary volume to primary volume in a data storage system
US6385683B1 (en) * 1999-08-13 2002-05-07 Lsi Logic Corporation Methods and apparatus for raid hardware sequencing to achieve a higher performance raid architecture
US6574709B1 (en) * 1999-09-30 2003-06-03 International Business Machine Corporation System, apparatus, and method providing cache data mirroring to a data storage system
US6928468B2 (en) * 1999-10-29 2005-08-09 Koninklijke Philips Electronics N.V. System for broadcasting software applications and portable data communications device for use in such a system
TW454120B (en) 1999-11-11 2001-09-11 Miralink Corp Flexible remote data mirroring
US7203732B2 (en) 1999-11-11 2007-04-10 Miralink Corporation Flexible remote data mirroring
EP1193962A1 (en) 2000-09-21 2002-04-03 Siemens Aktiengesellschaft Transmission of real-time data from a network element to an application server with recovery method for link failure
US20040233910A1 (en) 2001-02-23 2004-11-25 Wen-Shyen Chen Storage area network using a data communication protocol
US7457980B2 (en) * 2004-08-13 2008-11-25 Ken Qing Yang Data replication method over a limited bandwidth network by mirroring parities

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4982324A (en) * 1988-12-19 1991-01-01 International Business Machines Corporation Method of and system for using device drivers to couple the communication and data storage of remote computer systems
US5537533A (en) * 1994-08-11 1996-07-16 Miralink Corporation System and method for remote mirroring of digital data from a primary network server to a remote network server
US5745748A (en) * 1994-12-09 1998-04-28 Sprint Communication Co. L.P. System and method for direct accessing of remote data
US5901327A (en) * 1996-05-28 1999-05-04 Emc Corporation Bundling of write data from channel commands in a command chain for transmission over a data link between data storage systems for remote data mirroring

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP1240591A4 *

Cited By (59)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10003647B2 (en) 1999-11-11 2018-06-19 Intellectual Ventures Ii Llc Flexible remote data mirroring
US8762328B2 (en) 1999-11-11 2014-06-24 Intellectual Ventures Ii Llc Flexible remote data mirroring
US8856202B2 (en) 1999-11-11 2014-10-07 Intellectual Ventures Ii Llc Flexible remote data mirroring
US9270752B2 (en) 1999-11-11 2016-02-23 Intellectual Ventures Ii Llc Flexible remote data mirroring
EP1402420A4 (en) * 2001-06-26 2008-01-09 Emc Corp Mirroring network data to establish virtual storage area network
EP1402420A2 (en) * 2001-06-26 2004-03-31 EMC Corporation Mirroring network data to establish virtual storage area network
US9002910B2 (en) 2001-07-06 2015-04-07 Ca, Inc. Systems and methods of information backup
US7389292B2 (en) 2001-07-06 2008-06-17 Computer Associates Think, Inc. Systems and methods of information backup
WO2003005625A2 (en) * 2001-07-06 2003-01-16 Computer Associates Think, Inc. Systems and methods of information backup
WO2003005625A3 (en) * 2001-07-06 2003-05-30 Computer Ass Think Inc Systems and methods of information backup
US7552214B2 (en) 2001-07-06 2009-06-23 Computer Associates Think, Inc. Systems and methods of information backup
US8370450B2 (en) 2001-07-06 2013-02-05 Ca, Inc. Systems and methods for information backup
US7734594B2 (en) 2001-07-06 2010-06-08 Computer Associates Think, Inc. Systems and methods of information backup
GB2396723A (en) * 2001-09-20 2004-06-30 Pty Ltd Cebridge Data protection and retrival
WO2003025760A1 (en) * 2001-09-20 2003-03-27 Cebridge Pty. Ltd. Data protection and retrival
US6968369B2 (en) 2001-09-27 2005-11-22 Emc Corporation Remote data facility over an IP network
JP4543051B2 (en) * 2001-09-27 2010-09-15 イーエムシー コーポレイション Remote data facility on IP network
US7430589B2 (en) 2001-09-27 2008-09-30 Emc Corporation Remote data facility over an IP network
JP2007282197A (en) * 2001-09-27 2007-10-25 Emc Corp Remote data facility over ip network
EP1298530A1 (en) * 2001-09-27 2003-04-02 Emc Corporation Remote data facility over an IP network
US7730153B1 (en) 2001-12-04 2010-06-01 Netapp, Inc. Efficient use of NVRAM during takeover in a node cluster
US7293105B2 (en) 2001-12-21 2007-11-06 Cisco Technology, Inc. Methods and apparatus for implementing a high availability fibre channel switch
AU2002359739B2 (en) * 2001-12-21 2008-02-14 Cisco Technology, Inc. Methods and apparatus for implementing a high availability fibre channel switch
WO2003061222A1 (en) * 2001-12-21 2003-07-24 Cisco Technology, Inc. Methods and apparatus for implementing a high availability fibre channel switch
US7171452B1 (en) 2002-10-31 2007-01-30 Network Appliance, Inc. System and method for monitoring cluster partner boot status over a cluster interconnect
US7437423B1 (en) 2002-10-31 2008-10-14 Network Appliance, Inc. System and method for monitoring cluster partner boot status over a cluster interconnect
US7953924B1 (en) 2003-03-03 2011-05-31 Netapp, Inc. System and method for coordinating cluster state information
US7685358B1 (en) 2003-03-03 2010-03-23 Netapp, Inc. System and method for coordinating cluster state information
US7921273B2 (en) 2003-06-17 2011-04-05 International Business Machines Corporation Method, system, and article of manufacture for remote copying of data
WO2004111849A2 (en) * 2003-06-17 2004-12-23 International Business Machines Corporation Method, system and article of manufacture for remote copying of data
WO2004111849A3 (en) * 2003-06-17 2005-07-28 Ibm Method, system and article of manufacture for remote copying of data
US7266665B2 (en) 2003-06-17 2007-09-04 International Business Machines Corporation Method, system, and article of manufacture for remote copying of data
WO2005071544A1 (en) * 2003-06-18 2005-08-04 International Business Machines Corporation Method. system. and program for andling a failover to a remote storage location
CN100403272C (en) * 2003-06-18 2008-07-16 国际商业机器公司 Method, system, and program for handling a failover to a remote storage location
US7043665B2 (en) 2003-06-18 2006-05-09 International Business Machines Corporation Method, system, and program for handling a failover to a remote storage location
US7593996B2 (en) 2003-07-18 2009-09-22 Netapp, Inc. System and method for establishing a peer connection using reliable RDMA primitives
EP1498816A1 (en) * 2003-07-18 2005-01-19 Network Appliance, Inc. System and method for reliable peer communication in a clustered storage system
US7716323B2 (en) 2003-07-18 2010-05-11 Netapp, Inc. System and method for reliable peer communication in a clustered storage system
WO2005048124A3 (en) * 2003-11-04 2005-11-03 Constant Data Inc Hybrid real-time data replication
US8938419B2 (en) 2003-11-04 2015-01-20 Dell Software Inc. Hybrid real-time data replication
WO2005048124A2 (en) * 2003-11-04 2005-05-26 Constant Data, Inc. Hybrid real-time data replication
US7870354B2 (en) 2003-11-04 2011-01-11 Bakbone Software, Inc. Data replication from one-to-one or one-to-many heterogeneous devices
US7836014B2 (en) 2003-11-04 2010-11-16 Bakbone Software, Inc. Hybrid real-time data replication
US8438352B2 (en) 2003-11-04 2013-05-07 Quest Software, Inc. Data replication from one-to-one or one-to-many heterogeneous devices
US7293195B1 (en) 2004-01-29 2007-11-06 Network Appliance, Inc. System and method for coordinated bringup of a storage appliance in a cluster configuration
US8028054B1 (en) 2004-01-29 2011-09-27 Netapp, Inc. System and method for coordinated bringup of a storage appliance in a cluster configuration
US8170996B2 (en) 2004-02-03 2012-05-01 Hitachi, Ltd. Storage subsystem
EP1562117A3 (en) * 2004-02-03 2011-08-03 Hitachi, Ltd. Data restoration method using a log and corresponding storage subsystem
EP1562117A2 (en) 2004-02-03 2005-08-10 Hitachi, Ltd. Data restoration method using a log and corresponding storage subsystem
EP1647891A2 (en) * 2004-10-14 2006-04-19 Hitachi Ltd. Computer system performing remote copying via an intermediate site
EP1647891A3 (en) * 2004-10-14 2008-06-04 Hitachi Ltd. Computer system performing remote copying via an intermediate site
EP1914633A3 (en) * 2006-10-12 2008-08-06 Hitachi, Ltd. Storage apparatus
US7707456B2 (en) 2006-10-12 2010-04-27 Hitachi, Ltd. Storage system
EP2307975A1 (en) * 2008-07-02 2011-04-13 Hewlett-Packard Development Company, L.P. Verification of remote copies of data
EP2307975A4 (en) * 2008-07-02 2012-01-18 Hewlett Packard Development Co Verification of remote copies of data
US8688798B1 (en) 2009-04-03 2014-04-01 Netapp, Inc. System and method for a shared write address protocol over a remote direct memory access connection
US9544243B2 (en) 2009-04-03 2017-01-10 Netapp, Inc. System and method for a shared write address protocol over a remote direct memory access connection
GB2473079B (en) * 2009-08-28 2014-08-27 Bizconline Ltd Remote online backup system and method based on stager fragmented transmission
GB2473079A (en) * 2009-08-28 2011-03-02 Bizconline Ltd Remote backup using journals relayed via Stagers

Also Published As

Publication number Publication date
US20150113095A1 (en) 2015-04-23
US20160134698A1 (en) 2016-05-12
US20090313389A1 (en) 2009-12-17
US10003647B2 (en) 2018-06-19
US8856202B2 (en) 2014-10-07
EP1240591A4 (en) 2006-10-11
CA2427920A1 (en) 2001-05-17
US20050027892A1 (en) 2005-02-03
EP1240591A1 (en) 2002-09-18
US9270752B2 (en) 2016-02-23
HK1051417A1 (en) 2003-08-01
AU4325200A (en) 2001-06-06
TW454120B (en) 2001-09-11
US8762328B2 (en) 2014-06-24
US20100049798A1 (en) 2010-02-25

Similar Documents

Publication Publication Date Title
US10003647B2 (en) Flexible remote data mirroring
US7203732B2 (en) Flexible remote data mirroring
AU2001265335B2 (en) Flexible remote data mirroring
AU2001265335A1 (en) Flexible remote data mirroring
US6363462B1 (en) Storage controller providing automatic retention and deletion of synchronous back-up data
US7203796B1 (en) Method and apparatus for synchronous data mirroring
US9209989B2 (en) Causation of a data read operation against a first storage system by a server associated with a second storage system according to a host generated instruction
US8055745B2 (en) Methods and apparatus for accessing data from a primary data storage system for secondary storage
US9846706B1 (en) Managing mounting of file systems
EP2159680B1 (en) Secure virtual tape management system with balanced storage and multi-mirror options
US20060136685A1 (en) Method and system to maintain data consistency over an internet small computer system interface (iSCSI) network
US20100325091A1 (en) Data Mirroring Method
US8683144B2 (en) Causation of a data read against a first storage system to optionally store a data write to preserve the version to allow viewing and recovery
EP1825378A2 (en) Functional partitioning method for providing modular data storage systems
US10095444B1 (en) Tape emulation alternate data path
US8942073B1 (en) Maintaining tape emulation consistency
US20220100393A1 (en) Direct snapshot to external storage
WO2001082078A9 (en) Method and apparatus for maintaining the integrity of configuration data in redundant, fault tolerant network appliances

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AL AM AT AU AZ BA BB BG BR BY CA CH CN CR CU CZ DE DK DM EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GH GM KE LS MW SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
WWE Wipo information: entry into national phase

Ref document number: 2000923070

Country of ref document: EP

WWP Wipo information: published in national office

Ref document number: 2000923070

Country of ref document: EP

REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

WWE Wipo information: entry into national phase

Ref document number: 2427920

Country of ref document: CA