WO2013005208A1 - Procédé et appareil de fourniture de services de données de niveau hyperviseur pour virtualisation de serveur - Google Patents

Procédé et appareil de fourniture de services de données de niveau hyperviseur pour virtualisation de serveur Download PDF

Info

Publication number
WO2013005208A1
WO2013005208A1 PCT/IL2012/000271 IL2012000271W WO2013005208A1 WO 2013005208 A1 WO2013005208 A1 WO 2013005208A1 IL 2012000271 W IL2012000271 W IL 2012000271W WO 2013005208 A1 WO2013005208 A1 WO 2013005208A1
Authority
WO
WIPO (PCT)
Prior art keywords
virtual
cloud
enterprise
data services
protection group
Prior art date
Application number
PCT/IL2012/000271
Other languages
English (en)
Inventor
Ziv Kedem
Gil Levonai
Yair Kuszpet
Chen Burshan
Original Assignee
Zerto Ltd.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US13/175,892 external-priority patent/US11256529B2/en
Priority claimed from US13/175,898 external-priority patent/US9710294B2/en
Application filed by Zerto Ltd. filed Critical Zerto Ltd.
Publication of WO2013005208A1 publication Critical patent/WO2013005208A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5061—Partitioning or combining of resources
    • G06F9/5077—Logical partitioning of resources; Management or configuration of virtualized resources
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1479—Generic software techniques for error detection or fault masking
    • G06F11/1482—Generic software techniques for error detection or fault masking by means of middleware or OS functionality
    • G06F11/1484—Generic software techniques for error detection or fault masking by means of middleware or OS functionality involving virtual machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2035—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant without idle spare hardware
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/14—Error detection or correction of the data by redundancy in operation
    • G06F11/1402—Saving, restoring, recovering or retrying
    • G06F11/1471—Saving, restoring, recovering or retrying involving logging of persistent data for recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2041—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant with more than one idle spare processing component
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
    • G06F11/2048—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant where the redundant components share neither address space nor persistent storage
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/16—Error detection or correction of the data by redundancy in hardware
    • G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
    • G06F11/2097—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements maintaining the standby controller/processing unit updated

Definitions

  • the present invention relates to virtual server computing environments.
  • Virtual servers are logical entities that run as software in a server virtualization infrastructure, referred to as a "hypervisor".
  • hypervisors are VMWARE ® ESX manufactured by VMware, Inc. of Palo Alto, CA, HyperV manufactured by Microsoft Corporation of Redmond, WA, XENSERVER ® manufactured by Citrix Systems, Inc. of Fort Lauderdale, FL, Redhat KVM manufactured by Redhat, Inc. of Raleigh, NC, and Oracle VM manufactured by Oracle Corporation of Redwood Shores, CA.
  • a hypervisor provides storage device emulation, referred to as "virtual disks", to virtual servers. Hypervisor implements virtual disks using back-end technologies such as files on a dedicated file system, or raw mapping to physical devices.
  • virtual servers run their operating systems within an emulation layer that is provided by a hypervisor.
  • virtual servers are software, nevertheless they perform the same tasks as physical servers, including running server applications such as database applications, customer relation management applications and MICROSOFT EXCHANGE SERVER ® .
  • Most applications that run on physical servers are portable to run on virtual servers.
  • virtual servers run applications that service a large number of clients.
  • virtual servers depend critically on data services for their availability, security, mobility and compliance requirements. Data services include inter alia continuous data protection, disaster recovery, remote replication, data security, mobility, and data retention and archiving policies.
  • VDSA virtual data service appliance
  • a tapping filter driver is installed within the hypervisor kernel.
  • the tapping driver has visibility to I/O requests made by virtual servers running on the hypervisor.
  • a VDSA runs on each physical hypervisor.
  • the VDSA is a dedicated virtual server that provides data services; however, the VDSA does not necessarily reside in the actual I/O data path.
  • the VDSA receives the data outside the data path.
  • the tapping driver Whenever a virtual server performs I/O to a virtual disk, the tapping driver identifies the I/O requests to the virtual disk. The tapping driver copies the I/O requests, forwards one copy to the hypervisor' s backend, and forwards another copy to the VDSA.
  • the VDSA Upon receiving an I/O request, the VDSA performs a set of actions to enable various data services.
  • a first action is data analysis, to analyze the data content of the I/O request and to infer information regarding the virtual server's data state.
  • the VDSA may infer the operating system level and the status of the virtual server. This information is subsequently used for reporting and policy purposes.
  • a second action is to store each I/O write request in a dedicated virtual disk for journaling. Since all I/O write requests are journaled on this virtual disk, the virtual disk enables recovery data services for the virtual server, such as restoring the virtual server to an historical image.
  • a third action, optionally performed by the VDSA, is to send I/O write requests to different VDSAs, residing on hypervisors located at different locations, thus enabling disaster recovery data services.
  • the hypervisor architecture of the present invention scales to multiple host sites, each of which hosts multiple hypervisors.
  • the scaling flexibly allows for different numbers of hypervisors at different sites, and different numbers of virtual services and virtual disks within different hypervisors.
  • Each hypervisor includes a VDSA, and each site includes a data services manager to coordinate the VSDA's at the site, and across other sites.
  • Embodiments of the present invention enable flexibly designating one or more virtual servers within one or more hypervisors at a site as being a virtual protection group, and flexibly designating one or more hypervisors, or alternatively one or more virtual servers within one or more hypervisors at another site as being a replication target for the virtual protection group.
  • Write order fidelity is maintained for virtual protection groups.
  • a site may comprise any number of source and target virtual protection groups.
  • a virtual protection group may have more than one replication target. The number of hypervisors and virtual servers within a virtual protection group and its replication target are not required to be the same.
  • the hypervisor architecture of the present invention may be used to provide cloud-based hypervisor level data services to multiple enterprises on a shared physical infrastructure, while maintaining control and data path separation between enterprises for security.
  • the present invention provides bi-directional cloud-based data replication services; i.e., from the enterprise to the cloud, and from the cloud to the enterprise.
  • replication targets may be assigned to a pool of resources that do not expose the enterprise infrastructure, thus providing an additional layer of security and privacy between enterprises that share a target physical infrastructure.
  • the cloud-based data replication services of the present invention support enforcement of data export regulations. As such, data transfer between a source and a destination is automatically restricted if data export regulations restrict data transfer between the corresponding jurisdictions of the source and the destination.
  • a cross-host multi-hypervisor system including a plurality of host sites, each site including at least one hypervisor, each of which includes at least one virtual server, at least one virtual disk that is read from and written to by the at least one virtual server, a tapping driver in communication with the at least one virtual server, which intercepts write requests made by any one of the at least one virtual server to any one of the at least one virtual disk, and a virtual data services appliance, in communication with the tapping driver, which receives the intercepted write requests from the tapping driver, and which provides data services based thereon, and a data services manager for coordinating the virtual data services appliances at the site, and a network for communicatively coupling the plurality of sites, wherein the data services managers coordinate data transfer across the plurality of sites via the network.
  • a system for cloud-based data services for multiple enterprises including a plurality of cloud hypervisors that cooperatively provide cloud-based services to multiple enterprises, each hypervisor including a plurality of cloud virtual servers, each cloud virtual server being associated with an enterprise, at least one cloud virtual disk that is read from and written to by the at least one virtual server, each cloud virtual disk being associated with an enterprise, and a virtual data services appliance, which provides cloud-based data services, and multiple data services managers, one data services manager per respective enterprise, each of which coordinates the respective virtual data services appliances for those cloud hypervisors that service its corresponding enterprise.
  • a system for cloud-based data services for multiple enterprises including a plurality of cloud hypervisors that cooperatively provide cloud-based services to multiple enterprises, each hypervisor including a plurality of cloud virtual servers, each cloud virtual server being associated with an enterprise, at least one cloud virtual disk that is read from and written to by the at least one virtual server, each cloud virtual disk being associated with an enterprise, and at least one virtual data services appliance, one virtual data services appliance per respective enterprise serviced by the hypervisor, which provides cloud-based data services to its respective enterprise, and multiple data services managers, one data services manager per respective enterprise, each of which coordinates the virtual data services appliances that service its respective enterprise.
  • a system for cloud-based data services for multiple enterprises including a plurality of cloud hypervisors that cooperatively provide cloud-based services to multiple enterprises, each enterprise being located in a corresponding jurisdiction, and each hypervisor being associated with a corresponding jurisdiction, wherein each hypervisor includes a plurality of cloud virtual servers, each cloud virtual server being associated with an enterprise, and at least one cloud virtual disk that is read from and written to by the at least one virtual server, each cloud virtual disk being associated with an enterprise, and a rights manager that restricts access of an enterprise to a cloud hypervisor associated with the enterprise if there is a restriction to export data between the jurisdiction associated with the enterprise and the jurisdiction associated with the cloud hypervisor.
  • FIG. 1 is a simplified block diagram of a hypervisor architecture that includes a tapping driver and a virtual data services appliance, in accordance with an embodiment of the present invention
  • FIG. 2 is a simplified data flow chart for a virtual data services appliance, in accordance with an embodiment of the present invention
  • FIG. 3 is a simplified block diagram of a virtual replication system, in accordance with an embodiment of the present invention.
  • FIG. 4 is a simplified block diagram of a cross-host multiple hypervisor system that includes data services managers for multiple sites that have multiple hypervisors, in accordance with an embodiment of the present invention
  • FIG. 5 is a user interface screenshot of bi-directional replication of virtual protection groups, in accordance with an embodiment of the present invention.
  • FIG. 6 is a user interface screenshot of assignment of a replication target for a virtual protection group, in accordance with an embodiment of the present invention.
  • FIG. 7 is an example an environment for the system of FIG. 4, in accordance with an embodiment of the present invention.
  • FIG. 8 is a simplified block diagram of a system for multi-tenant and multi-site cloud-based data services, in accordance with an embodiment of the present invention.
  • FIG. 9 is a simplified block diagram of a first configuration for providing hypervisor level multi-tenant cloud-based data services, in accordance with an embodiment of the present invention.
  • FIG. 10 is a simplified block diagram of a second configuration for providing hypervisor level multi-tenant cloud-based data services, in accordance with an embodiment of the present invention.
  • FIG. 11 is a simplified block diagram of a system for multi-tenant and multi-site cloud-based data services with jurisdictional data separation, in accordance with an embodiment of the present invention.
  • Appendix I is an application programming interface for virtual replication site controller web services, in accordance with an embodiment of the present invention.
  • Appendix II is an application programming interface for virtual replication host controller web services, in accordance with an embodiment of the present invention.
  • Appendix III is an application programming interface for virtual replication protection group controller web services, in accordance with an embodiment of the present invention.
  • Appendix IV is an application programming interface for virtual replication command tracker web services, in accordance with an embodiment of the present invention.
  • Appendix V is an application programming interface for virtual replication log collector web services, in accordance with an embodiment of the present invention.
  • VDSA virtual data services appliance
  • FIG. 1 is a simplified block diagram of a hypervisor architecture that includes a tapping driver and a VDSA, in accordance with an embodiment of the present invention.
  • Shown in FIG, 1 is a hypervisor 100 with three virtual servers 110, three virtual disks 120, an I/O backend 130 and a physical storage array 140.
  • Hypervisor 100 uses a single physical server, but runs multiple virtual servers 110.
  • Virtual disks 120 are a storage emulation layer that provide storage for virtual servers 110.
  • Virtual disks 120 are implemented by hypervisor 100 via I/O backend 130, which connects to physical disk 140.
  • Hypervisor 100 also includes a tapping driver 150 installed within the hypervisor kernel. As shown in FIG. 1, tapping driver 150 resides in a software layer between virtual servers 110 and virtual disks 120. As such, tapping driver 150 is able to access I/O requests performed by virtual servers 110 on virtual disks 120. Tapping driver 150 has visibility to I/O requests made by virtual servers 110.
  • Hypervisor 100 also includes a VDSA 160.
  • a VDSA 160 runs on a separate virtual server within each physical hypervisor.
  • VDSA 160 is a dedicated virtual server that provides data services via one or more data services engines 170.
  • VDSA 160 does not reside in the actual I/O data path between I/O backend 130 and physical disk 140. Instead, VDSA 160 resides in a virtual I/O data path.
  • tapping driver 150 identifies the I/O requests that the virtual server makes. Tapping driver 150 copies the I/O requests, forwards one copy via the conventional path to I/O backend 130, and forwards another copy to VDSA 160.
  • VDSA 160 enables the one or more data services engines 170 to provide data services based on these I/O requests.
  • FIG. 2 is a simplified data flow chart for a VDSA, in accordance with an embodiment of the present invention. Shown in FIG. 2 are an I/O receiver 210, a hash generator 220, a TCP transmitter 230, a data analyzer and reporter 240, a journal manager 250 and a remote VDSA 260.
  • Remote VDSA 260 resides on different physical hardware, at a possibly different location.
  • I/O receiver 210 receives an intercepted I/O request from tapping driver 150.
  • VDSA 160 makes up to three copies of the received I/O requests, in order to perform a set of actions which enable the one or more data services engines 170 to provide various services.
  • a first copy is stored in persistent storage, and used to provide continuous data protection.
  • VDSA 160 sends the first copy to journal manager 250, for storage in a dedicated virtual disk 270. Since all I/O requests are journaled on virtual disk 270, journal manager 250 provides recovery data services for virtual servers 110, such as restoring virtual servers 110 to an historical image.
  • hash generator 220 derives a one-way hash from the I/O requests. Use of a hash ensures that only a single copy of any I/O request data is stored on disk.
  • An optional second copy is used for disaster recovery. It is sent via TCP transmitter 230 to remote VDSA 260. As such, access to all data is ensured even when the production hardware is not available, thus enabling disaster recovery data services.
  • Data analyzer and reporter 240 analyzes data content of the I/O requests and infers information regarding the data state of virtual servers 110. E.g., data analyzer and reporter 240 may infer the operating system level and the status of a virtual server 110.
  • FIG. 3 is a simplified block diagram of a virtual replication system, in accordance with an embodiment of the present invention. Shown in FIG. 3 is a protected site designated Site A, and a recovery site designated Site B.
  • Site A includes a hypervisor 100A with three virtual servers llOA-1, 110A-2 and 110A- 3, and a VDSA 160A.
  • Site A includes two physical disks 140A-1 and 140A-2.
  • Site B includes a hypervisor 100B with a VDSA 160B.
  • Site B includes two physical disks 140B-1 and 140B-2. All or some of virtual servers llOA-l, 110A-2 and 110A-3 may be designated as protected. Once a virtual server is designated as protected, all changes made on the virtual server are replicated at the recovery site.
  • every write command from a protected virtual server in hypervisor 100A is intercepted by tapping driver 150 (FIG. 1) and sent asynchronously by VDSA 160A to VDSA 160B for replication, via a wide area network (WAN) 320, while the write command continues to be processed by the protected server.
  • tapping driver 150 FIG. 1
  • VDSA 160A VDSA 160B for replication
  • WAN wide area network
  • the write command is passed to a journal manager 250 (FIG. 2), for journaling on a Site B virtual disk 270 (FIG. 2).
  • a checkpoint is written to the Site B journal, and during a recovery one of the checkpoints may be selected for recovering to that point.
  • checkpoints may be manually added to the Site B journal by an administrator, along with a description of the checkpoint.
  • a checkpoint may be added immediately prior to an event taking place that may result in the need to perform a recovery, such as a planned switch over to an emergency generator.
  • mirrors llOB-1, 110B-2 and 110B-3 of the respective protected virtual servers llOA-1, 110A-2 and 110A-3 at Site A are created at Site B.
  • the mirrors at Site B are updated at each checkpoint, so that they are mirrors of the corresponding virtual servers at Site A at the point of the last checkpoint.
  • an administrator can specify that he wants to recover the virtual servers using the latest data sent from the Site A.
  • the administrator can specify an earlier checkpoint, in which case the mirrors on the virtual servers llOB-l, 110-B-2 and 110B-3 are rolled back to the earlier checkpoint, and then the virtual servers are recovered to Site B.
  • the administrator can recover the environment to the point before any corruption, such as a crash or a virus, occurred, and ignore the write commands in the journal that were corrupted.
  • VDSAs 160A and 160B ensure write order fidelity; i.e., data at Site B is maintained in the same sequence as it was written at Site A.
  • Write commands are kept in sequence by assigning a timestamp or a sequence number to each write at Site A. The write commands are sequenced at Site A, then transmitted to Site B asynchronously, then reordered at Site B to the proper time sequence, and then written to the Site B journal.
  • the journal file is cyclic; i.e., after a pre-designated time period, the earliest entries in the journal are overwritten by the newest entries.
  • the virtual replication appliance of the present invention operates at the hypervisor level, and thus obviates the need to consider physical disks.
  • conventional replication systems operate at the physical disk level.
  • Embodiments of the present invention recover write commands at the application level.
  • Conventional replication systems recover write commands at the SCSI level.
  • conventional replication systems are not fully application-aware, whereas embodiment of the present invention are full application- aware, and replicate write commands from an application in a consistent manner.
  • the present invention offers many advantages.
  • VDSA 160 manages recovery of virtual servers and virtual disks, it is not tied to specific hardware that is used at the protected site or at the recovery site.
  • the hardware may be from the same vendor, or from different vendors. As long as the storage device supports the iSCSI protocol, any storage device, known today or to be developed in the future, can be used.
  • VDSA 160 resides in the hypervisor level, architectures of the present invention scale to multiple sites having multiple hypervisors, as described hereinbelow with reference to FIG. 4.
  • Control of Recovery An administrator controls when a recovery is initiated, and to what point in time it recovers.
  • RPO Near-Zero Recovery Point Objective
  • RTO Near-Zero Recovery Time Objective
  • WAN Failover Resilience In accordance with an embodiment of the present invention, data is cached prior to being transmitted to the recovery site. If WAN
  • the present invention enables multiple enterprises to use shared hardware at a recovery site for data replication, while maintaining security between the enterprises, as described hereinbelow with reference to FIGS. 9 and 10.
  • FIG. 4 is a simplified block diagram of a cross-host multiple hypervisor system 300 that includes data services managers for multiple sites that have multiple hypervisors, in accordance with an embodiment of the present invention.
  • the architecture of FIG. 4 includes three sites, designated Site A, Site B and Site C, the three sites being communicatively coupled via a network 320.
  • Each site includes one or more hypervisors 100.
  • Site A includes three hypervisors, lOOA/1, lOOA/2 and 100A/3
  • Site B includes two hypervisors, lOOB/1 and 100B/2
  • Site C includes one hypervisor lOOC/1.
  • the sites have respective one or more physical disks 140A, 140B and 140C.
  • the hypervisors are shown in system 300 with their respective VDSA's 160A/1, 160A/2, ... , and the other components of the hypervisors, such as the virtual servers 110 and virtual disks 120, are not shown for the sake of clarity.
  • An example system with virtual servers 110 is shown in FIG. 7, and described hereinbelow.
  • the sites include respective data services managers 310A, 310B and 310C that coordinate hypervisors in the sites, and coordinate hypervisors across the sites.
  • the system of FIG. 4 may be used for data replication, whereby data at one site is replicated at one or more other sites, for protection.
  • the solid communication lines 330 in FIG. 4 are used for in-site traffic, the dashed communication lines 340 are used for replication traffic between sites, and the dotted communication lines 350 are used for control traffic between data services managers.
  • Data services managers 310A, 310B and 310C are control elements.
  • the data services managers at each site communicate with one another to coordinate state and instructions.
  • the data services managers track the hypervisors in the environment, and track health and status of the VDS As 160A/1, 160A/2, ....
  • FIG. 4 may be re-configured by moving one or more virtual servers 110 from one hypervisor 100 to another, by moving one or more virtual disks 120 from one hypervisor 100 to another, and by adding one or more additional virtual servers 110 to a hypervisor 100.
  • the data services managers enable designating groups of specific virtual servers 110, referred to as virtual protection groups, to be protected. For virtual protection groups, write order fidelity is maintained.
  • the data services managers enable designating a replication target for each virtual protection group; i.e., one or more sites, and one or more hypervisors in the one or more sites, at which the virtual protection group is replicated.
  • a virtual protection group may have more than one replication target.
  • the number of hypervisors and virtual servers within a virtual protection group and its replication target are not required to be the same.
  • FIG. 5, is a user interface screenshot of bidirectional replication of virtual protection groups, in accordance with an embodiment of the present invention. Shown in FIG. 4 are virtual protection groups 301 ("Exchange"), 302 ("WebApp"), 303 ("Dummy-Rl"), 304 ("Windows 2003”) and 305 (Dummies-L"). Arrows 306 indicate direction of replication.
  • FIG. 6 is a user interface screenshot of assignment of a replication target for a virtual protection group, in accordance with an embodiment of the present invention. Shown in FIG. 6 is an entry 307 for designating a recovery host, and an entry 308 for designating a recovery datastore for virtual protection group 304 ("Windows 2003") of FIG. 5. Respective source and target datastores, [SAN ZeRTO-30] 309A and [datastorel] 309B, are shown as being paired.
  • the recovery host may be assigned to a cluster, instead of to a single hypervisor, and the recovery datastore may be assigned to a pool of resources, instead of to a single datastore.
  • Such assignments are of particular advantage when different enterprises share the same physical infrastructure for target replication, as such assignments mask the virtual infrastructure between the different enterprises.
  • the data services managers synchronize site topology information.
  • a target site's hypervisors and datastores may be configured from a source site.
  • Virtual protection groups enable protection of applications that run on multiple virtual servers and disks as a single unit.
  • an application that runs on virtual servers many require a web server and a database, each of which run on a different virtual server than the virtual server that runs the application.
  • These virtual servers may be bundled together using a virtual protection group.
  • data services managers 310A, 310B and 310C monitor changes in the environment, and automatically update virtual protection group settings accordingly. Such changes in the environment include inter alia moving a virtual server 110 from one hypervisor 100 to another, moving a virtual disk 120 from one hypervisor 100 to another, and adding a virtual server 110 to a hypervisor 100.
  • each VDSA 160A/1, 160A/2, ... replicates IOs to its corresponding replication target.
  • the VDSA can replicate all virtual servers to the same hypervisor, or to different hypervisors.
  • Each VDSA maintains write order fidelity for the IOs passing through it, and the data services manager coordinates the writes among the VDSAs.
  • each virtual server 110 in a virtual protection group may be specified arbitrarily, all virtual servers 110 in the virtual protection group may be replicated at a single hypervisor, or at multiple hypervisors. Moreover, the virtual servers 110 in the source site may migrate across hosts during replication, and the data services manager tracks the migration and accounts for it seamlessly.
  • FIG. 7 is an example an environment for system 300, in accordance with an embodiment of the present invention.
  • system 300 includes the following components.
  • Hypervisor lOOA/l virtual servers llOA/1-1, llOA/1-2, llOA/1-3.
  • Hypervisor 100A/2 virtual servers 110A/2-1, 110A/2-2, 110A/2-3.
  • Hypervisor 100A/3 virtual servers 110A/3-1, llOA 3-2, 110A/3-3.
  • Hypervisor lOOB/l virtual servers llOB/1-1, llOB/1-2, llOB/1-3.
  • Hypervisor 100B/2 virtual servers 110B/2-1, 110B/2-2, 110B/2-3.
  • Hypervisor lOOC/l virtual servers llOC/1-1, llOC/1-2, llOC/1-3, llOC/1-4.
  • system 300 includes the following virtual protection groups. Each virtual protection group is shown with a different hatching, for clarity.
  • VPG1 (shown with upward-sloping hatching)
  • Source at Site A virtual servers llOA/1-1, 110A/2-1, 110A/3-1
  • Replication Target at Site B virtual servers llOB/1-1, llOB/1-2, 110B/2-1
  • VPG2 (shown with downward-sloping hatching)
  • Source at Site B virtual servers llOB/1-3, 110B/2-2
  • VPG4 (shown with vertical hatching)
  • Source at Site A virtual servers llOA/1-3, 110A/2-3, 110A/3-2
  • the hypervisor architecture of FIG. 1 scales to multiple host sites, each of which hosts multiple hypervisors.
  • the scaling flexibly allows for different numbers of hypervisors at different sites, and different numbers of virtual services and virtual disks within different hypervisors.
  • the scaling flexibility of the present invention also allows extension to cloud-based data services provided by a cloud provider on a shared infrastructure, as explained hereinbelow.
  • Cloud-based data services enable data center providers to service multiple enterprises at data centers that are remote from the enterprises.
  • Cloud-based data services offer many advantages. Enterprises that use cloud-based data services obviate the needs for servers, SAN/NAS, networks, communication lines, installation, configuration and ongoing maintenance of information technology systems, and overhead expenses for electricity, cooling and space.
  • conventional cloud- based data suffer from weakness of security due to multiple enterprises sharing the same physical infrastructure, and due to multiple enterprises using the same networks and IPs for their services.
  • FIG. 8 is a simplified block diagram of a system 500 for multi-tenant and multi-site cloud-based data services, in accordance with an embodiment of the present invention. Shown in FIG. 8 are three enterprises, designated A, B and C, and a remote cloud-based facility 480 with two data centers, designated 1 and 2.
  • Enterprise A uses a NETAPP ® data management system
  • Enterprise B uses an IBM data management system
  • Enterprise C uses an EMC data management system.
  • Data Center 1 uses an EMC 2® data management system and services Enterprise A.
  • Data Center 2 uses a NETAPP ® data management system and services Enterprises B and C.
  • System 500 has many advantages over conventional data service systems. Inter alia, system 500 enables protection of heterogenic environments, enables remote control of enterprise sites, enables economies of scale, enables complete workload mobility, enables a complete web services API for seamless integration, and enables integration with other cloud-based management systems.
  • FIG. 9 is a simplified block diagram of a first configuration of system 500 for providing hypervisor level multi-tenant cloud-based data services, in accordance with an embodiment of the present invention.
  • the configuration shown in FIG. 9 includes two enterprises; namely, a smaller enterprise A and a larger enterprise B.
  • Enterprise A infrastructure includes a single hypervisor lOOA/l with five virtual servers lOOA/1-1, lOOA/1-2, lOOA/1-3, lOOA/1-4 and lOOA/1- 5 and a VDSA 160A 1, includes two physical disks 120A-1 and 120A-2, and includes a data services manager 310A.
  • Enterprise B infrastructure includes two hypervisors lOOB/l and 100B/2, includes three physical disks 120B-1, 120B-2 and 120B-3, and a data services manager 310B.
  • Hypervisor lOOB/l includes five virtual servers lOOB/1-1, lOOB/1-2, lOOB/1-3, lOOB/1-4 and lOOB/1-5 and a VDSA 160B/1; and
  • hypervisor 100B/2 includes five virtual servers 100B/2-1, 100B/2-2, 100B/2-3, 100B/2-4 and 100B/2-5 and a VDSA 160B/2.
  • Cloud-based facility 480 infrastructure includes two hypervisors 400/1 and 400/2, and four physical disks 420-1, 420-2, 420-3 and 420-4.
  • Hypervisor 400/1 includes six virtual servers 410/1-1, 410/1-2, 410/1-3, 410/1-4, 410/1-5 and 410/1-6; and hypervisor 400/2 includes two virtual servers 410/2-1 and 410/2-2.
  • Hypervisor 400/1 services Enterprises A and B, and hypervisor 400/2 services Enterprise B. As such, the infrastructure of cloud-based facility 480 is shared between Enterprises A and B.
  • hypervisor 400/1 includes a VDSA 460/1 and hypervisor 400/2 includes a VDSA 460/2.
  • FIG. 10 is a simplified block diagram of a second configuration of system 500 for providing hypervisor level multi-tenant cloud- based data services, in accordance with an embodiment of the present invention.
  • the configuration of FIG. 10 allocates one data services manager per enterprise and one VDSA per enterprise per hypervisor.
  • hypervisor 400/1 includes a VDSA 460A/1 for Enterprise A and a VDSA 460B/1 for Enterprise B
  • hypervisor 400/2 includes a VDSA 460B/2 for Enterprise B.
  • FIGS. 9 and 10 offer different advantages.
  • the configuration of FIG. 9 minimizes the cloud footprint, by using only one data services manager per hypervisor, but offers less data path security for enterprises. Specifically, the infrastructure is shared between enterprises and security is enforced via software.
  • the configuration of FIG. 10 maximizes data path security for enterprises, but the cloud footprint is larger, since multiple data services managers may be required per hypervisor.
  • the cloud-based hypervisor level data services systems of the present invention enable multi-tenancy and multi-side services; i.e., multiple enterprises and multiple sites may be serviced by the same physical infrastructure including inter alia the same hypervisors and storage.
  • the systems of the present invention provide path separation between enterprises, thus ensuring that each enterprise is only able to view and access his own virtual servers and storage, and each enterprise can only control its own data replication.
  • the systems of the present invention may be installed behind network address translators (NATs), and do not require a virtual private network (VPN). As such, these systems may obviate VPN setups for enterprises, and obviate use of public IPs.
  • NATs network address translators
  • VPN virtual private network
  • the systems of the present invention provide bi-directional cloud-based data replication services; i.e., from an enterprise to the cloud, and from the cloud to an enterprise, for the same enterprise or for different enterprises, simultaneously using the same shared infrastructure.
  • replication targets may be set as resources that do not expose the enterprise infrastructure, thus providing an additional layer of security and privacy between enterprises.
  • cloud-based facility 480 infrastructure is partitioned according to jurisdictions, and data recovery and failover for an enterprise is limited to one or more specific partitions according to jurisdictional regulations.
  • FIG. 11 is a simplified block diagram of a system 600 for multi-tenant and multi-site cloud-based data services with jurisdictional data separation, in accordance with an embodiment of the present invention.
  • Shown in FIG. 11 are three jurisdictional enterprises; namely, a USA Enterprise A, a UK Enterprise B, and an APAC Enterprise C.
  • Also shown in FIG. 11 are four data centers; namely, Data Center 1 and Data Center 2 in the USA, Data Center 3 in the UK and Data Center 4 in APAC.
  • system 600 includes a rights manager 610 that blocks access to a data center by an enterprise if data export is regulations restrict data transfer between their respective jurisdictions.
  • rights manager 610 blocks access by Enterprise A to Data Centers 3 and 4, blocks access by Enterprise B to Data Centers 1,2 and 4, and blocks access by Enterprise C to Data Centers 1, 2, and 3.
  • Enterprises A, B and C may be commonly owned, but access of the data centers by the enterprises is nevertheless blocked, in order to comply with data export regulations.
  • an administrator when configuring a virtual protection group, an administrator may set a territory/data center restriction.
  • system 600 verifies that the resource is located in a geography that does not violate a territory/data center restriction.
  • the present invention may be implemented through an application programming interface (API), exposed as web service operations.
  • API application programming interface
  • Appendices I - V which define an API for virtual replication web services, in accordance with an embodiment of the present invention. It will thus be appreciated that the present invention provides many advantages, including inter alia:
  • heterogeneous hypervisor replication for different types of sources and target hypervisor; e.g., from a VMWare hypervisor to a Xen hypervisor;
  • heterogeneous storage replication for different types of storage systems; e.g., from an EMC storage system to a NetApp storage systems;
  • These web services include methods and properties for pairing and un-pairing sites, and for managing site details.
  • This property is a globally unique identifier for the peer site of a given site.
  • This property includes parameters to access a site, including
  • This property is a globally unique identifier for a site.
  • This property includes a name, location and contact information for a site, including
  • This property indicates the globally unique identifier for the local site.
  • This method retrieves the IP address and port of the site paired with a given site.
  • This method retrieves site details, including inter alia the IP address and port of a designated server.
  • This method retrieves the name, location and contact information specified by an administrator for a designated site.
  • This method retrieves the identifiers for a site and for its paired site.
  • This method retrieves the TCP port to access the virtual data services application for a designated site. • GetUsername
  • This method retrieves the username for a hypervisor. ⁇ GetIP
  • This method retrieves the IP address or a hostname for a hypervisor.
  • This method returns true if the local site is paired with another site. Otherwise, it returns false.
  • This method pairs a local site with another site.
  • This method reconfigures hypervisor information.
  • This method sets the TCP port used to access the virtual data services appliances at a designated site. • Unpair
  • This method un-pairs a local and remote site.
  • These web services include methods and properties to identify hypervisors, and to deploy virtual data services appliances on hypervisors.
  • This property identifies a hypervisor, and includes
  • Serverldentifier serverldentifier - the globally unique identifier for the hypervisor.
  • This method retrieves a list of hypervisors where a virtual data services appliance is in the process of being installed, at a designated site.
  • This method retrieves a list of hypervisors where a virtual data services appliance is in the process of being un-deployed, at a designated site. • GetlnstalledHosts
  • This method retrieves a list of hypervisors where a virtual data services appliance is installed, at a designated site.
  • This method retrieves a list of hypervisors where a virtual data services appliance is not installed, at a designated site.
  • Hostldentifier hostldentifier Hostldentifier
  • This method deploys a virtual data services appliance on a specified hypervisor at a designated site, in accordance with a specified datastore, a specified type of network, and access details including inter alia an IP a subnet mask and gateway for the VDSA.
  • Hostldentifier hostldentifier )
  • This method un-deploys a virtual data services appliance from a specified hypervisor, at a designated site.
  • These web services include methods and properties to manage virtual protection groups.
  • This property identifies a checkpoint by an unsigned integer.
  • This property includes information returned from a failover test, including
  • DateTime EndTime The time the test ended, represented as a date and time of day;
  • Teststatus status A status of the test, including Unknown, FailedBySystem, AbortedByFailoverCommand,
  • This property defines settings for a protection group, including
  • This property defines settings for a virtual protection group, including string Name - a name used to identify the virtual protection group;
  • ProtectionGroupDefaults Defaults defaults used to identify the recovery site, including the IP address of the hypervisor host, the name of the datastore, and the networks to use for tests and for failovers
  • ProtectionGroupPriority Priority the priority assigned to the virtual protection group; namely, LOW, Medium or
  • int Remoteid the identifier assigned to the peer site; and int MaxTestintervaiinMinutes - the maximum time, in minutes, between tests.
  • This property indicates the status of a virtual protection group, from among Protecting, NeedReverseConfiguration, Promoting,
  • This property indicates settings for a virtual application, including
  • Serverldentifier serverldentifier - a globally unique identifier for the virtual application
  • Serverldentifier serverldentifier - a globally unique identifier tor the virtual application replication destination
  • Type the type of server hosting the virtual application at the destination; namely, host(0) or cluster(l). • VSManagementSettings
  • This property indicates settings for a virtual server, including
  • Serverldentifier serverldentifier - 3 globally unique identifier for the virtual server
  • ReplicationDestination the destination name, identifier and type (host / cluster) and the datastore name and identifier;
  • vsDestinationFoiders - a failover destination folder and a failover test destination folder, including a name and an identifier
  • VolumeManagementSettings settings that identify a volume, such as the directory, path and name of the disk and type of disk, such as a raw drive, and whether or not it is a swap disk.
  • This method removes the virtual protection groups defined at a
  • This method creates a virtual protection group at a designated site. ⁇ FailOver
  • This method performs a failover of the virtual servers in a designated virtual protection group, to a designated checkpoint instance or to the latest checkpoint.
  • This method performs a failover of the virtual servers in a designated virtual protection group, to a designated checkpoint or to the latest checkpoint, without creating reverse replication and without stopping protection of the virtual servers in the designated virtual protection group.
  • This method removes a virtual protection group irrespective of the state of the group. This method is used if the
  • RemoveProtectionGroup method is unable to complete successfully.
  • This method updates virtual protection group settings, including removal of virtual servers and disks that should have been removed using the RemoveProtectionGroup method. This method is used if the updateProtectionGroup method is unable to complete
  • This method retrieves a list of checkpoints for a specified virtual protection group.
  • This method retrieves information about failover tests for a specified virtual protection group. ⁇ GetProtectionGroupDefaultReverseSettings
  • This method retrieves the virtual protection group settings for a specified virtual protection group, for use as default values for reverse replication.
  • This method retrieves the settings for a designated virtual protection group. • GetProtectionGroups
  • This method retrieves a list of virtual protection groups.
  • This method retrieves the state of a specified virtual protection group, the state being "protected” or "recovered”. If the group is protected, 0 is returned; and if the group is recovered, 1 is returned.
  • This method retrieves the status of a specified virtual protection group, the status being inter alia "protecting", "testing” or
  • This method inserts a named checkpoint for a designated virtual protection group. The method returns immediately, without verifying whether or not the checkpoint was successfully written to the journal in the peer site.
  • This method migrates a specified virtual protection group to the peer site.
  • This method adds a designated virtual server to a virtual protection group, in accordance with designated settings.
  • This method removes a virtual protection group, unless the group is being replicated during a test failover or an actual failover, and unless the group is being migrated to the peer site. If this method does not return a success completion code, the
  • ForceRemoveProtectionGroup method may be used to force removal of the group.
  • This method discards information about a specified number of old failover tests for a designated virtual protection group, from the oldest test to the most recent test.
  • This method removes a designated virtual server from a designated virtual protection group.
  • This method updates settings of a specified virtual protection group. If the method does not return a success completion code, the
  • ForceupdateProtectionGroup method can be used to force the update.
  • This method waits for a checkpoint to be written to a journal on the peer site, after it was inserted, or times out if it takes too long.
  • These web services include methods and properties to monitor procedures being executed.
  • This property includes.
  • enum compietionCode the result of running a method, including NotAvailable, Success, Aborted, Failed and
  • int OriginaiinputParms input parameters
  • int pgid the globally unique identifier for a virtual protection group.
  • This method retrieves a list of all tasks that are currently active.
  • This method returns the completion code of a specified task.
  • Completion codes include Success, Aborted, Failed or
  • This method returns the command type, the completion code input parameters, and the virtual protection group identifier of a
  • This method returns the string associated with an exception, for a designated task that had an exception.
  • Getcompietioncode returns HadException if a task had an
  • This method returns progress as a percentage of a whole task, as an integer, for a specified task.
  • This method returns the identifier of a task currently being performed on a designated protection group.
  • ProtectionGroupIdentifier pgld This method returns the identifier of a task currently being performed on a specified protection group at a local site.
  • the returned result may be one of the following :
  • This method retrieves the current status of a specified task.
  • Status may be Active, Running, Aborted ⁇ Completed.
  • This method waits for a specified task to complete, by polling the task at specified time intervals, until a specified time out.
  • These web services include methods and properties to retrieve information for troubleshooting.
  • This property indicates details of a log request, including a level of detail of information, indicating whether information about a virtual data service appliance and core information should be included, and including start and end times for the information.
  • This method initiates a log request.
  • This method retrieves results of a log request.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Software Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

L'invention porte sur un système pour des services de données dans le nuage pour de multiples entreprises, comprenant une pluralité d'hyperviseurs dans le nuage qui fournissent d'une manière coopérative des services dans le nuage à de multiples entreprises, chaque hyperviseur comprenant une pluralité de serveurs virtuels dans le nuage, chaque serveur virtuel dans le nuage étant associé à une entreprise, au moins un disque virtuel dans le nuage qui est lu et écrit par l'au moins un serveur virtuel, chaque disque virtuel dans le nuage étant associé à une entreprise, et un appareil de services de données virtuel, qui fournit des services de données dans le nuage, et de multiples gestionnaires de services de données, un gestionnaire de services de données par entreprise respective, dont chacun coordonne les appareils de services de données virtuels respectifs pour les hyperviseurs dans le nuage qui desservent son entreprise correspondante.
PCT/IL2012/000271 2011-07-04 2012-07-03 Procédé et appareil de fourniture de services de données de niveau hyperviseur pour virtualisation de serveur WO2013005208A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US13/175,892 2011-07-04
US13/175,898 2011-07-04
US13/175,892 US11256529B2 (en) 2010-03-17 2011-07-04 Methods and apparatus for providing hypervisor level data services for server virtualization
US13/175,898 US9710294B2 (en) 2010-03-17 2011-07-04 Methods and apparatus for providing hypervisor level data services for server virtualization

Publications (1)

Publication Number Publication Date
WO2013005208A1 true WO2013005208A1 (fr) 2013-01-10

Family

ID=47436612

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IL2012/000271 WO2013005208A1 (fr) 2011-07-04 2012-07-03 Procédé et appareil de fourniture de services de données de niveau hyperviseur pour virtualisation de serveur

Country Status (1)

Country Link
WO (1) WO2013005208A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11038926B2 (en) * 2019-01-23 2021-06-15 Vmware, Inc. System and method for embedding infrastructure security services into management nodes

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040068561A1 (en) * 2002-10-07 2004-04-08 Hitachi, Ltd. Method for managing a network including a storage system
US20080195624A1 (en) * 2007-02-13 2008-08-14 Senthilkumar Ponnappan System and method for administration of virtual servers
US7475207B2 (en) * 2005-07-14 2009-01-06 Emc Corporation Maintaining write order fidelity on a multi-writer system
US20100017801A1 (en) * 2008-07-18 2010-01-21 Vmware, Inc. Profile based creation of virtual machines in a virtualization environment
US20100198972A1 (en) * 2009-02-04 2010-08-05 Steven Michael Umbehocker Methods and Systems for Automated Management of Virtual Resources In A Cloud Computing Environment
US20110022812A1 (en) * 2009-05-01 2011-01-27 Van Der Linden Rob Systems and methods for establishing a cloud bridge between virtual storage resources

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040068561A1 (en) * 2002-10-07 2004-04-08 Hitachi, Ltd. Method for managing a network including a storage system
US7475207B2 (en) * 2005-07-14 2009-01-06 Emc Corporation Maintaining write order fidelity on a multi-writer system
US20080195624A1 (en) * 2007-02-13 2008-08-14 Senthilkumar Ponnappan System and method for administration of virtual servers
US20100017801A1 (en) * 2008-07-18 2010-01-21 Vmware, Inc. Profile based creation of virtual machines in a virtualization environment
US20100198972A1 (en) * 2009-02-04 2010-08-05 Steven Michael Umbehocker Methods and Systems for Automated Management of Virtual Resources In A Cloud Computing Environment
US20110022812A1 (en) * 2009-05-01 2011-01-27 Van Der Linden Rob Systems and methods for establishing a cloud bridge between virtual storage resources

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11038926B2 (en) * 2019-01-23 2021-06-15 Vmware, Inc. System and method for embedding infrastructure security services into management nodes

Similar Documents

Publication Publication Date Title
US9710294B2 (en) Methods and apparatus for providing hypervisor level data services for server virtualization
US11650842B2 (en) Methods and apparatus for providing hypervisor level data services for server virtualization
US9389892B2 (en) Multiple points in time disk images for disaster recovery
US9442748B2 (en) Multi-RPO data protection
US11797395B2 (en) Application migration between environments
US11579991B2 (en) Dynamic allocation of compute resources at a recovery site
US9652326B1 (en) Instance migration for rapid recovery from correlated failures
US8996477B2 (en) Multiple node/virtual input/output (I/O) server (VIOS) failure recovery in clustered partition mobility
JP5026305B2 (ja) 視覚化および地理的分散データセンタ用記憶装置およびサーバプロビジョニング
US9619256B1 (en) Multi site and multi tenancy
US9575851B1 (en) Volume hot migration
US9575857B1 (en) Active/active replication
US9450700B1 (en) Efficient network fleet monitoring
US20120110274A1 (en) Operating System Image Management
US10649855B2 (en) Network configuration method to allow access to the backup and restores to Mtrees on a clustered backup appliance
CN110912991A (zh) 一种基于超融合双节点高可用的实现方法
US9229850B1 (en) Mapping data storage and virtual machines
US20120151095A1 (en) Enforcing logical unit (lu) persistent reservations upon a shared virtual storage device
US20220398163A1 (en) High frequency snapshot technique for improving data replication in disaster recovery environment
CN113849136B (zh) 一种基于国产平台的自动化fc块存储处理方法和系统
WO2013005208A1 (fr) Procédé et appareil de fourniture de services de données de niveau hyperviseur pour virtualisation de serveur
US12093151B2 (en) Managing reference snapshots across multiple sites for efficient failover/failback
US20240143462A1 (en) Monitoring input/output and persistent reservation activity patterns to detect degraded performance of a high availability and fault tolerant application
US20240036988A1 (en) Disaster recovery pipeline for block storage and dependent applications
US20200104218A1 (en) Method and system for reliably restoring virtual machines

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12808203

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12808203

Country of ref document: EP

Kind code of ref document: A1