EP3326062B1 - Attenuation de l'impact de la non-disponibilite intermittente de stockage distant sur des machines virtuelles - Google Patents

Attenuation de l'impact de la non-disponibilite intermittente de stockage distant sur des machines virtuelles Download PDF

Info

Publication number
EP3326062B1
EP3326062B1 EP16745952.8A EP16745952A EP3326062B1 EP 3326062 B1 EP3326062 B1 EP 3326062B1 EP 16745952 A EP16745952 A EP 16745952A EP 3326062 B1 EP3326062 B1 EP 3326062B1
Authority
EP
European Patent Office
Prior art keywords
virtual machine
remote storage
operable
unavailability condition
hosting environment
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP16745952.8A
Other languages
German (de)
English (en)
Other versions
EP3326062A1 (fr
Inventor
Vivek P. Divakara
Suraj Puri
Raghu Murthy
Kai Chen
Francis M. DAVID
Melur Raghuraman
Mian Naeem UL HAQUE
Saad Syed
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Publication of EP3326062A1 publication Critical patent/EP3326062A1/fr
Application granted granted Critical
Publication of EP3326062B1 publication Critical patent/EP3326062B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45575—Starting, stopping, suspending or resuming virtual machine instances
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45579—I/O management, e.g. providing access to device drivers or storage
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45583—Memory management, e.g. access or allocation

Definitions

  • a virtual machine is an operating system or application environment that is supported by software that imitates dedicated hardware.
  • the end user has essentially a similar experience on a virtual machine as they would have on dedicated hardware.
  • Specialized software (sometimes referred to as a hypervisor or a virtual machine monitor) emulates hardware resources of the physical machine upon which the virtual machine is installed, such as a central processing unit, system memory, hard disk, and network interface. This emulation enables multiple virtual machines to share the same hardware resources.
  • the specialized software may be able to emulate multiple virtual hardware platforms that are isolated from each other, allowing virtual machines to run different server operating systems on the same underlying physical host. Virtualization can save costs by reducing the need for physical hardware systems. Virtual machines can use hardware more efficiently, which lowers the quantity of hardware and associated maintenance costs, and reduces power and cooling demand. Administrators can take advantage of virtual environments to simplify backups, disaster recovery, new deployments and basic system administration tasks.
  • Virtual machines have become a key technology in realizing an "infrastructure-as-a-service" model in which computational capacity, in the form of virtual machines deployed in a resource provider's datacenter, is provisioned on-demand as a service.
  • the information technology (IT) staff can spin up new virtual machines based on user demand or a new project.
  • the use of virtual machines in this context offers multiple benefits, including the ability to securely partition physical servers and to provide service recipients with customized software environments.
  • a virtual machine may be executed in an infrastructure in which a storage device is used to provide storage (e.g., disk storage) services to the virtual machine.
  • the storage device may be "remote" in that it is connected to the physical machine upon which the virtual machine is running via a network. If the remote storage device experiences intermittent unavailability issues (e.g., due to issues with the network and/or the remote storage device itself), then the virtual machine may encounter input/output (I/O) errors or unacceptably high I/O latencies. This can result in undesired behavior by software executing within the virtual machine that could lead to application as well as operating system crashes.
  • I/O input/output
  • a technique for data de-duplication is applied to suspend and resume operations for virtual machines (VMs) to reduce the time needed to suspend or resume a virtual machine and store the suspended VMs in a space-efficient and time-efficient manner is known.
  • a VM's state is serialized and written to disk in the form of key values that are used to retrieve data from a key-data map.
  • the key-data map stores unique data blocks for suspended VM images.
  • the key-data map can be stored in a solid-state drive to further reduce latency and access time.
  • US 2011/055372 A1 suggests a method where a host controller or a host detects that a client has disconnected from a virtual machine running on the host. The host controller or host initiates a timer upon detecting that the client has disconnected. When the timer times out, the host controller or host saves the state of the virtual machine. The host controller or host causes the virtual machine to be shut down upon saving the state.
  • a virtual machine hosting environment executing on a physical machine is operable to instantiate a virtual machine for execution on the physical machine.
  • the virtual machine is cable of initiating I/O operations that are directed to remote storage to which the physical machine is communicatively connected.
  • the virtual machine hosting environment is further operable to determine that an unavailability condition exists with respect to the remote storage and to suspend the virtual machine in response to at least the determination that the unavailability condition exists.
  • the virtual machine hosting environment may be further operable to determine that the unavailability condition no longer exists and to resume the virtual machine in response to at least the determination that the unavailability condition no longer exists.
  • the virtual machine hosting environment may be still further operable to save one or more pending I/O requests generated by the virtual machine in conjunction with pausing the virtual machine and to send the one or more pending I/O requests to the remote storage in conjunction with resuming the virtual machine.
  • the virtual machine hosting environment may be operable to terminate the virtual machine in response to at least the determination that the unavailability condition exists and to restart the virtual machine in response to at least the determination that the unavailability condition no longer exists.
  • a virtual machine may be executed in an infrastructure in which a remote storage device is used to provide storage (e.g., disk storage) services to the virtual machine.
  • storage services may be provided to the virtual machine by a remote storage device that is connected via a network to the physical machine upon which the virtual machine is running. If the remote storage device experiences intermittent unavailability issues (e.g., due to issues with the network and/or the remote storage device itself), then the virtual machine may encounter I/O errors or unacceptably high I/O latencies. This can result in undesired behavior by software executing within the virtual machine that could lead to application as well as operating system crashes.
  • a virtual machine hosting environment is described herein that can mitigate the impact of such intermittent storage unavailability on one or more virtual machines executing within the hosting environment.
  • a virtual machine hosting environment is described herein that executes on a physical machine.
  • the virtual machine hosting environment is operable to instantiate a virtual machine for execution on the physical machine.
  • the virtual machine is capable of initiating I/O operations directed to a remote storage to which the physical machine is communicatively connected.
  • the virtual machine hosting environment is further operable to determine that an unavailability condition exists with respect to the remote storage and to suspend the virtual machine in response to at least the determination that the unavailability condition exists.
  • the virtual machine hosting environment may be further operable to determine that the unavailability condition no longer exists and to resume the virtual machine in response to at least the determination that the unavailability condition no longer exists.
  • the virtual machine hosting environment may be still further operable to save one or more pending I/O requests generated by the virtual machine in conjunction with pausing the virtual machine and to send the one or more pending I/O requests to the remote storage in conjunction with resuming the virtual machine.
  • embodiments described herein can advantageously help prevent software executing within the virtual machine from exhibiting undesired behavior that may be brought on by I/O errors or I/O latency and that can lead to application or operating system crashes.
  • the virtual machine hosting environment resumes the virtual machine, the virtual machine may experience a jump in "wall clock" time and its network connection may have been timed out by its peers, but the software and data inside the virtual machine are preserved and available for continued processing. Any pending I/O requests that were saved by the virtual machine hosting environment when the virtual machine was suspended can be resolved by accessing the remote storage after the virtual machine is resumed.
  • the virtual machine hosting environment may determine that the unavailability condition exists with respect to the remote storage by, for example, monitoring I/O operations initiated by the virtual machine and/or other virtual machines executing on the physical machine.
  • the virtual machine hosting environment may also determine that the unavailability condition exists based on I/O status information received from one or more other physical machines.
  • the virtual machine hosting environment may determine that the unavailability condition no longer exists by intermittently probing the remote storage in a lightweight (e.g., relatively infrequent) fashion until a response is received therefrom that indicates that the unavailability condition no longer exists.
  • the virtual machine hosting environment may also determine that the unavailability condition no longer exists by intermittently retrying I/O operations directed to the remote storage until those I/O operations succeed.
  • the virtual machine hosting environment may determine that the unavailability condition no longer exists based on I/O status information received from one or more other physical machines.
  • the virtual machine hosting environment may be operable to terminate the virtual machine in response to at least the determination that the unavailability condition exists and to restart the virtual machine in response to at least the determination that the unavailability condition no longer exists.
  • Such an approach may be utilized, for example, where an application or operating system running on a virtual machine does not respond well to the jump in "wall clock" time that occurs when a virtual machine is suspended and resumed. In such a case, it may be deemed preferable to terminate the virtual machine during a period of remote storage unavailability and restart the virtual machine after such period ends, even if such termination may result in the crashing of an application or operating system running on the virtual machine.
  • policies for dealing with intermittent remote storage unavailability may be selectively applied to different virtual machines depending on the applications and/or operating systems executed by the virtual machines, based on user/customer preference, or based on some other criteria.
  • Section II describes an example system that includes a virtual machine hosting environment that is operable to mitigate the impact of intermittent storage unavailability on virtual machines hosted thereby.
  • Section II further describes exemplary methods and computer program products for mitigating the impact of intermittent storage unavailability on virtual machines.
  • Section III describes an example processor-based computer system that may be used to implement various embodiments.
  • Section IV describes some additional exemplary embodiments.
  • Section V provides some concluding remarks.
  • FIG. 1 is a block diagram of a system 100 that includes a virtual machine hosting environment that is operable to mitigate the impact of intermittent storage unavailability on virtual machines hosted thereby.
  • system 100 includes a physical machine 102 and a remote storage 104 that is connected thereto via one or more networks 106.
  • Physical machine 102 represents a hardware-based device and may comprise, for example, a hardware-based computing device.
  • the term "device” refers to a machine or other tangible, manufactured object and excludes software and signals.
  • Physical machine 102 includes a plurality of hardware components 112.
  • Hardware components 112 include, but are by no means limited to, one or more processors 130, a system memory 132, and a network interface 134.
  • Hardware components 112 may include additional components that are not shown in FIG. 1 . Such additional components may include, for example and without limitation, a hard disk drive, a magnetic disk drive, an optical disk drive, a serial port, a display, or a user input device.
  • Processor(s) 130 may comprise one or more microprocessors, each of which may comprise one or more central processing units (CPUs) or microprocessor cores. Processor(s) 130 may be implemented using other types of integrated circuits as well. Processor(s) 130 operate in a well-known manner to execute computer programs (also referred to herein as computer program logic, software, and software components). The execution of such computer programs causes processor(s) 130 to perform operations including operations that will be described herein. Each of system memory 132, network interface 134, and various other hardware components 112 of physical machine 102 is connected to processor(s) 130 via one or more suitable interfaces.
  • System memory 132 comprises one or more computer-readable memory devices that operate to store computer programs and data.
  • the computer programs stored by system memory 132 may be executed by processor(s) 130 in a well-known manner.
  • System memory 132 may be implemented using any of a wide variety of computer-readable memory devices including, but not limited to, random access memory (RAM) devices.
  • Computer programs and data may be loaded into system memory 132, for example, from a hard drive or other memory device within hardware components 112 or from remote storage 104.
  • Network interface 134 comprises an interface that enables physical machine 102 102 to communicate over network(s) 106.
  • network interface 134 may comprise a wired network interface such as an Ethernet interface.
  • network interface 134 may comprise a wireless network interface such as an IEEE 802.11 ("Wi-Fi") interface or a 3G telecommunication interface.
  • Wi-Fi IEEE 802.11
  • 3G telecommunication interface 3G telecommunication interface
  • a number of software components are executed by physical machine 102.
  • these software components are executed by processor(s) 130.
  • processor(s) 130 During execution, some of these software components may utilize and/or interact with various ones of hardware components 112 of physical machine 102.
  • these software components include but are not limited to a virtual machine hosting environment and one or more virtual machines that are hosted by such hosting environment.
  • the software components of physical machine 102 that may be said to comprise the virtual machine hosting environment include a hypervisor 114, a host operating system 140, and a virtualization stack 150. Each of these components is loaded into system memory 132 and executed therefrom by processor(s) 130.
  • Hypervisor 114 comprises a software component that acts as an interface between one or more operating systems and hardware components 112 of physical machine 102. Hypervisor 114 operates to provide isolated execution environments which may be referred to as partitions. In the example shown in FIG. 1 , hypervisor 114 has created a parent (or root) partition 116, a first child partition 118, and a second child partition 120. Hypervisor 114 presents emulated representations of hardware components 112 to each of the partitions and controls and arbitrates access to hardware components 112 on behalf of software components executing within the partitions.
  • Parent partition 116 is a partition within which a host operating system 140 of physical machine 102 is executed. Generally speaking, parent partition 116 manages machine-level functions such as device drivers 146, power management, and device hot addition/removal. Parent partition 116 is the only partition that has direct access to system memory 132 and the other devices within hardware components 112.
  • host operating system 140 includes a host operating system kernel 142, a virtualization service provider (VSP) 144, a plurality of device drivers 146 and a virtual machine (VM) bus 148, all of which operate within a kernel mode of parent partition 116.
  • Host operating system kernel 142 includes components of host operating system 140 that interact with hardware components 112 via device drivers 146.
  • VSP 144 is operable to provide synthetic device support to child partitions that include virtualization service clients (VSCs) over VM bus 148.
  • VM bus 148 comprises a channel-based point-to-point in-memory bus architecture, running fully in kernel mode.
  • host operating system 140 comprises a server operating system, such as a version of a MICROSOFT® WINDOWS SERVER® that is operating in a Hyper-V role, although this example is by no means limiting.
  • Virtualization stack 150 comprises software components running in a user mode within parent partition 116. Such components may be said to be running "on top of' host operating system 140. As shown in FIG. 1 , virtualization stack 150 includes a virtual machine manager 152 and virtual machine worker processes 156.
  • Virtual machine manager 152 is operable to manage all the virtual machines executing in the child partitions on physical machine 102. Operations that may be performed by virtual machine manager 152 include, but are not limited to creating or instantiating a virtual machine, terminating or destroying a virtual machine, pausing or suspending a virtual machine, resuming a virtual machine, obtaining a list of running virtual machines, obtaining performance and utilization statistics for a virtual machine, or the like. Although not shown in FIG. 1 , virtual machine manager 152 may comprise a user interface via which a user can invoke the functionality thereof.
  • Each one of virtual machine worker processes 156 provides virtual machine management services from parent partition 116 to a corresponding guest operating system running in a child partition.
  • Virtual machine manager 152 spawns a separate virtual machine worker process for each running virtual machine.
  • virtual machine manager 152 has instantiated a first virtual machine which is represented by first child partition 118 and a second virtual machine which is represented by second child partition 120.
  • first virtual machine which is represented by first child partition 118
  • second virtual machine which is represented by second child partition 120.
  • the virtual machine hosting environment shown in FIG. 1 may be used to concurrently host more than two virtual machines as well as only one virtual machine.
  • a guest operating system 160 is running within a kernel mode of the virtual machine of first child partition 118.
  • Guest operating system 160 includes a guest operating system kernel 162, a virtualization service client (VSC) 164, and VM bus 148.
  • Guest operating system kernel 162 comprises a core component of guest operating system 162.
  • VSC 164 is a synthetic device instance that resides in first child partition 118.
  • VSC 164 utilizes hardware resources that are provided thereto by VSP 144 in parent partition 116.
  • VSC 164 communicates with VSP 144 over VM bus 148.
  • One or more applications 170 are running within a user mode of the virtual machine of first child partition 118. Such application(s) may be said to run "on top of' guest operating system 160.
  • a guest operating system 180 is running within a kernel mode of the virtual machine of second child partition 120.
  • Guest operating system 180 does not include a VSC, so requests for hardware resources generated by guest operating system 180 (for itself or on behalf of application(s) 190) are handled by hypervisor 114.
  • One or more applications 190 are running within a user mode of the virtual machine of second child partition 120. Such application(s) may be said to run "on top of' guest operating system 180.
  • remote storage 104 is utilized to provide non-volatile storage (e.g., disk storage) on behalf of each of the virtual machines running on physical machine 102.
  • Remote storage 104 may comprise any device, system or service that is capable of providing non-volatile (e.g., disk type) storage remotely.
  • remote storage 104 comprises a cloud-based storage service such as MICROSOFT® AZURE® STORAGE, AMAZON® ELASTIC BLOCK STORAGE (EBS), AMAZON® SIMPLE STORAGE SERVICE (S3), or the like.
  • remote storage 104 comprises a storage device such as a network attached storage device, disk array, tape library, optical or magnetic disk drive, or the like.
  • remote storage 104 comprises a storage area network.
  • Networks(s) 106 that connect physical machine 102 to remote storage 104 is intended to represent any type of network or combination of networks suitable for facilitating communication between devices, such as between physical machine 102 and one or more devices that comprise remote storage 104.
  • Network(s) 106 may include, for example and without limitation, a local area network, a wide area network, a private network, a public network, a packet network, a circuit-switched network, a wired network, and/or a wireless network.
  • Host operating system 140 includes the device driver(s) necessary for directing I/O requests via network interface 134 to remote storage 104.
  • Such device driver(s) comprise part of device drivers 146.
  • an application e.g., one of application(s) 170 or one of application(s) 190
  • guest operating system e.g., guest operating system 160 or guest operating system 180
  • VSC 164 in the case of the virtual machine of first child partition 118
  • hypervisor 114 in the case of the virtual machine of second child partition 120
  • I/O stack 149 for servicing by the appropriate device driver(s).
  • Host operating system 140 will maintain such I/O requests in I/O stack 149 until such I/O requests are resolved.
  • An I/O request may be resolved, for example, when a successful read/write occurs at remote storage 104 and an I/O response indicating the same is sent back from remote storage 104.
  • Such I/O response will be forwarded by host operating system 140 to the virtual machine that generated the corresponding I/O request.
  • remote storage 104 may become temporarily unavailable. This may occur, for example, due to issues with network(s) 106 or with remote storage 104.
  • the virtual machines of first child partition 118 and second child partition 120 may encounter I/O errors or unacceptably high I/O latencies.
  • remote storage 104 will deliver an I/O error message to host operating system 140 that indicates that an I/O request pending in I/O stack 149 cannot be serviced.
  • I/O requests will remain pending on I/O stack 149 for a relatively long amount of time due to a delayed I/O response from remote storage 104.
  • either of these situations can result in undesired behavior by software executing within each virtual machine that could lead to application as well as operating system crashes.
  • virtualization stack 150 executing on physical machine 102 includes a virtual machine I/O monitor 154 as shown in FIG. 1 .
  • Virtual machine I/O monitor 154 is operable to determine when an unavailability condition exists with respect to remote storage 104 and to notify virtual machine manager 152 of the same.
  • Virtual machine manager 152 is operable to suspend each virtual machine executing on physical machine 102 that relies on remote storage 104 in response to at least receiving such notification.
  • Virtual machine manager 152 is also operable to save one or more pending I/O requests (e.g., I/O requests pending in I/O stack 149) generated by each such virtual machine in conjunction with suspending the virtual machines.
  • Virtual machine I/O monitor 154 is further operable to determine when the unavailability condition no longer exists (i.e., to determine when remote storage 104 has become available again) and to notify virtual machine manager 152 of the same. Virtual machine manager 152 is further operable to resume the suspended virtual machines in response to receiving at least such notification. Virtual machine manager 152 is also operable to send the saved pending I/O requests to remote storage 104 in conjunction with resuming the virtual machines.
  • the virtual machine hosting environment of FIG. 1 can advantageously help prevent software executing within these virtual machines from exhibiting undesired behavior that may be brought on by I/O errors or I/O latency and that can lead to application or operating system crashes.
  • the virtual machine hosting environment of FIG. 1 resumes each of the virtual machines of first child partition 118 and second child partition 120, each such virtual machine may experience a jump in "wall clock" time and its network connection may have been timed out by its peers, but the software and data inside the virtual machine are preserved and available for continued processing.
  • any pending I/O requests that were saved by the virtual machine hosting environment of FIG. 1 when the virtual machine was suspended can be resolved by accessing remote storage 104 after the virtual machine is resumed.
  • the virtual machine hosting environment described above in reference to FIG. 1 is merely one example of how a virtual machine hosting environment may be structured.
  • the techniques described herein for mitigating the impact of intermittent storage unavailability are not limited to the embodiment shown in FIG. 1 and are generally applicable to any type of virtual machine hosting environment.
  • FIG. 2 depicts a flowchart 200 of a method for mitigating the impact of intermittent storage unavailability on virtual machines.
  • the method of flowchart 200 will be described with continued reference to system 100 of FIG. 1 .
  • the method is not limited to that embodiment.
  • the method of flowchart 200 may be performed by any of a wide variety of virtual machine hosting environments other than the particular one shown in FIG. 1 .
  • the method of flowchart 200 begins at step 202 in which a virtual machine hosting environment executing on a physical machine instantiates a virtual machine for execution on the physical machine.
  • the virtual machine is capable of initiating I/O operations directed to a remote storage to which the physical machine is communicatively connected.
  • this step may be performed when virtual machine manager 152 executing on physical machine 102 instantiates the virtual machine of first child partition 118 for execution on physical machine 102.
  • the virtual machine of first child partition 118 is capable of initiating I/O operations that will ultimately be directed to remote storage 104 to which physical machine 102 is communicatively connected via network(s) 106.
  • the virtual machine hosting environment determines that an unavailability condition exists with respect to the remote storage.
  • this step may be performed when virtual machine I/O monitor 154 executing on physical machine 102 determines that an unavailability condition exists with respect to remote storage 104.
  • the term "unavailability condition" encompasses a condition or state in which one or more I/O operations directed to the remote storage have been delayed or failed as well as a condition or state in which it is anticipated that one or more I/O operations directed to the remote storage will be delayed or fail.
  • the virtual machine hosting environment suspends the virtual machine and saves any pending I/O operations directed to the remote storage in response to at least the determination that the unavailability condition exists with respect to the remote storage.
  • this step may be performed when virtual machine manager 152 executing on physical machine 102 performs the following operations in response to at least receiving a notification from virtual machine I/O monitor 154 that an unavailability condition exists with respect to remote storage 104: (1) suspend the virtual machine of child partition 118 and (2) save any pending I/O operations in I/O stack 149 that were generated on behalf of the virtual machine of child partition 118 and are directed to remote storage 104.
  • suspending a virtual machine may entail prohibiting one or more virtual processors within the virtual machine from being scheduled on the physical processors of the physical machine. This disables the virtual processor(s) from doing and work and essentially "freezes" the virtual machine.
  • the virtual machine has no notion of time. During the suspended state, all virtual machine state is preserved so that the virtual machine processor(s) can more or less seamlessly continue execution from the point of suspension whenever the virtual machine is resumed.
  • Saving the pending I/O operations may entail simply maintaining the pending I/O operations on an I/O stack (e.g., I/O stack 149) or removing the pending I/O operations from an I/O stack and saving them in a buffer or other temporary storage until such time as the operations can be reinserted onto the I/O stack. Still other techniques for saving the pending I/O operations may be used.
  • the virtual machine hosting environment determines that the unavailability condition no longer exists.
  • this step may be performed when virtual machine I/O monitor 154 executing on physical machine 102 determines that an unavailability condition no longer exists with respect to remote storage 104.
  • Various techniques for determining that an unavailability condition no longer exists with respect to a remote storage will be described below.
  • the virtual machine hosting environment resumes the virtual machine and resolves any saved pending I/O operations directed to the remote storage in response to at least the determination that the unavailability condition no longer exists.
  • this step may be performed when virtual machine manager 152 executing on physical machine 102 performs the following operations in response to at least receiving a notification from virtual machine I/O monitor 154 that an unavailability condition no longer exists with respect to remote storage 104: (1) resume the virtual machine of child partition 118 and (2) resolve any saved pending I/O operations that were generated on behalf of the virtual machine of child partition 118 and are directed to remote storage 104.
  • resuming a suspended virtual machine may entail re-enabling one or more virtual processors within the virtual machine to be scheduled on the physical processors of the physical machine. This re-enables the virtual processor(s) to do work and essentially "unfreezes" the virtual machine.
  • the virtual machine may experience a jump in "wall clock” time and its network connection may have been timed out by its peers, but as noted above the software and data inside the virtual machine have been preserved and are available for continued processing.
  • Resolving the saved pending I/O operations may entail issuing the saved pending I/O operations from an I/O stack (e.g., I/O stack 149) to the remote storage.
  • an I/O stack e.g., I/O stack 149
  • this step may also entail reinserting the saved pending I/O operations into the I/O stack.
  • FIG. 3 depicts a flowchart 300 of a first method by which a virtual machine hosting environment may determine that an unavailability condition exists with respect to a remote storage.
  • the method of flowchart 300 will be described with continued reference to system 100 of FIG. 1 . However, the method is not limited to that embodiment.
  • the method of flowchart 300 may be performed by any of a wide variety of virtual machine hosting environments other than the particular one shown in FIG. 1 .
  • the method of flowchart 300 begins at step 302, in which the virtual machine hosting environment monitors at least one I/O operation initiated by a virtual machine and directed to the remote storage.
  • this step may be performed when virtual machine I/O monitor 154 monitors at least one I/O operation initiated by the virtual machine of first child partition 118 that is ultimately directed to remote storage 104.
  • this step may entail monitoring at least one I/O operation pending on I/O stack 149.
  • Monitoring the at least one I/O operation may involve, for example and without limitation, monitoring whether at least one I/O operation succeeds or fails (e.g., results in the receipt of an error message) or determining how long it takes for at least one I/O operation to be completed.
  • the virtual machine hosting environment determines that the unavailability condition exists with respect to the remote storage.
  • this step may be performed when virtual I/O monitor 154 determines that an unavailability condition exists with respect to remote storage 104 based on the results of the monitoring of step 302.
  • This step may entail, for example and without limitation, determining that the unavailability conditions exists because an I/O error was received for one or more monitored I/O operations or because a latency associated with one or more monitored I/O operations is determined to exceed a threshold.
  • other criteria relating to monitored I/O operation(s) may be used to determine that the unavailability condition exists.
  • FIG. 4 depicts a flowchart 400 of a second method by which a virtual machine hosting environment may determine that an unavailability condition exists with respect to a remote storage.
  • the method of flowchart 400 will be described with continued reference to system 100 of FIG. 1 .
  • the method is not limited to that embodiment.
  • the method of flowchart 400 may be performed by any of a wide variety of virtual machine hosting environments other than the particular one shown in FIG. 1 .
  • the method of flowchart 400 begins at step 402, in which the virtual machine hosting environment monitors a plurality of I/O operations respectively initiated by a plurality of virtual machines and directed to the remote storage.
  • this step may be performed when virtual machine I/O monitor 154 monitors at least one I/O operation initiated by the virtual machine of first child partition 118 that is ultimately directed to remote storage 104 as well as at least one I/O operation initiated by the virtual machine of second child partition 120 that is ultimately directed to remote storage 104.
  • Monitoring the plurality of I/O operations may involve, for example and without limitation, monitoring whether each I/O operation succeeds or fails (e.g., results in the receipt of an error message) or determining how long it takes for each I/O operation to be completed.
  • the virtual machine hosting environment determines that the unavailability condition exists with respect to the remote storage.
  • this step may be performed when virtual I/O monitor 154 determines that an unavailability condition exists with respect to remote storage 104 based on the results of the monitoring of step 402.
  • This step may entail, for example and without limitation, determining that the unavailability conditions exists because an I/O error was received for one or more monitored I/O operations or because a latency associated with one or more monitored I/O operations is determined to exceed a threshold.
  • other criteria relating to monitored I/O operation(s) may be used to determine that the unavailability condition exists.
  • an error or latency associated with one or more I/O requests from a first virtual machine may lead to the declaration of an unavailability condition that results in the suspension of a second virtual machine that is executing on the same physical machine as the first virtual machine. This is because the monitoring upon which the declaration of the unavailability condition is based takes into account I/O requests from multiple virtual machines executing on the same physical machine.
  • FIG. 5 depicts a flowchart 500 of a first method by which a virtual machine hosting environment may determine that an unavailability condition no longer exists with respect to a remote storage.
  • the method of flowchart 500 will be described with continued reference to system 100 of FIG. 1 .
  • the method is not limited to that embodiment.
  • the method of flowchart 500 may be performed by any of a wide variety of virtual machine hosting environments other than the particular one shown in FIG. 1 .
  • the method of flowchart 500 begins at step 502, in which the virtual machine hosting environment intermittently probes the remote storage.
  • this step may be performed when virtual machine I/O monitor 154 intermittently probes remote storage 104.
  • Intermittently probing remote storage 104 may comprise, for example, intermittently sending messages to the remote storage in a lightweight (e.g., relatively infrequent) fashion until a response is received therefrom that indicates that the unavailability condition no longer exists.
  • the virtual machine hosting environment determines that the unavailability condition no longer exists based at least on receiving a response from the remote storage that indicates that the unavailability condition no longer exists.
  • this step may be performed when virtual machine I/O monitor 154 determines that the unavailability condition no longer exists based at least on receiving a response from remote storage 104 that indicates that the unavailability condition no longer exists.
  • FIG. 6 depicts a flowchart 600 of a second method by which a virtual machine hosting environment may determine that an unavailability condition no longer exists with respect to a remote storage.
  • the method of flowchart 600 will be described with continued reference to system 100 of FIG. 1 .
  • the method is not limited to that embodiment.
  • the method of flowchart 600 may be performed by any of a wide variety of virtual machine hosting environments other than the particular one shown in FIG. 1 .
  • the method of flowchart 600 begins at step 602, in which the virtual machine hosting environment intermittently retries an I/O operation directed to the remote storage.
  • this step may be performed when virtual machine I/O monitor 154 or some other entity within the virtual machine hosting environment shown in FIG. 1 causes a pending I/O operation directed to remote storage 104 to be retried.
  • the pending I/O operation may be an I/O operation that was initiated by the virtual machine of first child partition 118 or the virtual machine of second child partition 120 and that is being managed by host operating system 140.
  • the pending I/O operation may be one for which an error response was previously received or one for which no response has previously been received.
  • step 604 when the I/O operation succeeds, the virtual machine hosting environment determines that the unavailability condition no longer exists.
  • this step may be performed when virtual machine I/O monitor 154 determines that the unavailability condition no longer exists when the I/O operation that was retried during step 602 succeeds.
  • the success of the I/O operation may be indicated, for example, by receiving a response from the remote storage that indicates that a read or write operation has completed successfully.
  • the virtual machine hosting environment may be operable to terminate the virtual machine in response to at least the determination that the unavailability condition exists with respect to the remote storage and to restart the virtual machine in response to at least the determination that the unavailability condition no longer exists.
  • Such an approach may be utilized, for example, where an application or operating system running on a virtual machine does not respond well to the jump in "wall clock" time that occurs when a virtual machine is suspended and resumed. In such a case, it may be deemed preferable to terminate the virtual machine during a period of remote storage unavailability and restart the virtual machine after such period ends, even if such termination involves crashing of an application or operating system running on the virtual machine.
  • FIG. 7 depicts a flowchart 700 of another method for mitigating the impact of intermittent storage unavailability on virtual machines.
  • the method of flowchart 700 will be described with continued reference to system 100 of FIG. 1 .
  • the method is not limited to that embodiment.
  • the method of flowchart 700 may be performed by any of a wide variety of virtual machine hosting environments other than the particular one shown in FIG. 1 .
  • the method of flowchart 700 begins at step 702 in which a virtual machine hosting environment executing on a physical machine instantiates a virtual machine for execution on the physical machine.
  • the virtual machine is capable of initiating I/O operations directed to a remote storage to which the physical machine is communicatively connected.
  • this step may be performed when virtual machine manager 152 executing on physical machine 102 instantiates the virtual machine of first child partition 118 for execution on physical machine 102.
  • the virtual machine of first child partition 118 is capable of initiating I/O operations that will ultimately be directed to remote storage 104 to which physical machine 102 is communicatively connected via network(s) 106.
  • the virtual machine hosting environment determines that an unavailability condition exists with respect to the remote storage.
  • this step may be performed when virtual machine I/O monitor 154 executing on physical machine 102 determines that an unavailability condition exists with respect to remote storage 104.
  • the virtual machine hosting environment terminates the virtual machine (also referred to as destroying or "tearing down" the virtual machine) in response to at least the determination that the unavailability condition exists with respect to the remote storage.
  • this step may be performed when virtual machine manager 152 executing on physical machine 102 terminates the virtual machine of child partition 118 in response to at least receiving a notification from virtual machine I/O monitor 154 that an unavailability condition exists with respect to remote storage 104.
  • the virtual machine hosting environment determines that the unavailability condition no longer exists.
  • this step may be performed when virtual machine I/O monitor 154 executing on physical machine 102 determines that an unavailability condition no longer exists with respect to remote storage 104.
  • the virtual machine hosting environment restarts the virtual machine in response to at least the determination that the unavailability condition no longer exists.
  • this step may be performed when virtual machine manager 152 executing on physical machine 102 restarts the virtual machine of child partition 118 in response to at least receiving a notification from virtual machine I/O monitor 154 that an unavailability condition no longer exists with respect to remote storage 104.
  • FIG. 8 is a block diagram of a system 800 that includes multiple virtual machine hosting environments, each of which is operable to mitigate the impact of intermittent storage unavailability on virtual machines hosted thereby.
  • system 800 includes a plurality of physical machines that includes at least a physical machine 802, a physical machine 804, a physical machine 806, and a physical machine 808.
  • Physical machine 804 includes a virtual machine hosting environment 820 that is executed thereon.
  • Virtual machine hosting environment 820 hosts a plurality of virtual machines that include at least a virtual machine 822 and a virtual machine 824.
  • Each of virtual machine 822 and virtual machine 824 is capable of initiating I/O operations directed to a remote storage 850 to which physical machine 804 is communicatively connected.
  • Physical machine 804 and virtual machine hosting environment 820 may be implemented using a software and hardware architecture that is substantially similar to that shown in FIG. 1 and described above.
  • Physical machine 806 and physical machine 808 are implemented in a like manner to physical machine 804.
  • physical machine 806 includes a virtual machine hosting environment 830 that hosts a plurality of virtual machines including a virtual machine 832 and a virtual machine 834, each of which is capable of initiating I/O operations directed to remote storage 850 to which physical machine 806 is communicatively connected.
  • physical machine 808 includes a virtual machine hosting environment 840 that hosts a plurality of virtual machines including a virtual machine 842 and a virtual machine 844, each of which is capable of initiating I/O operations directed to remote storage 850 to which physical machine 806 is communicatively connected.
  • physical machine 802 is communicatively connected to each of physical machine 804, physical machine 806 and physical machine 808.
  • Physical machine 802 includes a fabric controller 810 installed thereon.
  • Fabric controller 810 comprises a software component that enables a user thereof to monitor and manage each of physical machines 804, 806 and 808 and the virtual machines hosted respectively thereon.
  • Fabric controller 810 may also enable a user thereof to coordinate resources for applications running on such platforms.
  • Fabric controller 810 is operable to provide a user interface (e.g., a graphical user interface) by which a user may invoke the functionality thereof.
  • fabric controller 810 comprises a MICROSOFT® AZURE® FABRIC CONTROLLER, although this is only an example and is not intended to be limiting.
  • fabric controller 810 is operable to enable a user thereof to specify a policy that specifies criteria for determining when an unavailability condition exists with respect to a remote storage, such as remote storage 850.
  • the policy may specify a predefined I/O latency that must be exceed before an unavailability condition may be declared.
  • the policy may specify a number of I/O error messages that must be received before an unavailability condition may be declared.
  • the policy may be disseminated from fabric controller 810 to each of virtual machine hosting environment 820, virtual machine hosting environment 830 and virtual machine hosting environment 840 and stored thereby.
  • Each of virtual machine hosting environment 820, virtual machine hosting environment 830 and virtual machine hosting environment 840 may be operable to use the policy received from fabric controller 810 to determine when an unavailability condition exists with respect to remote storage 850. Based on this determination, each of virtual machine hosting environment 820, virtual machine hosting environment 830 and virtual machine hosting environment 840 may pause or terminate one or more of the virtual machines hosted thereby, depending upon the implementation. It is noted that the same or different policies may be disseminated to different virtual machine hosting environments.
  • FIG. 9 depicts a flowchart 900 of a third method by which a virtual machine hosting environment may determine that an unavailability condition exists with respect to a remote storage.
  • the method of flowchart 900 will be described with continued reference to system 800 of FIG. 8 . However, the method is not limited to that embodiment.
  • the method of flowchart 900 begins at step 902, in which a policy is received by a virtual machine hosting environment.
  • this step may be performed by any of virtual machine hosting environment 820, virtual machine hosting environment 830 or virtual machine hosting environment 840 when such virtual machine hosting environment receives a policy from fabric controller 810.
  • the policy may specify criteria for determining when an unavailability condition exists with respect to a remote storage, such as remote storage 850.
  • the virtual machine hosting environment determines that an unavailability condition exists with respect to the remote storage based at least on the criteria included in the policy.
  • this step may be performed by any of virtual machine hosting environment 820, virtual machine hosting environment 830 or virtual machine hosting environment 840 when such virtual machine hosting environment determines that an unavailability condition exists with respect to remote storage 850 based at least on the criteria included in the policy.
  • fabric controller 810 is operable to enable a user thereof to specify a policy that indicates whether a virtual machine should be suspended and resumed in response to intermittent storage unavailability (e.g., in accordance with the method described above in reference to flowchart 200 of FIG. 2 ) or whether the virtual machine should be terminated and restarted in response to intermittent storage unavailability (e.g., in accordance with the method described above in reference to flowchart 700 of FIG. 7 ).
  • This enables a user to selectively apply different policies to different virtual machines. Such selective application may be based on any of a wide variety of factors including but not limited to the applications and/or operating systems executed by the virtual machines and/or user/customer preference.
  • the policy may also specify a set of criteria to be considered by a virtual machine hosting environment in determining which approach (i.e., suspend/resume vs. terminate/restart) to apply to a particular virtual machine.
  • FIG. 10 depicts a flowchart 1000 of a method for selecting one of a plurality of techniques for mitigating the impact of intermittent storage unavailability on virtual machines.
  • the method of flowchart 1000 will be described with continued reference to system 800 of FIG. 8 .
  • the method is not limited to that embodiment.
  • the method of flowchart 1000 begins at step 1002, in which a policy is received by a virtual machine hosting environment.
  • this step may be performed by any of virtual machine hosting environment 820, virtual machine hosting environment 830 or virtual machine hosting environment 840 when such virtual machine hosting environment receives a policy from fabric controller 810.
  • the policy may indicates whether a virtual machine should be suspended and resumed in response to intermittent storage unavailability (e.g., in accordance with the method described above in reference to flowchart 200 of FIG. 2 ) or whether the virtual machine should be terminated and restarted in response to intermittent storage unavailability (e.g., in accordance with the method described above in reference to flowchart 700 of FIG. 7 ).
  • the policy may alternatively specify a set of criteria to be considered in determining which approach (i.e., suspend/resume vs. terminate/restart) to apply to a particular virtual machine.
  • the virtual machine hosting environment determines whether to suspend/resume or terminate/restart a virtual machine based at least on the policy.
  • this step may be performed by any of virtual machine hosting environment 820, virtual machine hosting environment 830 or virtual machine hosting environment 840 when such virtual machine hosting environment determines whether to suspend/resume or terminate/restart a virtual machine hosted thereby based at least on the policy.
  • each of virtual machine hosting environment 804, virtual machine hosting environment 806 and virtual machine hosting environment 808 is be operable to share I/O status information with the other virtual machine hosting environments and such other virtual machine hosting environments are operable to utilize the shared I/O status information in determining whether an unavailability condition exists with respect to remote storage 850.
  • virtual machine hosting environment 820 may then send I/O status messages to each of virtual machine hosting environment 830 and virtual machine hosting environment 840 to alert them to this fact.
  • Each of virtual machine hosting environment 830 and virtual machine hosting environment 840 may then consider this information in formulating a determination of whether an unavailability condition exists with respect to remote storage 850.
  • peer-to-peer messaging may be leveraged to enable multiple virtual machine hosting environments to react quickly to an unavailability condition with respect to a commonly-used remote storage.
  • the aforementioned I/O status messages may be sent proactively by a virtual machine hosting environment or in response to a query received from another virtual machine hosting environment.
  • fabric controller 810 may be operable to enable a user thereof to select a subset of virtual machine hosting environments that should report I/O status to other virtual machine hosting environments.
  • FIG. 11 depicts a flowchart 1100 of a fourth method by which a virtual machine hosting environment determines that an unavailability condition exists with respect to a remote storage according to the invention.
  • the method of flowchart 1100 will be described with continued reference to system 800 of FIG. 8 .
  • the method is not limited to that embodiment.
  • the method of flowchart 1100 begins at step 1102, in which a virtual machine hosting environment executing on a physical machine receives an I/O status message from a different physical machine.
  • this step may step may be performed by virtual machine hosting environment 820 executing on physical machine 804 when it receives an I/O status message from virtual machine hosting environment 830 executing on physical machine 806 or from virtual machine hosting environment 840 executing on physical machine 808.
  • the virtual machine hosting environment determines that an unavailability condition exists with respect to a remote storage based at least on at least I/O status message received during step 1102.
  • this step may be performed by virtual machine hosting environment 820 when it determines that an unavailability condition exists with respect to remote storage 850 based at least on the I/O status message received during step 1102.
  • a similar peer-to-peer messaging approach may be used to help each of virtual machine hosting environment 804, virtual machine hosting environment 806 and virtual machine hosting environment 808 determine whether an unavailability condition no longer exists with respect to remote storage 850.
  • virtual machine hosting environment 820 may then send I/O status messages to each of virtual machine hosting environment 830 and virtual machine hosting environment 840 to alert them to this fact.
  • Each of virtual machine hosting environment 830 and virtual machine hosting environment 840 may then consider this information in formulating a determination of whether an unavailability condition no longer exists with respect to remote storage 850.
  • FIG. 12 depicts a flowchart of a third method by which a virtual machine hosting environment determines that an unavailability condition no longer exists with respect to a remote storage according to the invention.
  • the method of flowchart 1200 will be described with continued reference to system 800 of FIG. 8 .
  • the method is not limited to that embodiment.
  • the method of flowchart 1200 begins at step 1202, in which a virtual machine hosting environment executing on a physical machine receives an I/O status message from a different physical machine.
  • this step may step may be performed by virtual machine hosting environment 820 executing on physical machine 804 when it receives an I/O status message from virtual machine hosting environment 830 executing on physical machine 806 or from virtual machine hosting environment 840 executing on physical machine 808.
  • the virtual machine hosting environment determines that an unavailability condition no longer exists with respect to a remote storage based at least on at least I/O status message received during step 1202.
  • this step may be performed by virtual machine hosting environment 820 when it determines that an unavailability condition no longer exists with respect to remote storage 850 based at least on the I/O status message received during step 1202.
  • FIG. 13 depicts an example processor-based computer system 1300 that may be used to implement various embodiments described herein.
  • system 1300 may be used to implement any of physical machine 102, physical machine 802, physical machine 804, physical machine 806 and physical machine 808 as described above in reference to FIGS. 1 and 8 .
  • System 1300 may also be used to implement any or all of the steps of any or all of the flowcharts depicted in FIGS. 2-7 and 9-12 .
  • the description of system 1300 provided herein is provided for purposes of illustration, and is not intended to be limiting. Embodiments may be implemented in further types of computer systems, as would be known to persons skilled in the relevant art(s).
  • system 1300 includes a processing unit 1302, a system memory 1304, and a bus 1306 that couples various system components including system memory 1304 to processing unit 1302.
  • Processing unit 1302 may comprise one or more microprocessors or microprocessor cores.
  • Bus 1306 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures.
  • System memory 1304 includes read only memory (ROM) 1308 and random access memory (RAM) 1310.
  • ROM read only memory
  • RAM random access memory
  • a basic input/output system 1312 (BIOS) is stored in ROM 1308.
  • System 1300 also has one or more of the following drives: a hard disk drive 1314 for reading from and writing to a hard disk, a magnetic disk drive 1316 for reading from or writing to a removable magnetic disk 1318, and an optical disk drive 1320 for reading from or writing to a removable optical disk 1322 such as a CD ROM, DVD ROM, BLU-RAYTM disk or other optical media.
  • Hard disk drive 1314, magnetic disk drive 1316, and optical disk drive 1320 are connected to bus 1306 by a hard disk drive interface 1324, a magnetic disk drive interface 1326, and an optical drive interface 1328, respectively.
  • the drives and their associated computer-readable media provide nonvolatile storage of computer-readable instructions, data structures, program modules and other data for the computer.
  • a hard disk a removable magnetic disk and a removable optical disk
  • other types of computer-readable memory devices and storage structures can be used to store data, such as flash memory cards, digital video disks, random access memories (RAMs), read only memories (ROM), and the like.
  • program modules may be stored on the hard disk, magnetic disk, optical disk, ROM, or RAM These program modules include an operating system 1330, one or more application programs 1332, other program modules 1334, and program data 1336.
  • the program modules may include computer program logic that is executable by processing unit 1302 to perform any or all of the functions and features of physical machine 102, physical machine 802, physical machine 804, physical machine 806 and physical machine 808 as described above in reference to FIGS. 1 and 8 .
  • the program modules may also include computer program logic that, when executed by processing unit 1302, performs any of the steps or operations shown or described in reference to the flowcharts of FIGS. 2-7 and 9-12 .
  • a user may enter commands and information into system 1300 through input devices such as a keyboard 1338 and a pointing device 1340 (e.g., a mouse).
  • Other input devices may include a microphone, joystick, game controller, scanner, or the like.
  • a touch screen is provided in conjunction with a display 1344 to allow a user to provide user input via the application of a touch (as by a finger or stylus for example) to one or more points on the touch screen.
  • processing unit 1302 through a serial port interface 1342 that is coupled to bus 1306, but may be connected by other interfaces, such as a parallel port, game port, or a universal serial bus (USB).
  • Such interfaces may be wired or wireless interfaces.
  • Display 1344 is connected to bus 1306 via an interface, such as a video adapter 1346.
  • system 1300 may include other peripheral output devices (not shown) such as speakers and printers.
  • System 1300 is connected to a network 1348 (e.g., a local area network or wide area network such as the Internet) through a network interface or adapter 1350, a modem 1352, or other suitable means for establishing communications over the network.
  • a network 1348 e.g., a local area network or wide area network such as the Internet
  • modem 1352 or other suitable means for establishing communications over the network.
  • Modem 1352 which may be internal or external, is connected to bus 1306 via serial port interface 1342.
  • computer program medium As used herein, the terms “computer program medium,” “computer-readable medium,” and “computer-readable storage medium” are used to generally refer to memory devices or storage structures such as the hard disk associated with hard disk drive 1314, removable magnetic disk 1318, removable optical disk 1322, as well as other memory devices or storage structures such as flash memory cards, digital video disks, random access memories (RAMs), read only memories (ROM), and the like.
  • Such computer-readable storage media are distinguished from and non-overlapping with communication media (do not include communication media).
  • Communication media typically embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave.
  • modulated data signal means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.
  • communication media includes wireless media such as acoustic, RF, infrared and other wireless media. Embodiments are also directed to such communication media.
  • computer programs and modules may be stored on the hard disk, magnetic disk, optical disk, ROM, or RAM Such computer programs may also be received via network interface 1350, serial port interface 1342, or any other interface type. Such computer programs, when executed or loaded by an application, enable system 1300 to implement features of embodiments of the present invention discussed herein. Accordingly, such computer programs represent controllers of the system 1300.
  • Embodiments are also directed to computer program products comprising software stored on any computer useable medium. Such software, when executed in one or more data processing devices, causes a data processing device(s) to operate as described herein.
  • Embodiments of the present invention employ any computer-useable or computer-readable medium, known now or in the future. Examples of computer-readable mediums include, but are not limited to memory devices and storage structures such as RAM, hard drives, floppy disks, CD ROMs, DVD ROMs, zip disks, tapes, magnetic storage devices, optical storage devices, MEMs, nanotechnology-based storage devices, and the like.
  • system 1300 may be implemented as hardware logic/electrical circuitry or firmware.
  • one or more of these components may be implemented in a system-on-chip (SoC).
  • SoC may include an integrated circuit chip that includes one or more of a processor (e.g., a microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and/or further circuits and/or embedded firmware to perform its functions.
  • a processor e.g., a microcontroller, microprocessor, digital signal processor (DSP), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Claims (15)

  1. Appareil (800) comprenant :
    une première machine physique (802) comprenant au moins un processeur et une mémoire qui stocke un environnement d'hébergement de machine virtuelle pour son exécution par ledit au moins un processeur, l'environnement d'hébergement de machine virtuelle comprenant :
    un gestionnaire de machine virtuelle (152) utilisable pour instancier une machine virtuelle pour son exécution par ledit au moins un processeur, la machine virtuelle étant capable d'initier des opérations d'entrée/sortie (I/O) adressées à un stockage distant (850) auquel la première machine physique (802) est connectée par communication ;
    un moniteur d'entrée/sortie de machine virtuelle (154) utilisable pour déterminer qu'une condition d'indisponibilité existe concernant le stockage distant (850) et pour déterminer ensuite que la condition d'indisponibilité n'existe plus ;
    le gestionnaire de machine virtuelle (152) étant en outre utilisable pour suspendre la machine virtuelle en réponse au moins à la détermination par le moniteur d'entrée/sortie de machine virtuelle du fait que la condition d'indisponibilité existe concernant le stockage distant et pour réactiver la machine virtuelle en réponse au moins à la détermination par le moniteur d'entrée/sortie de machine virtuelle du fait que la condition d'indisponibilité n'existe plus ;
    dans lequel l'appareil comprend en outre une pluralité d'environnements d'hébergement de machine virtuelle (820, 830, 840), chaque environnement d'hébergement de machine virtuelle étant hébergé sur une machine physique différente et chaque environnement d'hébergement de machine virtuelle étant connecté au stockage distant utilisé communément (850), dans lequel chacun des environnements de la pluralité d'environnements d'hébergement de machine virtuelle et des différentes machines physiques est implémenté en fonction des premiers environnement d'hébergement de machine virtuelle et machine physique ;
    caractérisé en ce que :
    chacun des environnements d'hébergement de machine virtuelle (820, 830, 840) est utilisable pour partager de l'information d'état d'entrée/sortie du stockage distant utilisé communément, l'information d'état d'entrée/sortie étant reçue d'une machine physique différente, avec les autres environnements d'hébergement de machine virtuelle, et pour utiliser l'information d'état d'entrée/sortie afin de déterminer (1104) si une condition d'indisponibilité existe ou n'existe plus concernant le stockage distant ;
    dans lequel chacun des environnements d'hébergement de machine virtuelle (820, 830, 840) est adapté pour déterminer (1204) que la condition d'indisponibilité n'existe plus en réessayant de façon intermittente des opérations d'entrée/sortie adressées au stockage distant jusqu'à ce que ces opérations d'entrée/sortie réussissent.
  2. Appareil selon la revendication 1, dans lequel le moniteur d'entrée/sortie de machine virtuelle (154) est utilisable pour déterminer que la condition d'indisponibilité existe en surveillant au moins une opération d'entrée/sortie initiée par la machine virtuelle et adressée au stockage distant.
  3. Appareil selon la revendication 2, dans lequel le moniteur d'entrée/sortie de machine virtuelle (154) est utilisable pour déterminer que la condition d'indisponibilité existe sur base au moins d'une latence associée à ladite au moins une opération d'entrée/sortie.
  4. Appareil selon la revendication 2, dans lequel le moniteur d'entrée/sortie de machine virtuelle (154) est utilisable pour déterminer que la condition d'indisponibilité existe sur base au moins d'une erreur d'entrée/sortie associée à ladite au moins une opération d'entrée/sortie.
  5. Appareil selon la revendication 1, dans lequel :
    le gestionnaire de machine virtuelle (152) est utilisable pour instancier une pluralité de machines virtuelles pour leur exécution par ledit au moins un processeur, chaque machine de la pluralité de machines virtuelles étant capable d'initier des opérations d'entrée/sortie adressées au stockage distant (850) ; et
    le moniteur d'entrée/sortie de machine virtuelle (154) est utilisable pour déterminer que la condition d'indisponibilité existe en surveillant une pluralité d'opérations d'entrée/sortie initiées respectivement par la pluralité de machines virtuelles et adressées au stockage distant (850).
  6. Appareil selon la revendication 1, dans lequel le moniteur d'entrée/sortie de machine virtuelle (154) est utilisable pour déterminer que la condition d'indisponibilité existe sur base d'au moins un message d'état d'entrée/sortie reçu d'au moins une deuxième machine physique à laquelle la première machine physique est connectée par communication.
  7. Appareil selon la revendication 1, dans lequel le moniteur d'entrée/sortie de machine virtuelle (154) est utilisable pour déterminer que la condition d'indisponibilité existe sur base de critères inclus dans une politique spécifiée par l'utilisateur.
  8. Appareil selon la revendication 1, dans lequel le moniteur d'entrée/sortie de machine virtuelle (154) est utilisable pour déterminer que la condition d'indisponibilité n'existe plus en sondant de façon intermittente le stockage distant (850) jusqu'à ce qu'une réponse provienne de celui-ci, qui indique que la condition d'indisponibilité n'existe plus.
  9. Appareil selon la revendication 1, dans lequel le moniteur d'entrée/sortie de machine virtuelle (154) est utilisable pour déterminer que la condition d'indisponibilité n'existe plus en réessayant de façon intermittente une opération d'entrée/sortie adressée au stockage distant jusqu'à ce que l'opération d'entrée/sortie réussisse.
  10. Appareil selon la revendication 1, dans lequel le moniteur d'entrée/sortie de machine virtuelle (154) est utilisable pour déterminer que la condition d'indisponibilité n'existe plus sur base d'au moins un message d'état d'entrée/sortie reçu d'au moins une deuxième machine physique à laquelle la première machine physique est connectée par communication.
  11. Appareil selon la revendication 1, dans lequel le gestionnaire de machine virtuelle (152) est en outre utilisable pour sauvegarder une ou plusieurs demandes d'entrée/sortie en attente générées par la machine virtuelle conjointement à une mise en veille de la machine virtuelle, et pour envoyer lesdites une ou plusieurs demandes d'entrée/sortie en attente au stockage distant (850) conjointement à la réactivation de la machine virtuelle.
  12. Système comprenant :
    un système de stockage ; et
    un premier ordinateur connecté par communication au système de stockage et sur lequel sont installés pour leur exécution :
    un environnement d'hébergement de machine virtuelle utilisable pour instancier une machine virtuelle pour son exécution sur le premier ordinateur (802), la machine virtuelle étant capable d'initier des opérations d'entrée/sortie (I/O) adressée au système de stockage, l'environnement d'hébergement de machine virtuelle étant en outre utilisable pour déterminer qu'une condition d'indisponibilité existe concernant le système de stockage, pour désactiver la machine virtuelle en réponse au moins à la détermination du fait que la condition d'indisponibilité existe, pour déterminer que la condition d'indisponibilité n'existe plus, et pour réactiver la machine virtuelle en réponse au moins à la détermination du fait que la condition d'indisponibilité n'existe plus ;
    le système comprenant en outre une pluralité d'environnements d'hébergement de machine virtuelle (820, 830, 840), chaque environnement d'hébergement de machine virtuelle étant hébergé sur un ordinateur différent et chaque environnement d'hébergement de machine virtuelle étant connecté à un stockage distant utilisé communément (850), dans lequel chacun des environnements de la pluralité d'environnements d'hébergement de machine virtuelle et des différents ordinateurs est implémenté en fonction des premiers environnement d'hébergement de machine virtuelle et ordinateur ;
    caractérisé en ce que :
    chacun des environnements d'hébergement de machine virtuelle est utilisable pour partager de l'information d'état d'entrée/sortie du stockage distant utilisé communément, l'information d'état d'entrée/sortie étant reçue d'un ordinateur différent, avec les autres environnements d'hébergement de machine virtuelle, et pour utiliser l'information d'état d'entrée/sortie afin de déterminer (1104) si une condition d'indisponibilité existe ou n'existe plus concernant le stockage distant ;
    dans lequel chacun des environnements d'hébergement de machine virtuelle est adapté pour déterminer (1204) que la condition d'indisponibilité n'existe plus en réessayant de façon intermittente des opérations d'entrée/sortie adressées au stockage distant jusqu'à ce que ces opérations d'entrée/sortie réussissent.
  13. Système selon la revendication 12, comprenant en outre :
    un deuxième ordinateur qui est connecté par communication au premier ordinateur et utilisable pour envoyer un message d'état d'entrée/sortie au premier ordinateur ;
    dans lequel l'environnement d'hébergement de machine virtuelle (820, 830, 840) est utilisable pour déterminer que la condition d'indisponibilité existe sur base au moins du message d'état d'entrée/sortie.
  14. Système selon la revendication 12, comprenant en outre :
    un contrôleur de fabric (810) qui est exécuté sur un deuxième ordinateur connecté par communication au premier ordinateur et utilisable pour envoyer une politique spécifiée par l'utilisateur au premier ordinateur ;
    dans lequel l'environnement d'hébergement de machine virtuelle (820, 830, 840) est utilisable pour déterminer que la condition d'indisponibilité existe sur base de critères inclus dans la politique spécifiée par l'utilisateur.
  15. Système selon la revendication 12, comprenant en outre :
    un deuxième ordinateur qui est connecté par communication au premier ordinateur et utilisable pour envoyer un message d'état d'entrée/sortie au premier ordinateur ;
    dans lequel l'environnement d'hébergement de machine virtuelle (820, 830, 840) est utilisable pour déterminer que la condition d'indisponibilité n'existe plus sur base au moins du message d'état d'entrée/sortie.
EP16745952.8A 2015-07-22 2016-07-21 Attenuation de l'impact de la non-disponibilite intermittente de stockage distant sur des machines virtuelles Active EP3326062B1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US14/805,800 US11138029B2 (en) 2015-07-22 2015-07-22 Mitigation of impact of intermittent storage unavailability on virtual machines
PCT/US2016/043218 WO2017015412A1 (fr) 2015-07-22 2016-07-21 Atténuation de l'impact d'une indisponibilité intermittente d'un stockage à distance sur des machines virtuelles

Publications (2)

Publication Number Publication Date
EP3326062A1 EP3326062A1 (fr) 2018-05-30
EP3326062B1 true EP3326062B1 (fr) 2019-12-04

Family

ID=56561484

Family Applications (1)

Application Number Title Priority Date Filing Date
EP16745952.8A Active EP3326062B1 (fr) 2015-07-22 2016-07-21 Attenuation de l'impact de la non-disponibilite intermittente de stockage distant sur des machines virtuelles

Country Status (4)

Country Link
US (1) US11138029B2 (fr)
EP (1) EP3326062B1 (fr)
CN (1) CN107851033B (fr)
WO (1) WO2017015412A1 (fr)

Family Cites Families (25)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4674038A (en) 1984-12-28 1987-06-16 International Business Machines Corporation Recovery of guest virtual machines after failure of a host real machine
US4912628A (en) 1988-03-15 1990-03-27 International Business Machines Corp. Suspending and resuming processing of tasks running in a virtual machine data processing system
US5790779A (en) * 1995-03-10 1998-08-04 Microsoft Corporation Method and system for consolidating related error reports in a computer system
US7213246B1 (en) 2002-03-28 2007-05-01 Veritas Operating Corporation Failing over a virtual machine
US20060271677A1 (en) * 2005-05-24 2006-11-30 Mercier Christina W Policy based data path management, asset management, and monitoring
US9189265B2 (en) 2006-12-21 2015-11-17 Vmware, Inc. Storage architecture for virtual machines
US7809985B2 (en) 2007-09-28 2010-10-05 Microsoft Corporation Offline hardware diagnostic environment
US20090164994A1 (en) 2007-12-20 2009-06-25 Virtual Computer, Inc. Virtual computing management systems and methods
US8914567B2 (en) 2008-09-15 2014-12-16 Vmware, Inc. Storage management system for virtual machines
US8769049B2 (en) 2009-04-24 2014-07-01 Microsoft Corporation Intelligent tiers of backup data
US9043454B2 (en) 2009-08-26 2015-05-26 Red Hat Israel, Ltd. Auto suspense of virtual machine on client disconnection
US8473947B2 (en) * 2010-01-18 2013-06-25 Vmware, Inc. Method for configuring a physical adapter with virtual function (VF) and physical function (PF) for controlling address translation between virtual disks and physical storage regions
US8751857B2 (en) 2010-04-13 2014-06-10 Red Hat Israel, Ltd. Monitoring of highly available virtual machines
US8326803B1 (en) * 2010-05-06 2012-12-04 Symantec Corporation Change tracking of individual virtual disk files
US9329947B2 (en) * 2010-06-22 2016-05-03 Red Hat Israel, Ltd. Resuming a paused virtual machine without restarting the virtual machine
US8489699B2 (en) * 2010-08-13 2013-07-16 Vmware, Inc. Live migration of virtual machine during direct access to storage over SR IOV adapter
US10203974B2 (en) 2010-12-20 2019-02-12 Microsoft Technology Licensing, Llc Probe insertion via background virtual machine
KR20130022091A (ko) 2011-08-24 2013-03-06 주식회사 케이티 클라우드 컴퓨팅 서버 시스템의 가상머신 제어 장치 및 방법
US8825851B2 (en) 2012-08-17 2014-09-02 Vmware, Inc. Management of a virtual machine in a storage area network environment
US9361199B2 (en) * 2012-08-24 2016-06-07 Vmware, Inc. Protecting virtual machines against storage connectivity failures
US9250827B2 (en) 2012-12-14 2016-02-02 Vmware, Inc. Storing checkpoint file in high performance storage device for rapid virtual machine suspend and resume
US8935502B2 (en) * 2012-12-21 2015-01-13 Red Hat, Inc. Synchronous management of disk flush requests
US9483362B2 (en) * 2013-05-08 2016-11-01 Commvault Systems, Inc. Use of auxiliary data protection software in failover operations
US9104469B2 (en) 2013-06-11 2015-08-11 Vmware, Inc. Suspend-resume of virtual machines using de-duplication
US9348646B1 (en) * 2014-09-22 2016-05-24 Amazon Technologies, Inc. Reboot-initiated virtual machine instance migration

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Also Published As

Publication number Publication date
US20170024233A1 (en) 2017-01-26
US11138029B2 (en) 2021-10-05
CN107851033A (zh) 2018-03-27
CN107851033B (zh) 2022-03-29
WO2017015412A1 (fr) 2017-01-26
EP3326062A1 (fr) 2018-05-30

Similar Documents

Publication Publication Date Title
CN109564514B (zh) 用于部分卸载的虚拟化管理器处的存储器分配的方法和系统
US20220129299A1 (en) System and Method for Managing Size of Clusters in a Computing Environment
US9489230B1 (en) Handling of virtual machine migration while performing clustering operations
JP6219512B2 (ja) 仮想ハドゥープマネジャ
JP5851503B2 (ja) 高可用性仮想機械環境におけるアプリケーションの高可用性の提供
US9489274B2 (en) System and method for performing efficient failover and virtual machine (VM) migration in virtual desktop infrastructure (VDI)
US9430266B2 (en) Activating a subphysical driver on failure of hypervisor for operating an I/O device shared by hypervisor and guest OS and virtual computer system
CN109564523B (zh) 使用机会性管理程序降低性能可变性
US9912535B2 (en) System and method of performing high availability configuration and validation of virtual desktop infrastructure (VDI)
US20170017511A1 (en) Method for memory management in virtual machines, and corresponding system and computer program product
US20150378782A1 (en) Scheduling of tasks on idle processors without context switching
US20190235902A1 (en) Bully vm detection in a hyperconverged system
US9703651B2 (en) Providing availability of an agent virtual computing instance during a storage failure
US11675611B2 (en) Software service intervention in a computing system
US9430265B1 (en) System and method for handling I/O timeout deadlines in virtualized systems
WO2022052580A1 (fr) Procédé et système de surveillance de commutateur sur la base d'un bmc, dispositif et support
US7797473B2 (en) System for executing system management interrupts and methods thereof
EP3326062B1 (fr) Attenuation de l'impact de la non-disponibilite intermittente de stockage distant sur des machines virtuelles
US11797287B1 (en) Automatically terminating deployment of containerized applications
EP4195021A1 (fr) Procédé et système de migration en ligne pour serveur sans système d'exploitation
US10514975B2 (en) Preventing corruption by blocking requests
JP2012003510A (ja) 計算機及び転送プログラム
EP4155922A1 (fr) Attribution de dispositif fiable pour récipients basés sur machine virtuelle
US20240338282A1 (en) In-place recovery of fatal system errors at virtualization hosts
US7853758B2 (en) Avoiding failure of an initial program load in a logical partition of a data storage system

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20180117

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20190417

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20190709

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1210243

Country of ref document: AT

Kind code of ref document: T

Effective date: 20191215

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602016025572

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20191204

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200304

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200304

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200305

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200429

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200404

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602016025572

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1210243

Country of ref document: AT

Kind code of ref document: T

Effective date: 20191204

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

26N No opposition filed

Effective date: 20200907

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20200731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200721

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200731

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200731

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200721

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191204

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230505

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20240620

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20240619

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240619

Year of fee payment: 9