US20170272515A1 - Efficient live-migration of remotely accessed data - Google Patents
Efficient live-migration of remotely accessed data Download PDFInfo
- Publication number
- US20170272515A1 US20170272515A1 US15/071,852 US201615071852A US2017272515A1 US 20170272515 A1 US20170272515 A1 US 20170272515A1 US 201615071852 A US201615071852 A US 201615071852A US 2017272515 A1 US2017272515 A1 US 2017272515A1
- Authority
- US
- United States
- Prior art keywords
- physical machine
- data
- machine
- data subset
- physical
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0662—Virtualisation aspects
- G06F3/0664—Virtualisation aspects at device level, e.g. emulation of a storage device or system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
- G06F9/5088—Techniques for rebalancing the load in a distributed system involving task migration
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45583—Memory management, e.g. access or allocation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45595—Network integration; Enabling network access in virtual machine instances
Definitions
- Cloud computing refers to network-based computing in which collections of servers housed in data centers or “server farms” provide computational resources and data storage as needed to remote end users.
- Some cloud computing services provide access to software applications such as word processors and other commonly used applications to end users who interface with the applications through web browsers or other client-side software. Users' electronic data files are usually stored in the server farm rather than on the users' computing devices. Maintaining software applications and user data on a server farm simplifies management of end user computing devices.
- Some cloud computing services allow end users to execute software applications in virtual machines. In a public cloud computing environment, multiple users are able to launch virtual machines (VMs).
- VMs virtual machines
- Live-migration of data is the process of moving data off of one physical machine to another physical machine while the virtual machine (or alternatively, a non-virtual processing entity) performs arbitrary reads and writes on the data.
- This specification relates to live migration of data.
- This document describes a systematic method and system for moving data off of a storage computer to another storage computer while providing consistent and high performance access to the data to a third-party processing device (e.g., a virtual machine or an application) that is accessing the data remotely from another physical machine.
- a third-party processing device e.g., a virtual machine or an application
- one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of storing, in a first physical machine, data for a data processing process running on a second physical machine that is separate from the first physical machine, the storing data comprising storing the data according to a plurality of data subsets that are each exclusive of each other; for each data subset, logically mapping in a mapping, by the data processing process, an address range for the data subset on the first physical machine to a respective logical address range for the data processing process; enabling read and write access to the data by the data processing process according to the logical addressing; determining that a first data subset stored on the first physical machine is to be migrated to a third physical machine separate from the first physical machine and the second physical machine, and in response: blocking read and write access by the data processing process to the first data subset while maintaining read and write access by the data processing process to the other data subsets, migrating, from the first physical machine to the third physical machine, the first data subset
- a system that includes a first physical machine storing data according to a plurality of data subsets that are each exclusive of each other; a second physical machine that is separate from the first physical machine and a virtual machine on the second physical machine having read and write access to the data stored on the first physical machine and that, for each data subset, logically maps, in a mapping, an address range for the data subset on the first physical machine to a respective logical address range for the virtual machine; wherein in response to a determination that a first data subset stored on the first physical machine is to be migrated to a third physical machine separate from the first physical machine and the second physical machine: the first data subset is migrated from the first physical machine to the third physical machine; read and write access to the first data subset for the virtual machine is blocked during the migration while read and write access by the virtual machine to the other data subsets is maintained; and the mapping is updated by the virtual machine to logically map an address range for the first data
- the method provides better performance guarantees to the virtual machine than the other methods of live-migration such as pre-copy and post-copy live migration.
- pre-copy live-migration considers the whole data address space as a whole migrating unit, and thus the entire address space exists completely on the originating side or completely on the destination side of the migration.
- the virtual machine is allowed to continue reading/writing the data, but writes are tracked so that changes can be resent to the receiving physical machine(s) of the live migration. This resending takes more read bandwidth, network bandwidth, CPU processing, and time.
- a busy virtual machine will typically be changing the data faster than the changes can be sent over the network to the destination; in such situations, the virtual machines access rate must be slowed, resulting in performance degradation.
- the virtual machine With post-copy live-migration, the virtual machine is informed of the destination physical machine and requests the destination physical machine for the data.
- the destination physical machine provides the data if the data is stored on the destination physical machine; otherwise the destination physical machine fetches the data from the originating physical machine which the data is being migrated and then provides the data.
- the virtual machine experiences an increased latency. With many accesses from the destination physical machine to the originating physical machine, there is a significant overall bandwidth performance degradation.
- the virtual machine experiences virtually no impact to performance if it is not accessing the data subset being migrated.
- the amount of time the virtual machine must be blocked while waiting for the data subset to migrate decreases.
- the amount of metadata for storage mapping decreases.
- the live migration described below may be stopped at any time without losing progress. This is in contrast to pre-copy live migration, which must completely start over if stopped, and also in contrast to post-copy live migration, which cannot be stopped after accesses for the virtual machine are switched to the destination side.
- FIG. 1 is a block diagram of a cloud-based environment in which data may undergo live migration.
- FIG. 2 is a flow chart of an example process for the live migration of data.
- FIG. 3 is a flow chart of an example process for the live migration of data in which data subsets are migrated directly from a first physical machine to a third physical machine.
- FIG. 4 is a flow chart of an example process for the live migration of data in which data subsets are migrated from a first physical machine to a third physical machine and through the physical machine in which the virtual machine is implemented.
- a first physical machine stores data according to multiple data subsets that are each exclusive of each other.
- the data is stored for a data processing process, such as a virtual machine, running on a second physical machine that is separate from the first physical machine.
- the data processing process has read and write access to the data stored on the first physical machine, and for each data subset logically maps, in a mapping, an address range for a data subset on the first physical machine to a respective logical address range for the data processing process.
- the following steps are taken.
- the data are migrated in data subsets. For each data subset undergoing migration (e.g., in the process of being “in flight” from the first physical machine to the third physical machine), read and write access by the data processing process to the data subset is blocked. However, read and write access by the data processing process to the other data subsets not undergoing data migration is maintained. In this way the data processing process may still access much of the data that is stored in the first physical machine.
- the mapping is updated by the data processing process.
- an address range for the particular data subset on the third physical machine is mapped to the respective logical address range for the data processing process, and the pre-migration mapping of the particular data subset is thus replaced by the updated mapping.
- Read and write access to the first data subset for the data processing process is restored after the migration of the first data subset from the first physical machine to the third physical machine. Thereafter, the data processing process accesses the third physical machine when data stored in the migrated data subset is needed by the data processing process.
- a data processing process may not be notified of a migration of a data subset. Should the data processing process request access to data stored in the data subset from the first physical machine, it will then receive a response informing it that the data subset is now stored on a third physical machine (or currently undergoing migration to the third physical machine). If the data subset is currently undergoing migration, the third physical machine may block access to the data subset until the migration is complete.
- the data processing process may be proactively informed of the migration and may calculate an expected time of completion of the migration. The data processing process may then wait to request the data until the expected time of completion.
- a data subset may be sent directly from the first physical machine to the third physical machine, and then one of the first or third physical machines will inform the data processing process of the new location of the data subset when the migration is complete.
- the data subset may be passed through the data processing process acting as an intermediary, and the data processing process is responsible for migration. This allows the data processing process to be up-to-date about where data resides in near real time.
- the data processing process controls migration but the data subset is sent directly from one storage machine to the other.
- the virtual machine sends a “transfer address range” message to first storage machine instructing the first storage machine to read a specific address range specified by the transfer address range and write that data to another address range a second storage machine.
- the storage machines are stateless, but by the instructions sent from the virtual machine, the data-subset is sent directly from the first storage machine to the second storage machine. This results in less data transfer than passing data through the VM, and is more scalable if many storage machines are involved.
- a data subset may be a fixed size of memory unrelated to page or block size, e.g., 1 MB, 10 MB, or even 1 GB.
- data subsets may be realized at a block or page level, and a “watermark” is used such that all addresses below X are on the first physical machine, and all addresses at or above X are on the third physical machine. The value of X is updated in the data processing process as data are migrated. This can eliminate the data processing process mapping of logical address to data subset, and instead partitions the physical machines storing data according to the watermark value of X.
- FIG. 1 is a block diagram of a cloud-based environment 100 in which data may undergo live migration.
- a virtual machine In the written description below, an example implementation of a virtual machine is described. However, data migration of data for some of data processing process, such a client-based application have cloud-bases storage, or a cloud-based application having cloud based storage, may also be facilitated by the systems and methods described below.
- a host machine 110 which is a physical machine, in the cloud-based environment 100 , can contain one or more data processing apparatuses such as rack mounted servers or other computing devices.
- Storage machine 140 and 150 which are also physical machines, store data for a data processing process executing on the host machine 110 .
- the storage machines 140 and 150 may also be one or more data processing apparatuses such as rack mounted servers or other computing devices, and typically are designed to facilitate storage of data for cloud-based access by the host machine 110 communicating through a network 102 .
- the host machine 110 executes a host operating system 112 that manages host machine resources.
- the host operating systems 112 run software that virtualizes the underlying host machine hardware and manages concurrent execution of one or more virtual machines 120 .
- the host operating system 112 manages one virtual machine 120 .
- a host machine can, in general, manage larger quantities of virtual machines; however, the quantity may be limited based on physical resources of the host machine. For simplicity, only one virtual machine 120 is shown in FIG. 1 .
- the virtual machine 120 uses a simulated version of an underlying host machine hardware, which can be referred to as virtual hardware 122 .
- Software that is executed by the virtual hardware 122 can be referred to as guest software, e.g., a guest operating system 124 and guest applications 126 .
- guest software cannot determine if it is being executed by virtual hardware or by a physical host machine.
- a host machine's microprocessor(s) can include processor-level mechanisms to enable virtual hardware to execute software applications efficiently by allowing guest software instructions to be executed directly on the host machine's microprocessor without requiring code-rewriting, recompilation, or instruction emulation.
- the host machine 120 is allocated a set of virtual memory pages from the virtual memory of the underlying host operating system 112 and is allocated virtual disk blocks from one or more virtual disk drives for use by the guest software executing on the virtual machine.
- the actual physical storage need not be on the host machine 110 , and in the example shown, the storage is realized by the storage machine 140 .
- virtual disk blocks are allocated on physical disk drives managed by the storage machine and communicating with the host machine 110 through the network 102 .
- the virtual machine 120 can be allocated network addresses through which their respective processes can communicate with other processes via the network 102 .
- the guest data 142 need not initially be stored on a single physical machine, and can instead be initially stored across multiple storage machines. However, for simplicity of description, the starting point for this example is a single storage machine.
- the guest data 142 is stored according to multiple data subsets that are each exclusive of each other. As shown in FIG. 1 , the guest data 142 is stored in data subsets 144 and each data subset is illustratively indexed by one of the indices 0 . . . n.
- the mapping data 128 logically maps an address range for the data subset on the storage machine 140 to a respective logical address range for the virtual machine 120 . Thus, by use of the mapping data 128 , the virtual machine 120 can map a logical address space to a particular data subset stored on a particular physical machine. Finally, while the mapping data 128 is illustrated as being within the virtual machine 120 , the mapping data 128 may also be maintained by the host operating system 112 .
- Events may occur that may cause some or all of the guest data 142 to be migrated to one or more other storage machines.
- Such events may include a storage machine 140 preparing to go offline for service, which requires migration of all the data stored at the storage machine; load balancing, which requires the migration of at least a portion of the data stored at the storage machine; or quality of service requirements not being met, which may require the migration of at least a portion of the data stored at the storage machine.
- the storage machine may determine when a migration is necessary, and in other situations the virtual machine (or host machine) may determine when a migration is necessary.
- a process or entity external to the virtual machine, host machine and storage machines can also determine when a migration is necessary, and either the virtual machine can control the migration or the storage machines can control the migration, as described in general above and as will be described in more detail below.
- FIG. 2 A generalized process for the live migration of data, which is indicated by the arrow with reference callout 2 in FIG. 1 , is described with reference to FIG. 2 .
- One example process in which the physical storage machines partially (or fully) control the migration is described with reference to FIG. 3 , and is indicated by the arrow with reference callout 2 in combination with the arrows with reference callouts 3 A and 3 B in FIG. 1 .
- an example process in which the virtual machine (or host machine) partially (or fully) controls the migration is described with reference to FIG. 4 , and is indicated by the arrow with reference callout 2 in combination with the arrows with reference callouts 4 A and 4 B in FIG. 1 .
- the migration example described below will detail the migrating of data to one other physical machine—storage machine 150 .
- the guest data may be migrated from one of the storage machines to another storage machine that currently is storing some of the guest data, or a new storage machine that is not currently storing the guest data for the virtual machine 120 .
- FIG. 2 is a flow chart of an example process 200 for the live migration of data.
- the process 200 may be implemented in the physical machines 110 , 140 and 150 of FIG. 1 .
- the process 200 stores, in a first physical machine, data for a virtual machine running on a second physical machine that is separate from the first physical machine ( 202 ).
- the data 142 is stored according to data subsets that are each exclusive of each other.
- a “data subset” of the data 142 can be either a predefined data construct, such as a block, sector or page, or may be an arbitrarily defined unit of data, such as a 1 KB, 1 MB, LOMB, or even 1 GB amount of data.
- the block or page may be of the size as virtually realized for the virtual machine, or, alternatively, may be of a physical size as determined by the physical hardware used.
- the process 200 for each data subset, logically maps, by the virtual machine, an address range for the data subset on the first physical machine to a respective logical address range for the virtual machine ( 204 ).
- the virtual machine (or, alternatively, the host machine) logically maps the address at which the data appears to reside from the perspective of the virtual machine to the physical address at which the data actually resides.
- Any appropriate address translation process that can map a logical address in a data processing process on a first machine to a physical address on a second machine separate from the first machine can be used.
- the data subset in which the data subset is of an arbitrary size, several factors may be considered by an administrator when determining the size. The smaller the size of the data subset, the more mapping data 128 will be required. However, because the amount of time the data subset is in-flight during migration decreases as the size of the data subset decreases, smaller data subsets tend to result in fewer read and write delays that may occur when the virtual machine 120 attempts to access the data subset undergoing a migration.
- the virtual machine 120 may compare the rate of data access blocks due to migrations to a maximum block rate threshold. If the rate exceeds a maximum block rate threshold, then a memory management process is invoked by the virtual machine (or, alternatively by the storage machines storing the data) to reduce the data subset size. Thereafter, a new rate of data access blocks is determined. The process may continue until the rate is below the maximum block rate threshold.
- the virtual machine 120 may compare a size metric value derived from the size of the mapping data 128 to a maximum size threshold. If the size metric value exceeds the maximum size threshold, the then memory management process invoked by the virtual machine (or, alternatively by the storage machines storing the data) may increase the data subset size so that the amount of metadata required for the logical to physical mapping is reduced.
- the rate of data access blocks and the size metric value derived from the size of the mapping data 128 may both be used to manage the size of the data subsets. Trade-offs may be determined based on weightings that indicate the relative importance of the two performance considerations.
- the process 200 enables read and write access to the data by the virtual machine according to the logical addressing ( 206 ). For example, when no data subsets are being migrated, the virtual machine 120 has access to all data subsets of the guest data 142 .
- the process 200 determines that a first data subset stored on the first physical machine is to be migrated to a third physical machine separate from the first physical machine and the second physical machine ( 208 ). For example, some or all of the data stored on the storage machine 140 may need to be migrated. Again, a variety of events may require migration of some of the data or all of the data. In this example, assume that one data subset, indicated by the data subset index 2, shown in phantom in FIG. 1 , is to be migrated from the storage machine 140 to the storage machine 150 .
- the process 200 blocks read and write access by the virtual machine to the first data subset and maintains read and write access by the virtual machine to the other data subsets ( 210 ).
- the blocking may be done by the storage machine 140 .
- the storage machine 140 may send a notification to the virtual machine 120 so that it does not have access to data stored in the data subset undergoing migration, and the virtual machine 120 may then hold any read or write operations until it receives a notification that the migration is complete from either the storage machine 140 or the storage machine 150 , as indicated by arrows 3 A and 3 B, respectively.
- the storage machine 140 may not notify the virtual machine 120 of the migration, and the virtual machine 120 is only notified when the migration is complete. Should the virtual machine 120 request data from the data subset when the data subset is in flight, it may then be notified of the migration, and/or redirected to the second storage machine 150 .
- the data subset is migrated directly from the storage machine 140 to the storage machine 150 , and not through the host machine 110 .
- the data subset may be transferred through the host machine 110 .
- the data subsets are transferred to the virtual machine and then sent from the virtual machine 120 to the storage machine 150 . This is shown in FIG. 1 by arrows 4 A and 4 B, which are indicative of the actual data path of the migration indicated by arrow 2 .
- the virtual machine 120 may select the second storage machine 150 from one of multiple different storage machines available. This latter implementation facilitates “stateless” storage machines that store the data subsets without reference to an address of the virtual machine or any other storage machine, and without having to track a state of a migration and identify itself as in a “migration state.” Instead, management of data storage is handled by the virtual machine 120 .
- the process 200 migrates, from the first physical machine to the third physical machine, the first data subset to store the data subset on the third physical machine ( 212 ).
- the data subset may be sent directly from the storage machine 140 to the storage machine 150 , or, alternatively, may be fetched by the virtual machine 120 from the first storage machine 140 and then sent to the second storage machine 150 .
- the process 200 updates the mapping by logically mapping an address range for the first data subset on the third physical machine to the respective logical address range for the virtual machine ( 214 ).
- the data used to update the mapping depends on the implementation used. For example, in the implementation in which the virtual or host machine controls the migration, the virtual or host machine can update the mapping data based on the address of the storage machine to which the virtual machine sent the data subset. In the implementations in which the storage machines control the migration of the data subset, the virtual or host machine can update the mapping data based on a notification received by the virtual machine that indicates the address of the storage machine to which the data subset was sent.
- FIG. 3 is a flow chart of an example process 300 for the live migration of data in which data subsets are migrated directly from a first physical machine to a third physical machine.
- the process 300 may be implemented in one or both of the storage machines 140 and 150 .
- the process 300 determines that the first data subset stored on the first physical machine is to be migrated to the third physical machine ( 302 ).
- the storage machine 140 may determine that it is to go offline for maintenance and needs to migrate all the data stored at the storage machine, or that it has reached a storage capacity limit and needs to migrate a portion of the data stored at the storage machine.
- the process 300 blocks read and write access by the virtual machine to the first data subset and maintains read and write access by the virtual machine to the other data subsets ( 304 ).
- the storage machine 140 sends a notification to the virtual machine identifying the data subset that is being migrated and instructing the virtual machine to not attempt to write to the data subset or read the data subset until it is notified of the successful migration. This is indicated by the arrow 3 A of FIG. 1 .
- the process 300 migrates, from the first physical machine directly to the third physical machine, the first data subset ( 306 ).
- the storage machine 140 sends the data subset to the second storage machine 150 without involving the host machine 110 as an intermediary.
- the process 300 provides a notification to the virtual machine that the migration is complete and enables read and write access to the first data subset ( 308 ).
- the first storage machine 140 may receive an acknowledgement from the second storage machine 150 of the successful receipt of the data subset, and in turn may send a notification of the migration of the data subset and the address of the second storage machine 150 to the virtual machine 120 .
- the virtual machine 120 may then update its mapping data 128 and resume access to the data subset at the new location on the storage machine 150 .
- the second storage machine may send the notification of the migration of the data subset and the address of the second storage machine 150 to the virtual machine 120 .
- the virtual machine 120 may then update its mapping data 128 and resume access to the data subset at the new location on the storage machine 150 .
- FIG. 4 is a flow chart of an example process 400 for the live migration of data in which data subsets are migrated from a first physical machine to a third physical machine and through the physical machine in which the virtual machine is implemented.
- the process may be implemented in the virtual machine 120 (or host machine 110 ).
- the process 400 determines that the first data subset stored on the first physical machine is to be migrated to the third physical machine ( 402 ).
- the virtual machine 120 may determine that the storage machine 140 has a high latency; or may determine that a load balancing operation is necessary; or may even receive a notification from the first storage machine 140 that the first storage machine 140 is going offline for maintenance and needs to migrate the data stored for the virtual machine 120 .
- the process 400 instructs the first physical machine to migrate the first data subset to the third physical machine ( 404 ).
- the virtual machine 120 instructs the storage machine to migrate the data subset to the storage machine 150 .
- the virtual machine 120 will also not access the data subset until the migration is complete.
- the virtual machine 120 may receive the data subset from the first storage machine 140 and send the data subset to the storage machine 150 , as indicated by arrows 3 A and 3 B. In other implementations, the virtual machine 120 may instruct the storage machine 140 to send the data subset directly to the storage machine 150 .
- the process 400 updates the mapping by logically mapping an address range for the first data subset on the third physical machine to the respective logical address range for the virtual machine ( 406 ). For example, upon receiving a notification of a successful migration, e.g., from an acknowledgement message from the storage machine 150 , the virtual machine 120 updates the mapping data 128 and restores access to the data subset.
- the virtual machine may calculate an expected time of completion of the migration. After the expected time has passed, the virtual machine may attempt to access the data subset. If unsuccessful, it may wait for another period of time, or may instead invoke a memory error event.
- data subsets may be realized at a block or page level, and a “watermark” is used such that all addresses below X are on the first physical machine, and all addresses at or above X are on the third physical machine (or, when data is stored in three or more physical machines, contiguous address ranges may be used for each physical machine).
- the value of X is updated in the data processing process as data are migrated. This can eliminate the data processing process mapping of logical address to data subset, and instead maps partitions of the physical machines storing data according to the watermark value of X. Accordingly, metadata requirements to realize the mapping data 128 are reduced.
- pre-copy and post-copy migration techniques may be used on a per-data subset basis. This implementation reduces or eliminates the waiting period of a virtual machine during migration at the expense of system complexity.
- Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.
- Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus.
- a computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them.
- a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal.
- the computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).
- the operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.
- the term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing.
- the apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
- the apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them.
- the apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.
- a computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment.
- a computer program may, but need not, correspond to a file in a file system.
- a program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code).
- a computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
- the processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output.
- the processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., a FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
- special purpose logic circuitry e.g., a FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
- processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer.
- a processor will receive instructions and data from a read-only memory or a random access memory or both.
- the essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data.
- a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks.
- mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks.
- a computer need not have such devices.
- a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few.
- Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.
- the processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
- a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer.
- a display device e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor
- keyboard and a pointing device e.g., a mouse or a trackball
- Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
- a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a
- Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components.
- the components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network.
- Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).
- LAN local area network
- WAN wide area network
- inter-network e.g., the Internet
- peer-to-peer networks e.g., ad hoc peer-to-peer networks.
- the computing system can include users and servers.
- a user and server are generally remote from each other and typically interact through a communication network. The relationship of user and server arises by virtue of computer programs running on the respective computers and having a user-server relationship to each other.
- a server transmits data (e.g., an HTML, page) to a user device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the user device).
- Data generated at the user device e.g., a result of the user interaction
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Financial Or Insurance-Related Operations Such As Payment And Settlement (AREA)
Abstract
Description
- Cloud computing refers to network-based computing in which collections of servers housed in data centers or “server farms” provide computational resources and data storage as needed to remote end users. Some cloud computing services provide access to software applications such as word processors and other commonly used applications to end users who interface with the applications through web browsers or other client-side software. Users' electronic data files are usually stored in the server farm rather than on the users' computing devices. Maintaining software applications and user data on a server farm simplifies management of end user computing devices. Some cloud computing services allow end users to execute software applications in virtual machines. In a public cloud computing environment, multiple users are able to launch virtual machines (VMs).
- Often times the data for a particular virtual machine is stored on one or more physical machines that are separate from the physical machine on which the virtual machine is instantiated. For a variety of reasons—load sharing, server maintenance, etc.—some or all of the data stored on a particular physical machine may be migrated to another physical machine. Live-migration of data is the process of moving data off of one physical machine to another physical machine while the virtual machine (or alternatively, a non-virtual processing entity) performs arbitrary reads and writes on the data.
- This specification relates to live migration of data.
- This document describes a systematic method and system for moving data off of a storage computer to another storage computer while providing consistent and high performance access to the data to a third-party processing device (e.g., a virtual machine or an application) that is accessing the data remotely from another physical machine.
- In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of storing, in a first physical machine, data for a data processing process running on a second physical machine that is separate from the first physical machine, the storing data comprising storing the data according to a plurality of data subsets that are each exclusive of each other; for each data subset, logically mapping in a mapping, by the data processing process, an address range for the data subset on the first physical machine to a respective logical address range for the data processing process; enabling read and write access to the data by the data processing process according to the logical addressing; determining that a first data subset stored on the first physical machine is to be migrated to a third physical machine separate from the first physical machine and the second physical machine, and in response: blocking read and write access by the data processing process to the first data subset while maintaining read and write access by the data processing process to the other data subsets, migrating, from the first physical machine to the third physical machine, the first data subset to store the data subset on the third physical machine, and updating the mapping by logically mapping, by the data processing process, an address range for the first data subset on the third physical machine to the respective logical address range for the data processing process. Other embodiments of this aspect include corresponding systems, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.
- In general, another aspect of the subject matter described in this specification can be embodied in a system that includes a first physical machine storing data according to a plurality of data subsets that are each exclusive of each other; a second physical machine that is separate from the first physical machine and a virtual machine on the second physical machine having read and write access to the data stored on the first physical machine and that, for each data subset, logically maps, in a mapping, an address range for the data subset on the first physical machine to a respective logical address range for the virtual machine; wherein in response to a determination that a first data subset stored on the first physical machine is to be migrated to a third physical machine separate from the first physical machine and the second physical machine: the first data subset is migrated from the first physical machine to the third physical machine; read and write access to the first data subset for the virtual machine is blocked during the migration while read and write access by the virtual machine to the other data subsets is maintained; and the mapping is updated by the virtual machine to logically map an address range for the first data subset on the third physical machine to the respective logical address range for the virtual machine; and read and write access to the first data subset for the virtual machine is restored after the migration of the first data subset from the first physical machine to the third physical machine. Other embodiments of this aspect include corresponding methods, apparatus, and computer programs, configured to perform the actions of the methods, encoded on computer storage devices.
- Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. The method provides better performance guarantees to the virtual machine than the other methods of live-migration such as pre-copy and post-copy live migration. For example, pre-copy live-migration considers the whole data address space as a whole migrating unit, and thus the entire address space exists completely on the originating side or completely on the destination side of the migration. To provide “live” access to the data while the migration is ongoing, the virtual machine is allowed to continue reading/writing the data, but writes are tracked so that changes can be resent to the receiving physical machine(s) of the live migration. This resending takes more read bandwidth, network bandwidth, CPU processing, and time. Furthermore, a busy virtual machine will typically be changing the data faster than the changes can be sent over the network to the destination; in such situations, the virtual machines access rate must be slowed, resulting in performance degradation.
- With post-copy live-migration, the virtual machine is informed of the destination physical machine and requests the destination physical machine for the data. The destination physical machine provides the data if the data is stored on the destination physical machine; otherwise the destination physical machine fetches the data from the originating physical machine which the data is being migrated and then provides the data. When the data must be fetched from the originating physical machine the virtual machine experiences an increased latency. With many accesses from the destination physical machine to the originating physical machine, there is a significant overall bandwidth performance degradation.
- The methods and systems described below, however, overcome some or all of these operational characteristics, resulting in an improvement in the technology area of data storage and management. By processing the migrating data in data subsets, which may be a chunk of X MB of data, or a page of data, etc., the migration is much more granular than pre-copy live-migration. The data subset undergoing migration is precluded from being accessed by the virtual machine. Thus, tracking of writes need not be performed. Once a data subset is migrated, it does not need to be resent because all future accesses go directly to the destination side.
- The virtual machine experiences virtually no impact to performance if it is not accessing the data subset being migrated. As the data subset size decreases, the amount of time the virtual machine must be blocked while waiting for the data subset to migrate decreases. Conversely, as the data subset size increases, the amount of metadata for storage mapping decreases. Thus, by selectively evaluating the trade-off of wait time v. mapping maintenance, a system administration may tailor data subset size for a particular application that results in an improved migration performance operation for the application.
- While the migration techniques described below do utilize some bandwidth for overhead, the amount utilized is relatively small compared to the bandwidth utilized by pre-copy or post-copy migration. This is still yet another improvement to the technological field of data migration.
- Because read and write access is blocked for the data subset undergoing migration, no overhead mechanism for tracking changes to the data subset is needed, nor is there a need to specifically order virtual machine accesses to the data subset for the purpose of migration. For example, if the data store is a disk and the virtual machine performs a write to a location while there is an outstanding read to the same location for the purpose of migration, then the result of the read access is undefined. The systems and methods herein preclude concurrent access to the same location by blocking the virtual machine from accessing the specific region being migrated.
- The live migration described below may be stopped at any time without losing progress. This is in contrast to pre-copy live migration, which must completely start over if stopped, and also in contrast to post-copy live migration, which cannot be stopped after accesses for the virtual machine are switched to the destination side.
- The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.
-
FIG. 1 is a block diagram of a cloud-based environment in which data may undergo live migration. -
FIG. 2 is a flow chart of an example process for the live migration of data. -
FIG. 3 is a flow chart of an example process for the live migration of data in which data subsets are migrated directly from a first physical machine to a third physical machine. -
FIG. 4 is a flow chart of an example process for the live migration of data in which data subsets are migrated from a first physical machine to a third physical machine and through the physical machine in which the virtual machine is implemented. - Like reference numbers and designations in the various drawings indicate like elements.
- Overview
- A first physical machine stores data according to multiple data subsets that are each exclusive of each other. The data is stored for a data processing process, such as a virtual machine, running on a second physical machine that is separate from the first physical machine. The data processing process has read and write access to the data stored on the first physical machine, and for each data subset logically maps, in a mapping, an address range for a data subset on the first physical machine to a respective logical address range for the data processing process.
- When data from the first physical machine is to be migrated to the third physical machine, the following steps are taken. The data are migrated in data subsets. For each data subset undergoing migration (e.g., in the process of being “in flight” from the first physical machine to the third physical machine), read and write access by the data processing process to the data subset is blocked. However, read and write access by the data processing process to the other data subsets not undergoing data migration is maintained. In this way the data processing process may still access much of the data that is stored in the first physical machine. In response to a migration of a particular data subset from the first physical machine to the third physical machine, the mapping is updated by the data processing process. In particular, an address range for the particular data subset on the third physical machine is mapped to the respective logical address range for the data processing process, and the pre-migration mapping of the particular data subset is thus replaced by the updated mapping. Read and write access to the first data subset for the data processing process is restored after the migration of the first data subset from the first physical machine to the third physical machine. Thereafter, the data processing process accesses the third physical machine when data stored in the migrated data subset is needed by the data processing process.
- Several variations to the above process may be advantageous, depending on system requirements. For example, a data processing process may not be notified of a migration of a data subset. Should the data processing process request access to data stored in the data subset from the first physical machine, it will then receive a response informing it that the data subset is now stored on a third physical machine (or currently undergoing migration to the third physical machine). If the data subset is currently undergoing migration, the third physical machine may block access to the data subset until the migration is complete.
- Alternatively the data processing process may be proactively informed of the migration and may calculate an expected time of completion of the migration. The data processing process may then wait to request the data until the expected time of completion.
- A data subset may be sent directly from the first physical machine to the third physical machine, and then one of the first or third physical machines will inform the data processing process of the new location of the data subset when the migration is complete. Alternatively, however, the data subset may be passed through the data processing process acting as an intermediary, and the data processing process is responsible for migration. This allows the data processing process to be up-to-date about where data resides in near real time.
- In yet another implementation, the data processing process controls migration but the data subset is sent directly from one storage machine to the other. For example, the virtual machine sends a “transfer address range” message to first storage machine instructing the first storage machine to read a specific address range specified by the transfer address range and write that data to another address range a second storage machine. The storage machines are stateless, but by the instructions sent from the virtual machine, the data-subset is sent directly from the first storage machine to the second storage machine. This results in less data transfer than passing data through the VM, and is more scalable if many storage machines are involved.
- Finally, the data subsets may be realized by a variety of different data management techniques. For example, a data subset may be a fixed size of memory unrelated to page or block size, e.g., 1 MB, 10 MB, or even 1 GB. Alternatively, data subsets may be realized at a block or page level, and a “watermark” is used such that all addresses below X are on the first physical machine, and all addresses at or above X are on the third physical machine. The value of X is updated in the data processing process as data are migrated. This can eliminate the data processing process mapping of logical address to data subset, and instead partitions the physical machines storing data according to the watermark value of X.
- These features and other features are described in more detail below.
- Example Operating Environment
-
FIG. 1 is a block diagram of a cloud-basedenvironment 100 in which data may undergo live migration. In the written description below, an example implementation of a virtual machine is described. However, data migration of data for some of data processing process, such a client-based application have cloud-bases storage, or a cloud-based application having cloud based storage, may also be facilitated by the systems and methods described below. - A
host machine 110, which is a physical machine, in the cloud-basedenvironment 100, can contain one or more data processing apparatuses such as rack mounted servers or other computing devices.Storage machine host machine 110. Thestorage machines host machine 110 communicating through anetwork 102. - The
host machine 110 executes ahost operating system 112 that manages host machine resources. In this example, thehost operating systems 112 run software that virtualizes the underlying host machine hardware and manages concurrent execution of one or morevirtual machines 120. As illustrated inFIG. 1 , thehost operating system 112 manages onevirtual machine 120. A host machine can, in general, manage larger quantities of virtual machines; however, the quantity may be limited based on physical resources of the host machine. For simplicity, only onevirtual machine 120 is shown inFIG. 1 . - The
virtual machine 120 uses a simulated version of an underlying host machine hardware, which can be referred to asvirtual hardware 122. Software that is executed by thevirtual hardware 122 can be referred to as guest software, e.g., aguest operating system 124 andguest applications 126. In some implementations, guest software cannot determine if it is being executed by virtual hardware or by a physical host machine. A host machine's microprocessor(s) can include processor-level mechanisms to enable virtual hardware to execute software applications efficiently by allowing guest software instructions to be executed directly on the host machine's microprocessor without requiring code-rewriting, recompilation, or instruction emulation. - The
host machine 120 is allocated a set of virtual memory pages from the virtual memory of the underlyinghost operating system 112 and is allocated virtual disk blocks from one or more virtual disk drives for use by the guest software executing on the virtual machine. The actual physical storage need not be on thehost machine 110, and in the example shown, the storage is realized by thestorage machine 140. - In some implementations, virtual disk blocks are allocated on physical disk drives managed by the storage machine and communicating with the
host machine 110 through thenetwork 102. Thevirtual machine 120 can be allocated network addresses through which their respective processes can communicate with other processes via thenetwork 102. - Assume that, initially, all the data for the
virtual machine 120 is stored on thestorage machine 140. This data for thevirtual machine 120 is referred to asguest data 142. Theguest data 142 need not initially be stored on a single physical machine, and can instead be initially stored across multiple storage machines. However, for simplicity of description, the starting point for this example is a single storage machine. - The
guest data 142 is stored according to multiple data subsets that are each exclusive of each other. As shown inFIG. 1 , theguest data 142 is stored indata subsets 144 and each data subset is illustratively indexed by one of theindices 0 . . . n. Themapping data 128 logically maps an address range for the data subset on thestorage machine 140 to a respective logical address range for thevirtual machine 120. Thus, by use of themapping data 128, thevirtual machine 120 can map a logical address space to a particular data subset stored on a particular physical machine. Finally, while themapping data 128 is illustrated as being within thevirtual machine 120, themapping data 128 may also be maintained by thehost operating system 112. - Events may occur that may cause some or all of the
guest data 142 to be migrated to one or more other storage machines. Such events may include astorage machine 140 preparing to go offline for service, which requires migration of all the data stored at the storage machine; load balancing, which requires the migration of at least a portion of the data stored at the storage machine; or quality of service requirements not being met, which may require the migration of at least a portion of the data stored at the storage machine. As will be described below, in some situations the storage machine may determine when a migration is necessary, and in other situations the virtual machine (or host machine) may determine when a migration is necessary. In still other situations, a process or entity external to the virtual machine, host machine and storage machines can also determine when a migration is necessary, and either the virtual machine can control the migration or the storage machines can control the migration, as described in general above and as will be described in more detail below. - A generalized process for the live migration of data, which is indicated by the arrow with
reference callout 2 inFIG. 1 , is described with reference toFIG. 2 . One example process in which the physical storage machines partially (or fully) control the migration is described with reference toFIG. 3 , and is indicated by the arrow withreference callout 2 in combination with the arrows withreference callouts FIG. 1 . Finally, an example process in which the virtual machine (or host machine) partially (or fully) controls the migration is described with reference toFIG. 4 , and is indicated by the arrow withreference callout 2 in combination with the arrows withreference callouts FIG. 1 . - For simplicity, the migration example described below will detail the migrating of data to one other physical machine—
storage machine 150. However, should the guest data be stored on multiple storage machines, the guest data may be migrated from one of the storage machines to another storage machine that currently is storing some of the guest data, or a new storage machine that is not currently storing the guest data for thevirtual machine 120. - Live Migration from First Physical Machine to Second Physical Machine
-
FIG. 2 is a flow chart of anexample process 200 for the live migration of data. Theprocess 200 may be implemented in thephysical machines FIG. 1 . - The
process 200 stores, in a first physical machine, data for a virtual machine running on a second physical machine that is separate from the first physical machine (202). For example, as shown inFIG. 1 , thedata 142 is stored according to data subsets that are each exclusive of each other. A “data subset” of thedata 142 can be either a predefined data construct, such as a block, sector or page, or may be an arbitrarily defined unit of data, such as a 1 KB, 1 MB, LOMB, or even 1 GB amount of data. In the case of the former, the block or page may be of the size as virtually realized for the virtual machine, or, alternatively, may be of a physical size as determined by the physical hardware used. - The
process 200, for each data subset, logically maps, by the virtual machine, an address range for the data subset on the first physical machine to a respective logical address range for the virtual machine (204). For example, the virtual machine (or, alternatively, the host machine) logically maps the address at which the data appears to reside from the perspective of the virtual machine to the physical address at which the data actually resides. Any appropriate address translation process that can map a logical address in a data processing process on a first machine to a physical address on a second machine separate from the first machine can be used. - In the case of the latter implementation, in which the data subset is of an arbitrary size, several factors may be considered by an administrator when determining the size. The smaller the size of the data subset, the
more mapping data 128 will be required. However, because the amount of time the data subset is in-flight during migration decreases as the size of the data subset decreases, smaller data subsets tend to result in fewer read and write delays that may occur when thevirtual machine 120 attempts to access the data subset undergoing a migration. - In some implementations, the
virtual machine 120, orhost machine 110, may compare the rate of data access blocks due to migrations to a maximum block rate threshold. If the rate exceeds a maximum block rate threshold, then a memory management process is invoked by the virtual machine (or, alternatively by the storage machines storing the data) to reduce the data subset size. Thereafter, a new rate of data access blocks is determined. The process may continue until the rate is below the maximum block rate threshold. - In other implementations, the
virtual machine 120, orhost machine 110, may compare a size metric value derived from the size of themapping data 128 to a maximum size threshold. If the size metric value exceeds the maximum size threshold, the then memory management process invoked by the virtual machine (or, alternatively by the storage machines storing the data) may increase the data subset size so that the amount of metadata required for the logical to physical mapping is reduced. - In still other implementations, the rate of data access blocks and the size metric value derived from the size of the
mapping data 128 may both be used to manage the size of the data subsets. Trade-offs may be determined based on weightings that indicate the relative importance of the two performance considerations. - The
process 200 enables read and write access to the data by the virtual machine according to the logical addressing (206). For example, when no data subsets are being migrated, thevirtual machine 120 has access to all data subsets of theguest data 142. - The
process 200 determines that a first data subset stored on the first physical machine is to be migrated to a third physical machine separate from the first physical machine and the second physical machine (208). For example, some or all of the data stored on thestorage machine 140 may need to be migrated. Again, a variety of events may require migration of some of the data or all of the data. In this example, assume that one data subset, indicated by thedata subset index 2, shown in phantom inFIG. 1 , is to be migrated from thestorage machine 140 to thestorage machine 150. - The
process 200 blocks read and write access by the virtual machine to the first data subset and maintains read and write access by the virtual machine to the other data subsets (210). In some implementations, the blocking may be done by thestorage machine 140. Thestorage machine 140 may send a notification to thevirtual machine 120 so that it does not have access to data stored in the data subset undergoing migration, and thevirtual machine 120 may then hold any read or write operations until it receives a notification that the migration is complete from either thestorage machine 140 or thestorage machine 150, as indicated byarrows - Alternative, the
storage machine 140 may not notify thevirtual machine 120 of the migration, and thevirtual machine 120 is only notified when the migration is complete. Should thevirtual machine 120 request data from the data subset when the data subset is in flight, it may then be notified of the migration, and/or redirected to thesecond storage machine 150. - In the example implementations above, the data subset is migrated directly from the
storage machine 140 to thestorage machine 150, and not through thehost machine 110. However, in other implementations, the data subset may be transferred through thehost machine 110. For example, in implementations in which thevirtual machine 120 handles the migration of the data subsets, the data subsets are transferred to the virtual machine and then sent from thevirtual machine 120 to thestorage machine 150. This is shown inFIG. 1 byarrows arrow 2. - The virtual machine 120 (or host machine 110) may select the
second storage machine 150 from one of multiple different storage machines available. This latter implementation facilitates “stateless” storage machines that store the data subsets without reference to an address of the virtual machine or any other storage machine, and without having to track a state of a migration and identify itself as in a “migration state.” Instead, management of data storage is handled by thevirtual machine 120. - The
process 200 migrates, from the first physical machine to the third physical machine, the first data subset to store the data subset on the third physical machine (212). As described above, the data subset may be sent directly from thestorage machine 140 to thestorage machine 150, or, alternatively, may be fetched by thevirtual machine 120 from thefirst storage machine 140 and then sent to thesecond storage machine 150. - The
process 200 updates the mapping by logically mapping an address range for the first data subset on the third physical machine to the respective logical address range for the virtual machine (214). The data used to update the mapping depends on the implementation used. For example, in the implementation in which the virtual or host machine controls the migration, the virtual or host machine can update the mapping data based on the address of the storage machine to which the virtual machine sent the data subset. In the implementations in which the storage machines control the migration of the data subset, the virtual or host machine can update the mapping data based on a notification received by the virtual machine that indicates the address of the storage machine to which the data subset was sent. - Live Migration Subject to Storage Machine Control
-
FIG. 3 is a flow chart of anexample process 300 for the live migration of data in which data subsets are migrated directly from a first physical machine to a third physical machine. Theprocess 300 may be implemented in one or both of thestorage machines - The
process 300 determines that the first data subset stored on the first physical machine is to be migrated to the third physical machine (302). For example, thestorage machine 140 may determine that it is to go offline for maintenance and needs to migrate all the data stored at the storage machine, or that it has reached a storage capacity limit and needs to migrate a portion of the data stored at the storage machine. - The
process 300 blocks read and write access by the virtual machine to the first data subset and maintains read and write access by the virtual machine to the other data subsets (304). For example, thestorage machine 140 sends a notification to the virtual machine identifying the data subset that is being migrated and instructing the virtual machine to not attempt to write to the data subset or read the data subset until it is notified of the successful migration. This is indicated by thearrow 3A ofFIG. 1 . - The
process 300 migrates, from the first physical machine directly to the third physical machine, the first data subset (306). For example, thestorage machine 140 sends the data subset to thesecond storage machine 150 without involving thehost machine 110 as an intermediary. - The
process 300 provides a notification to the virtual machine that the migration is complete and enables read and write access to the first data subset (308). For example, thefirst storage machine 140 may receive an acknowledgement from thesecond storage machine 150 of the successful receipt of the data subset, and in turn may send a notification of the migration of the data subset and the address of thesecond storage machine 150 to thevirtual machine 120. Thevirtual machine 120 may then update itsmapping data 128 and resume access to the data subset at the new location on thestorage machine 150. Alternatively, after thesecond storage machine 150 successfully receives the data subset, the second storage machine may send the notification of the migration of the data subset and the address of thesecond storage machine 150 to thevirtual machine 120. Thevirtual machine 120 may then update itsmapping data 128 and resume access to the data subset at the new location on thestorage machine 150. - Live Migration Subject to Virtual Machine or Host Machine Control
-
FIG. 4 is a flow chart of anexample process 400 for the live migration of data in which data subsets are migrated from a first physical machine to a third physical machine and through the physical machine in which the virtual machine is implemented. The process may be implemented in the virtual machine 120 (or host machine 110). - The
process 400 determines that the first data subset stored on the first physical machine is to be migrated to the third physical machine (402). For example, thevirtual machine 120 may determine that thestorage machine 140 has a high latency; or may determine that a load balancing operation is necessary; or may even receive a notification from thefirst storage machine 140 that thefirst storage machine 140 is going offline for maintenance and needs to migrate the data stored for thevirtual machine 120. - The
process 400 instructs the first physical machine to migrate the first data subset to the third physical machine (404). For example, thevirtual machine 120 instructs the storage machine to migrate the data subset to thestorage machine 150. Thevirtual machine 120 will also not access the data subset until the migration is complete. - In some implementations, the
virtual machine 120 may receive the data subset from thefirst storage machine 140 and send the data subset to thestorage machine 150, as indicated byarrows virtual machine 120 may instruct thestorage machine 140 to send the data subset directly to thestorage machine 150. - The
process 400 updates the mapping by logically mapping an address range for the first data subset on the third physical machine to the respective logical address range for the virtual machine (406). For example, upon receiving a notification of a successful migration, e.g., from an acknowledgement message from thestorage machine 150, thevirtual machine 120 updates themapping data 128 and restores access to the data subset. - Variations to the example system and processes described above may be implemented to realize additional features. For example, instead of waiting for a notification of successful migration, the virtual machine may calculate an expected time of completion of the migration. After the expected time has passed, the virtual machine may attempt to access the data subset. If unsuccessful, it may wait for another period of time, or may instead invoke a memory error event.
- In other implementations, data subsets may be realized at a block or page level, and a “watermark” is used such that all addresses below X are on the first physical machine, and all addresses at or above X are on the third physical machine (or, when data is stored in three or more physical machines, contiguous address ranges may be used for each physical machine). The value of X is updated in the data processing process as data are migrated. This can eliminate the data processing process mapping of logical address to data subset, and instead maps partitions of the physical machines storing data according to the watermark value of X. Accordingly, metadata requirements to realize the
mapping data 128 are reduced. - In other implementations, pre-copy and post-copy migration techniques may be used on a per-data subset basis. This implementation reduces or eliminates the waiting period of a virtual machine during migration at the expense of system complexity.
- The examples above are described in the context of a cloud-based system or in data centers. However, the systems and methods described herein can be utilized in any system that manages stored data remotely from a computer on which an application or virtual machine that accesses the data is running.
- Embodiments of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on computer storage medium for execution by, or to control the operation of, data processing apparatus.
- A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially-generated propagated signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).
- The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.
- The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.
- A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub-programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
- The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., a FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).
- Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
- To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's user device in response to requests received from the web browser.
- Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).
- The computing system can include users and servers. A user and server are generally remote from each other and typically interact through a communication network. The relationship of user and server arises by virtue of computer programs running on the respective computers and having a user-server relationship to each other. In some embodiments, a server transmits data (e.g., an HTML, page) to a user device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the user device). Data generated at the user device (e.g., a result of the user interaction) can be received from the user device at the server.
- While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any features or of what may be claimed, but rather as descriptions of features specific to particular embodiments. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
- Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
- Thus, particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.
Claims (21)
Priority Applications (18)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/071,852 US9936019B2 (en) | 2016-03-16 | 2016-03-16 | Efficient live-migration of remotely accessed data |
AU2016398043A AU2016398043B2 (en) | 2016-03-16 | 2016-12-02 | Efficient live-migration of remotely accessed data |
SG11201807848PA SG11201807848PA (en) | 2016-03-16 | 2016-12-02 | Efficient live-migration of remotely accessed data |
KR1020197017825A KR102055325B1 (en) | 2016-03-16 | 2016-12-02 | Efficient live-migration of remotely accessed data |
KR1020187026783A KR101993915B1 (en) | 2016-03-16 | 2016-12-02 | Efficient live-transfer of remotely accessed data |
SG10202100763RA SG10202100763RA (en) | 2016-03-16 | 2016-12-02 | Efficient live-migration of remotely accessed data |
EP16816813.6A EP3414661B1 (en) | 2016-03-16 | 2016-12-02 | Efficient live-migration of remotely accessed data |
CN201680083580.2A CN108780404A (en) | 2016-03-16 | 2016-12-02 | Effective real-time migration of remote access data |
JP2018548837A JP6728381B2 (en) | 2016-03-16 | 2016-12-02 | Efficient live migration of remotely accessed data |
PCT/US2016/064738 WO2017160359A1 (en) | 2016-03-16 | 2016-12-02 | Efficient live-migration of remotely accessed data |
CN202111145963.7A CN113821348B (en) | 2016-03-16 | 2016-12-02 | Efficient live migration of remotely accessed data |
US15/902,844 US10187466B2 (en) | 2016-03-16 | 2018-02-22 | Efficient live-migration of remotely accessed data |
US16/250,822 US10645160B2 (en) | 2016-03-16 | 2019-01-17 | Efficient live-migration of remotely accessed data |
AU2019257477A AU2019257477A1 (en) | 2016-03-16 | 2019-10-31 | Efficient live-migration of remotely accessed data |
US16/734,037 US11005934B2 (en) | 2016-03-16 | 2020-01-03 | Efficient live-migration of remotely accessed data |
JP2020114071A JP7174739B2 (en) | 2016-03-16 | 2020-07-01 | Efficient live migration of remotely accessed data |
AU2020260536A AU2020260536B2 (en) | 2016-03-16 | 2020-10-30 | Efficient live-migration of remotely accessed data |
US17/224,239 US11824926B2 (en) | 2016-03-16 | 2021-04-07 | Efficient live-migration of remotely accessed data |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/071,852 US9936019B2 (en) | 2016-03-16 | 2016-03-16 | Efficient live-migration of remotely accessed data |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/902,844 Continuation US10187466B2 (en) | 2016-03-16 | 2018-02-22 | Efficient live-migration of remotely accessed data |
Publications (2)
Publication Number | Publication Date |
---|---|
US20170272515A1 true US20170272515A1 (en) | 2017-09-21 |
US9936019B2 US9936019B2 (en) | 2018-04-03 |
Family
ID=57610407
Family Applications (5)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/071,852 Active 2036-09-04 US9936019B2 (en) | 2016-03-16 | 2016-03-16 | Efficient live-migration of remotely accessed data |
US15/902,844 Active US10187466B2 (en) | 2016-03-16 | 2018-02-22 | Efficient live-migration of remotely accessed data |
US16/250,822 Active US10645160B2 (en) | 2016-03-16 | 2019-01-17 | Efficient live-migration of remotely accessed data |
US16/734,037 Active US11005934B2 (en) | 2016-03-16 | 2020-01-03 | Efficient live-migration of remotely accessed data |
US17/224,239 Active US11824926B2 (en) | 2016-03-16 | 2021-04-07 | Efficient live-migration of remotely accessed data |
Family Applications After (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/902,844 Active US10187466B2 (en) | 2016-03-16 | 2018-02-22 | Efficient live-migration of remotely accessed data |
US16/250,822 Active US10645160B2 (en) | 2016-03-16 | 2019-01-17 | Efficient live-migration of remotely accessed data |
US16/734,037 Active US11005934B2 (en) | 2016-03-16 | 2020-01-03 | Efficient live-migration of remotely accessed data |
US17/224,239 Active US11824926B2 (en) | 2016-03-16 | 2021-04-07 | Efficient live-migration of remotely accessed data |
Country Status (8)
Country | Link |
---|---|
US (5) | US9936019B2 (en) |
EP (1) | EP3414661B1 (en) |
JP (2) | JP6728381B2 (en) |
KR (2) | KR101993915B1 (en) |
CN (2) | CN108780404A (en) |
AU (3) | AU2016398043B2 (en) |
SG (2) | SG10202100763RA (en) |
WO (1) | WO2017160359A1 (en) |
Cited By (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108388599A (en) * | 2018-02-01 | 2018-08-10 | 平安科技(深圳)有限公司 | Electronic device, Data Migration and call method and storage medium |
US20190095232A1 (en) * | 2017-09-22 | 2019-03-28 | Fujitsu Limited | Non-transitory computer-readable recording medium, adjustment device, and adjustment method |
US20190265902A1 (en) * | 2018-02-28 | 2019-08-29 | International Business Machines Corporation | Live migration of applications using capi flash |
CN110347483A (en) * | 2018-04-08 | 2019-10-18 | 中兴通讯股份有限公司 | Physical machine is to virtual machine migration method, device and storage medium |
US10592484B1 (en) * | 2017-01-06 | 2020-03-17 | Sprint Communications Company L.P. | Data migration between different lightweight directory access protocol (LDAP) based wireless communication subscriber data stores |
US20200159434A1 (en) * | 2018-11-19 | 2020-05-21 | Micron Technology, Inc. | Systems, devices, techniques, and methods for data migration |
US10782911B2 (en) | 2018-11-19 | 2020-09-22 | Micron Technology, Inc. | Data migration dynamic random access memory |
US11074099B2 (en) * | 2018-02-06 | 2021-07-27 | Nutanix, Inc. | System and method for storage during virtual machine migration |
US11182090B2 (en) | 2018-11-19 | 2021-11-23 | Micron Technology, Inc. | Systems, devices, and methods for data migration |
US11256437B2 (en) | 2018-11-19 | 2022-02-22 | Micron Technology, Inc. | Data migration for memory operation |
US20220229774A1 (en) * | 2021-01-15 | 2022-07-21 | Nutanix, Inc. | Just-in-time virtual per-vm swap space |
US11409619B2 (en) | 2020-04-29 | 2022-08-09 | The Research Foundation For The State University Of New York | Recovering a virtual machine after failure of post-copy live migration |
US11632319B2 (en) * | 2019-02-01 | 2023-04-18 | Nippon Telegraph And Telephone Corporation | Processing device and moving method |
US12068975B2 (en) * | 2020-09-22 | 2024-08-20 | Xi'an Zhongxing New Software Co., Ltd. | Resource scheduling method and system, electronic device, computer readable storage medium |
Families Citing this family (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11036392B2 (en) * | 2013-02-26 | 2021-06-15 | Pure Storage, Inc. | Determining when to use convergent encryption |
US9936019B2 (en) * | 2016-03-16 | 2018-04-03 | Google Llc | Efficient live-migration of remotely accessed data |
CN108959573B (en) * | 2018-07-05 | 2022-07-15 | 京东方科技集团股份有限公司 | Desktop cloud based data migration method and device, electronic equipment and storage medium |
CN108958889A (en) * | 2018-07-12 | 2018-12-07 | 郑州云海信息技术有限公司 | The management method and device of virtual machine in cloud data system |
US10924587B1 (en) | 2019-05-01 | 2021-02-16 | Amazon Technologies, Inc. | Live migration for highly available data stores |
US11086549B2 (en) | 2019-05-21 | 2021-08-10 | International Business Machines Corporation | Just-in-time data migration in a live system |
US10979303B1 (en) | 2019-06-06 | 2021-04-13 | Amazon Technologies, Inc. | Segmentation of maintenance on distributed systems |
US11119994B1 (en) | 2019-06-06 | 2021-09-14 | Amazon Technologies, Inc. | Record-by-record live migration using segmentation |
US10924429B1 (en) * | 2019-11-29 | 2021-02-16 | Amazon Technologies, Inc. | Using edge-optimized compute instances to execute user workloads at provider substrate extensions |
US11372827B2 (en) | 2020-05-06 | 2022-06-28 | Amazon Technologies, Inc. | Record-by-record live migration using a lock store |
CN112463132B (en) * | 2020-11-13 | 2023-06-06 | 四川新网银行股份有限公司 | Database switching tool and switching method |
CN112402979B (en) * | 2020-12-02 | 2023-11-17 | 网易(杭州)网络有限公司 | Game data processing method and device and electronic equipment |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8005788B2 (en) * | 2008-01-28 | 2011-08-23 | International Business Machines Corporation | System and method for legacy system component incremental migration |
US8769241B2 (en) * | 2009-12-04 | 2014-07-01 | Marvell World Trade Ltd. | Virtualization of non-volatile memory and hard disk drive as a single logical drive |
US9003159B2 (en) * | 2009-10-05 | 2015-04-07 | Marvell World Trade Ltd. | Data caching in non-volatile memory |
US20150261576A1 (en) * | 2014-03-17 | 2015-09-17 | Vmware, Inc. | Optimizing memory sharing in a virtualized computer system with address space layout randomization enabled in guest operating systems |
US9229878B2 (en) * | 2013-06-10 | 2016-01-05 | Red Hat Israel, Ltd. | Memory page offloading in multi-node computer systems |
US9465561B2 (en) * | 2013-04-18 | 2016-10-11 | Hitachi, Ltd. | Storage system and storage control method |
US9483298B2 (en) * | 2014-04-23 | 2016-11-01 | Vmware, Inc. | Converting virtual machine I/O requests |
Family Cites Families (50)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3138575B2 (en) * | 1993-09-02 | 2001-02-26 | 日本電気株式会社 | File copy transfer method |
US6405284B1 (en) * | 1998-10-23 | 2002-06-11 | Oracle Corporation | Distributing data across multiple data storage devices in a data storage system |
US6698017B1 (en) * | 1999-07-16 | 2004-02-24 | Nortel Networks Limited | Software migration on an active processing element |
US8051270B2 (en) * | 2005-05-23 | 2011-11-01 | Panasonic Corporation | Memory controller, nonvolatile storage device, nonvolatile storage system, and memory control method |
US8806480B2 (en) * | 2007-06-29 | 2014-08-12 | Microsoft Corporation | Virtual machine smart migration |
US8910152B1 (en) * | 2007-11-30 | 2014-12-09 | Hewlett-Packard Development Company, L.P. | Migrating a virtual machine by using a hot-plug event |
US9715401B2 (en) | 2008-09-15 | 2017-07-25 | International Business Machines Corporation | Securing live migration of a virtual machine from a secure virtualized computing environment, over an unsecured network, to a different virtualized computing environment |
US7996484B2 (en) * | 2008-12-11 | 2011-08-09 | Microsoft Corporation | Non-disruptive, reliable live migration of virtual machines with network data reception directly into virtual machines' memory |
WO2010126048A1 (en) | 2009-04-28 | 2010-11-04 | 日本電気株式会社 | Rapid movement system for virtual devices in a computing system, management device, and method and program therefor |
US8429647B2 (en) | 2009-05-06 | 2013-04-23 | Vmware, Inc. | Virtual machine migration across network by publishing routes to the associated virtual networks via virtual router after the start of migration of the virtual machine |
JP5621229B2 (en) | 2009-08-27 | 2014-11-12 | 日本電気株式会社 | Storage system, management method and program |
US8478725B2 (en) | 2009-09-14 | 2013-07-02 | Vmware, Inc. | Method and system for performing live migration of persistent data of a virtual machine |
US8327060B2 (en) * | 2009-11-30 | 2012-12-04 | Red Hat Israel, Ltd. | Mechanism for live migration of virtual machines with memory optimizations |
US8924675B1 (en) | 2010-09-24 | 2014-12-30 | Emc Corporation | Selective migration of physical data |
US8645653B2 (en) * | 2010-10-14 | 2014-02-04 | Hitachi, Ltd | Data migration system and data migration method |
CN102073462B (en) | 2010-11-29 | 2013-04-17 | 华为技术有限公司 | Virtual storage migration method and system and virtual machine monitor |
US20120159634A1 (en) | 2010-12-15 | 2012-06-21 | International Business Machines Corporation | Virtual machine migration |
US9612855B2 (en) | 2011-01-10 | 2017-04-04 | International Business Machines Corporation | Virtual machine migration based on the consent by the second virtual machine running of the target host |
US9003149B2 (en) * | 2011-05-26 | 2015-04-07 | International Business Machines Corporation | Transparent file system migration to a new physical location |
US8856191B2 (en) * | 2011-08-01 | 2014-10-07 | Infinidat Ltd. | Method of migrating stored data and system thereof |
US9116633B2 (en) | 2011-09-30 | 2015-08-25 | Commvault Systems, Inc. | Information management of virtual machines having mapped storage devices |
US9461881B2 (en) * | 2011-09-30 | 2016-10-04 | Commvault Systems, Inc. | Migration of existing computing systems to cloud computing sites or virtual machines |
US20130138764A1 (en) | 2011-11-30 | 2013-05-30 | Soumendu S. Satapathy | Method and system for virtual machine data migration |
US9397954B2 (en) | 2012-03-26 | 2016-07-19 | Oracle International Corporation | System and method for supporting live migration of virtual machines in an infiniband network |
US9164795B1 (en) * | 2012-03-30 | 2015-10-20 | Amazon Technologies, Inc. | Secure tunnel infrastructure between hosts in a hybrid network environment |
JP6028415B2 (en) | 2012-06-28 | 2016-11-16 | 日本電気株式会社 | Data migration control device, method and system for virtual server environment |
WO2014032233A1 (en) | 2012-08-29 | 2014-03-06 | 华为技术有限公司 | System and method for live migration of virtual machine |
US9372726B2 (en) | 2013-01-09 | 2016-06-21 | The Research Foundation For The State University Of New York | Gang migration of virtual machines using cluster-wide deduplication |
US9619258B2 (en) | 2013-01-21 | 2017-04-11 | International Business Machines Corporation | Live virtual machine migration quality of service |
US9405642B2 (en) * | 2013-01-29 | 2016-08-02 | Red Hat Israel, Ltd. | Providing virtual machine migration reliability using an intermediary storage device |
CN103198028B (en) * | 2013-03-18 | 2015-12-23 | 华为技术有限公司 | A kind of internal storage data moving method, Apparatus and system |
CN104243427B (en) * | 2013-06-19 | 2018-04-06 | 日电(中国)有限公司 | The online moving method of virtual machine, data pack transmission method and equipment |
US9454400B2 (en) * | 2013-08-16 | 2016-09-27 | Red Hat Israel, Ltd. | Memory duplication by origin host in virtual machine live migration |
US9043576B2 (en) * | 2013-08-21 | 2015-05-26 | Simplivity Corporation | System and method for virtual machine conversion |
CN103455363B (en) * | 2013-08-30 | 2017-04-19 | 华为技术有限公司 | Command processing method, device and physical host of virtual machine |
CN104598303B (en) * | 2013-10-31 | 2018-04-10 | 中国电信股份有限公司 | Online moving method and device between virtual machine based on KVM |
WO2015100622A1 (en) * | 2013-12-31 | 2015-07-09 | 华为技术有限公司 | Method and server for virtual machine live migration |
US9851918B2 (en) * | 2014-02-21 | 2017-12-26 | Red Hat Israel, Ltd. | Copy-on-write by origin host in virtual machine live migration |
CN104917784B (en) * | 2014-03-10 | 2018-06-05 | 华为技术有限公司 | A kind of data migration method, device and computer system |
US9336039B2 (en) * | 2014-06-26 | 2016-05-10 | Vmware, Inc. | Determining status of migrating virtual machines |
WO2016018383A1 (en) * | 2014-07-31 | 2016-02-04 | Hewlett-Packard Development Company | Live migration of data |
US10372335B2 (en) * | 2014-09-16 | 2019-08-06 | Kove Ip, Llc | External memory for virtualization |
US9626108B2 (en) * | 2014-09-16 | 2017-04-18 | Kove Ip, Llc | Dynamically provisionable and allocatable external memory |
CN104750542B (en) * | 2015-04-22 | 2018-01-16 | 成都睿峰科技有限公司 | A kind of data migration method based on cloud platform |
US10114958B2 (en) * | 2015-06-16 | 2018-10-30 | Microsoft Technology Licensing, Llc | Protected regions |
EP3311272B1 (en) * | 2015-06-16 | 2023-04-12 | Telefonaktiebolaget LM Ericsson (PUBL) | A method of live migration |
US10083062B2 (en) * | 2015-07-31 | 2018-09-25 | Cisco Technology, Inc. | Data suppression for faster migration |
US20170060929A1 (en) * | 2015-08-31 | 2017-03-02 | Linkedln Corporation | Controlling servicing of requests in a data migration system |
US9880870B1 (en) * | 2015-09-24 | 2018-01-30 | Amazon Technologies, Inc. | Live migration of virtual machines using packet duplication |
US9936019B2 (en) * | 2016-03-16 | 2018-04-03 | Google Llc | Efficient live-migration of remotely accessed data |
-
2016
- 2016-03-16 US US15/071,852 patent/US9936019B2/en active Active
- 2016-12-02 SG SG10202100763RA patent/SG10202100763RA/en unknown
- 2016-12-02 SG SG11201807848PA patent/SG11201807848PA/en unknown
- 2016-12-02 WO PCT/US2016/064738 patent/WO2017160359A1/en active Application Filing
- 2016-12-02 CN CN201680083580.2A patent/CN108780404A/en active Pending
- 2016-12-02 EP EP16816813.6A patent/EP3414661B1/en active Active
- 2016-12-02 CN CN202111145963.7A patent/CN113821348B/en active Active
- 2016-12-02 KR KR1020187026783A patent/KR101993915B1/en active Application Filing
- 2016-12-02 KR KR1020197017825A patent/KR102055325B1/en active IP Right Grant
- 2016-12-02 JP JP2018548837A patent/JP6728381B2/en active Active
- 2016-12-02 AU AU2016398043A patent/AU2016398043B2/en active Active
-
2018
- 2018-02-22 US US15/902,844 patent/US10187466B2/en active Active
-
2019
- 2019-01-17 US US16/250,822 patent/US10645160B2/en active Active
- 2019-10-31 AU AU2019257477A patent/AU2019257477A1/en not_active Abandoned
-
2020
- 2020-01-03 US US16/734,037 patent/US11005934B2/en active Active
- 2020-07-01 JP JP2020114071A patent/JP7174739B2/en active Active
- 2020-10-30 AU AU2020260536A patent/AU2020260536B2/en active Active
-
2021
- 2021-04-07 US US17/224,239 patent/US11824926B2/en active Active
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8005788B2 (en) * | 2008-01-28 | 2011-08-23 | International Business Machines Corporation | System and method for legacy system component incremental migration |
US9003159B2 (en) * | 2009-10-05 | 2015-04-07 | Marvell World Trade Ltd. | Data caching in non-volatile memory |
US8769241B2 (en) * | 2009-12-04 | 2014-07-01 | Marvell World Trade Ltd. | Virtualization of non-volatile memory and hard disk drive as a single logical drive |
US9164895B2 (en) * | 2009-12-04 | 2015-10-20 | Marvell World Trade Ltd. | Virtualization of solid state drive and mass storage drive devices with hot and cold application monitoring |
US9465561B2 (en) * | 2013-04-18 | 2016-10-11 | Hitachi, Ltd. | Storage system and storage control method |
US9229878B2 (en) * | 2013-06-10 | 2016-01-05 | Red Hat Israel, Ltd. | Memory page offloading in multi-node computer systems |
US20150261576A1 (en) * | 2014-03-17 | 2015-09-17 | Vmware, Inc. | Optimizing memory sharing in a virtualized computer system with address space layout randomization enabled in guest operating systems |
US9483298B2 (en) * | 2014-04-23 | 2016-11-01 | Vmware, Inc. | Converting virtual machine I/O requests |
Cited By (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10592484B1 (en) * | 2017-01-06 | 2020-03-17 | Sprint Communications Company L.P. | Data migration between different lightweight directory access protocol (LDAP) based wireless communication subscriber data stores |
US20190095232A1 (en) * | 2017-09-22 | 2019-03-28 | Fujitsu Limited | Non-transitory computer-readable recording medium, adjustment device, and adjustment method |
US11010186B2 (en) * | 2017-09-22 | 2021-05-18 | Fujitsu Limited | Non-transitory computer-readable recording medium, adjustment device, and adjustment method |
CN108388599A (en) * | 2018-02-01 | 2018-08-10 | 平安科技(深圳)有限公司 | Electronic device, Data Migration and call method and storage medium |
US11074099B2 (en) * | 2018-02-06 | 2021-07-27 | Nutanix, Inc. | System and method for storage during virtual machine migration |
US20190265902A1 (en) * | 2018-02-28 | 2019-08-29 | International Business Machines Corporation | Live migration of applications using capi flash |
CN110347483A (en) * | 2018-04-08 | 2019-10-18 | 中兴通讯股份有限公司 | Physical machine is to virtual machine migration method, device and storage medium |
US11886902B2 (en) | 2018-04-08 | 2024-01-30 | Xi'an Zhongxing New Software Co., Ltd. | Physical-to-virtual migration method and apparatus, and storage medium |
US11182090B2 (en) | 2018-11-19 | 2021-11-23 | Micron Technology, Inc. | Systems, devices, and methods for data migration |
US11709613B2 (en) | 2018-11-19 | 2023-07-25 | Micron Technology, Inc. | Data migration for memory operation |
US10782911B2 (en) | 2018-11-19 | 2020-09-22 | Micron Technology, Inc. | Data migration dynamic random access memory |
US11256437B2 (en) | 2018-11-19 | 2022-02-22 | Micron Technology, Inc. | Data migration for memory operation |
US20200159434A1 (en) * | 2018-11-19 | 2020-05-21 | Micron Technology, Inc. | Systems, devices, techniques, and methods for data migration |
US11853578B2 (en) | 2018-11-19 | 2023-12-26 | Micron Technology, Inc. | Systems, devices, and methods for data migration |
US11442648B2 (en) | 2018-11-19 | 2022-09-13 | Micron Technology, Inc. | Data migration dynamic random access memory |
US11163473B2 (en) * | 2018-11-19 | 2021-11-02 | Micron Technology, Inc. | Systems, devices, techniques, and methods for data migration |
US11782626B2 (en) | 2018-11-19 | 2023-10-10 | Micron Technology, Inc. | Systems, devices, techniques, and methods for data migration |
US11632319B2 (en) * | 2019-02-01 | 2023-04-18 | Nippon Telegraph And Telephone Corporation | Processing device and moving method |
US11409619B2 (en) | 2020-04-29 | 2022-08-09 | The Research Foundation For The State University Of New York | Recovering a virtual machine after failure of post-copy live migration |
US11983079B2 (en) | 2020-04-29 | 2024-05-14 | The Research Foundation For The State University Of New York | Recovering a virtual machine after failure of post-copy live migration |
US12068975B2 (en) * | 2020-09-22 | 2024-08-20 | Xi'an Zhongxing New Software Co., Ltd. | Resource scheduling method and system, electronic device, computer readable storage medium |
US11656982B2 (en) * | 2021-01-15 | 2023-05-23 | Nutanix, Inc. | Just-in-time virtual per-VM swap space |
US20220229774A1 (en) * | 2021-01-15 | 2022-07-21 | Nutanix, Inc. | Just-in-time virtual per-vm swap space |
Also Published As
Publication number | Publication date |
---|---|
US10187466B2 (en) | 2019-01-22 |
AU2019257477A1 (en) | 2019-11-21 |
CN113821348B (en) | 2024-04-19 |
KR101993915B1 (en) | 2019-06-27 |
US11824926B2 (en) | 2023-11-21 |
WO2017160359A1 (en) | 2017-09-21 |
EP3414661A1 (en) | 2018-12-19 |
KR102055325B1 (en) | 2019-12-12 |
US20180183869A1 (en) | 2018-06-28 |
JP2019512804A (en) | 2019-05-16 |
AU2016398043A1 (en) | 2018-10-04 |
KR20180117641A (en) | 2018-10-29 |
AU2020260536A1 (en) | 2020-11-26 |
SG10202100763RA (en) | 2021-02-25 |
JP7174739B2 (en) | 2022-11-17 |
US9936019B2 (en) | 2018-04-03 |
CN108780404A (en) | 2018-11-09 |
JP6728381B2 (en) | 2020-07-22 |
SG11201807848PA (en) | 2018-10-30 |
US11005934B2 (en) | 2021-05-11 |
US20210258378A1 (en) | 2021-08-19 |
EP3414661B1 (en) | 2023-07-26 |
AU2016398043B2 (en) | 2019-09-19 |
US20190158588A1 (en) | 2019-05-23 |
US10645160B2 (en) | 2020-05-05 |
AU2020260536B2 (en) | 2021-02-11 |
CN113821348A (en) | 2021-12-21 |
US20200145488A1 (en) | 2020-05-07 |
KR20190073619A (en) | 2019-06-26 |
JP2020173840A (en) | 2020-10-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11824926B2 (en) | Efficient live-migration of remotely accessed data | |
US9720952B2 (en) | Virtual block devices | |
EP3762826B1 (en) | Live migration of virtual machines in distributed computing systems | |
US9880779B1 (en) | Processing copy offload requests in a storage system | |
US10817333B2 (en) | Managing memory in devices that host virtual machines and have shared memory | |
US10389852B2 (en) | Method and system for providing a roaming remote desktop | |
US9940293B1 (en) | Method for efficient storage and backup of data via SCSI transport | |
US20160350010A1 (en) | Providing block size compatibility with a storage filter | |
US10733153B2 (en) | Snapshot management in distributed file systems | |
US9760577B2 (en) | Write-behind caching in distributed file systems | |
US11467735B2 (en) | I/O operations in log structured arrays | |
US10530870B2 (en) | Direct volume migration in a storage area network | |
US10838783B1 (en) | Data management system and method | |
US20230176884A1 (en) | Techniques for switching device implementations for virtual devices |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: GOOGLE INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SANDERSON, TYLER;REEL/FRAME:038025/0181 Effective date: 20160314 |
|
AS | Assignment |
Owner name: GOOGLE LLC, CALIFORNIA Free format text: CHANGE OF NAME;ASSIGNOR:GOOGLE INC.;REEL/FRAME:044129/0001 Effective date: 20170929 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
CC | Certificate of correction | ||
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |