WO2010116402A1 - Information processor - Google Patents

Abstract

Provided is an information processor equipped with a virtual information processor implementation means that implements one or more virtual information processors by managing hardware resources; a data transmission means that transmits data directly from an input-output device that controls data input-output with an external device to the first memory area assigned to the virtual information processor, by converting an address in a first memory area assigned virtually to a virtual information processor to an address in a second memory area that is the real memory of the first memory area; a transmitted data detection means that detects data directly transmitted from the input-output device to the first memory area assigned to the virtual information processor; an update information registration means that generates update information pertaining to the first memory area which is modified by the data detected with the transmitted data detecting means, and stores the update information in an update information storage means; and an update information output means that outputs the update information stored with the update information storage means.

Description

Information processing device

The present invention relates to an information processing apparatus that realizes one or more virtual machines.

2. Description of the Related Art Conventionally, a server virtualization technique for operating a virtual server called a plurality of VMs (Virtual Machines) on a server as an information processing apparatus is known.
FIG. 1 is a diagram showing an overview of server virtualization technology.

The server 101 operates software called a VMM (Virtual Machine Monitor) 102. The VMM 102 manages hardware resources related to the memory and IO (Input / Output) of the server 101. Further, the VMM 102 provides each VM # 0 to # 2 with emulation (imitation) of hardware resources required by each VM.

When operating a plurality of VMs, exclusive control of the memory is performed so that the plurality of VMs do not use the same memory area of the physical memory provided in the server 101 redundantly. This exclusive control is performed by memory management of the VMM 102.

FIG. 2 is a diagram showing an outline of memory management by the VMM 102.
The memory recognized by the VM is called guest physical memory. This guest physical memory is recognized as a continuous memory area from the VM. On the other hand, when viewed from the VM 102, the guest physical memory is grasped as a separate memory space existing for each VM.

FIG. 2 shows the guest physical memory when two VMs, VM # 0 and VM # 1, are operating on the VMM102. In this case, there are two guest physical memories, the guest physical memory # 0 of VM # 0 and the guest physical memory # 1 of VM # 1.

For example, one address X exists in each of the guest physical memories # 0 and # 1. If this address X is assigned as it is to the real memory provided in the server 101, that is, the host physical memory, a conflict occurs.

Therefore, the VMM 102 assigns the address X of the guest physical memory # 0 to the address Z of the host physical memory, and assigns the address X of the guest physical memory # 1 to the address Y of the host physical memory. In this way, the VMM 102 avoids memory contention by allocating host physical memory to different host physical memories.

Similar memory management is required when the VM activates DMA (Direct Memory Access).
For example, consider a case where VM # 0 writes data to address X of guest physical memory # 0 by DMA transfer. Note that writing data to a predetermined memory area by DMA transfer is called “DMA write”. A memory address indicating a data write destination by DMA transfer is referred to as a “DMA address”.

In this case, when the VM # 0 sets the address X as the DMA address in the IO adapter, the DMA write is performed on the address X of the host physical memory. Since the address X of the host physical memory is a memory area unrelated to the address X of the guest physical memory # 0, memory destruction is caused. If memory corruption is triggered, the system panics and then shuts down.

Therefore, execution of DMA write by VM # 0 is performed according to the following procedure.
(1) The VM # 0 requests the IO adapter to perform a DMA write to the address X of the guest physical memory # 0.
(2) The VMM 102 traps the request for the DMA write of VM # 0 and converts the address X of the guest physical memory # 0 into the address W of the buffer area of the host physical memory. Then, the VMM 102 sets the converted address W in the DMA address setting register of the IO adapter.
(3) VM # 0 instructs the IO adapter to start DMA.
(4) The IO adapter executes DMA write for the address W.
(5) When the DMA write is completed, the IO adapter notifies the completion of the DMA write by a DMA completion interrupt.
(6) The VMM 102 copies the data stored at the address W of the host physical memory to the address X of the guest physical memory # 0.
(7) The VMM 102 notifies the VM # 0 of the completion of the DMA write by a DMA completion interrupt.
(8) The VM # 0 extracts data from the address X of the guest physical memory # 0.

One of the functions related to server virtualization is a technology called live migration. This live migration is a technique for migrating a VM running on a certain server to another server without stopping the operation.

FIG. 3 is a diagram showing an outline of live migration.
As shown in FIG. 3, live migration includes two servers # 0 and # 1, a storage 320 shared by servers # 0 and # 1, and a network 330 to which servers # 0 and # 1 are connected. It is possible to perform in the environment provided. Live migration is performed according to the following procedure.
(1) The VMM 311 operating on the migration destination server # 1 prepares the VM 312.
(2) The migration source VM 302 transfers the contents of the memory used by the migration source VM 302 to the migration destination VM 310 while the migration source VM 302 continues the business. This process is called “pre-copy”.
(3) The task of the migration source VM 302 is temporarily stopped, and the migration source VM 301 transfers the memory contents used by the migration source VM 302 to the migration destination VM 310. This process is referred to as “stop and copy”.
(4) In response to an instruction from the migration destination VMM 311, the migration destination VM 312 resumes the business.

As mentioned above, VMM performs all memory copies such as pre-copy and stop-and-copy. The VMM can execute the memory copy because the VMM manages the host physical memory.

The VMM mediates not only the write processing from the CPU (Central Processing Unit) to the host physical memory but also the DMA write, and detects the memory data change of the migration source VM.

There is a technology called IOMMU (Input / Output Memory Management Unit) as an IO virtualization technology. This IOMMU is a technique for increasing the speed by reducing the memory access overhead caused by the VMM as described above.

The IOMMU is a memory management unit that is connected to the IO adapter and the host physical memory and performs conversion between the guest physical memory address and the host physical memory address.
When the IOMMU is mounted, as shown in FIG. 4, it is possible to execute DMA directly from the IO device to the guest physical memory of the VM. When this technology is used, it is not necessary to intervene the VMM when accessing the guest physical memory by the DMA. Further, it is not necessary for the VMM to copy data between the host physical memory and the guest physical memory.

In connection with the above technique, a bridge for a multiprocessor system having a dirty RAM mechanism in which a predetermined value is set when a memory area is written by DMA access is known.

Also known is a virtual machine control method that eliminates the complexity of resetting the configuration when moving a virtual machine and improves the availability of the system by moving, copying, or saving the virtual machine and the virtual machine control program. Yes.

Also, a mechanism for temporarily stopping a DMA operation selected in a physical I / O adapter in order to enable data migration between physical pages that are accessed by the physical I / O adapter is known.

A virtualization system for a host computer having at least one host processor and system resources including memory divided into system memory having the highest privilege and user memory having the lower privilege is known.
JP 2002-518734 A JP 2006-072591 A JP 2007-287140 A JP 2007-510198 A

As described above, when IOMMU is used, no VMM is interposed during DMA access. In this case, since the VMM cannot grasp the update of the guest physical memory by the DMA write, the VMM cannot correctly execute the memory copy. For this reason, there is a problem that live migration cannot be realized if IOMMU is used.

The present information processing apparatus has been made in view of the above-described problems, and the problem to be solved is to provide an information processing apparatus capable of performing live migration even when IOMMU is used. It is.

In order to solve the above problems, the information processing apparatus includes the following means.
The virtual machine realization means realizes one or more virtual machines by managing hardware resources.

The data transfer means mutually converts the address of the first memory area allocated to the virtual machine and the address of the second memory area that is the real memory of the first memory area. The data transfer means transfers data directly from the input / output device that controls input / output of data to / from an external device to the first memory area assigned to the virtual machine.

The transfer data detection means detects data directly transferred from the input / output device to the first memory area allocated to the virtual machine.
The update information registration unit generates update information regarding the first memory area that is changed by the data detected by the transfer data detection unit, and stores the update information in the update information storage unit.

The update information output means outputs the update information stored in the update information storage means.
According to the information processing apparatus, when the virtual machine directly transfers data from the external device to the first memory area using the data transfer means, the change information registration means changes the first memory changed by the data transfer. Information about the area is stored in the update information storage means. The update information output means outputs update information stored in the update information output means.

According to the information processing apparatus described above, live migration can be executed even when IOMMU is used.

It is a figure which shows the outline | summary of a server virtualization technique. It is a figure which shows the outline | summary of the memory management by VMM. It is a figure which shows the outline | summary of live migration. It is a figure explaining IOMMU. It is a figure which shows the structural example of information processing apparatus. It is a figure explaining the operation | movement of the DMA process in a north bridge. It is a figure explaining the operation | movement of the DMA process in a PCIe switch. It is a figure which shows the structural example of FIFO # 0. It is a figure which shows the structural example of the header of a DMA packet. It is a flowchart which shows the registration process of dirty page information by a dirty page management unit. It is a flowchart which shows the output process of dirty page information by a dirty page management unit. It is a figure explaining the outline | summary of the live migration which concerns on a present Example. It is a flowchart which shows the outline | summary of the live migration which concerns on a present Example. It is a flowchart which shows the specific process of a pre copy. It is a flowchart which shows the specific process of stop and copy.

Hereinafter, an example of this embodiment will be described with reference to FIGS.
(Information processing apparatus according to this embodiment)
FIG. 5 is a diagram illustrating a configuration example of the information processing apparatus 500.

The information processing device 500 includes CPUs 510 and 511 as arithmetic processing devices, a memory 520 as a main storage device, and a north bridge 530 as a memory control device. The information processing apparatus 500 includes IO adapters 540, 541, and 542 as input / output control apparatuses, and a PCIe switch (PCI Express Switch) 550.

CPUs 510 and 511 are arithmetic processing units that execute programs developed in the memory 520. The CPUs 510 and 511 implement server virtualization by executing predetermined programs. In addition, the CPUs 510 and 511 implement the live migration according to the present embodiment by reading and executing a predetermined program from the storage # 0 or # 1.

Note that conventional technologies can be used for technologies such as management of hardware resources and memory necessary for server virtualization.
The memory 520 is a volatile memory that stores programs and data executed by the CPUs 510a and 510b. For example, a RAM (Random Access Memory) or the like is used as the memory 520. A nonvolatile memory may be used as necessary.

The north bridge 530 is connected to the CPUs 510a and 510b, the memory 520, and the PCIe switch 550. The north bridge 530 controls the data transfer path so that the CPUs 510a and 510b and the memory 520 and the PCIe switch 550 can communicate with each other.

The north bridge 530 includes an IOMMU 531 and an IO table storage unit 532.
The IOMMU 531 is a memory management unit that performs memory management such as conversion between the address of the guest physical memory and the address of the host physical memory. Hereinafter, the address of the guest physical memory is referred to as “guest physical address”. The address of the host physical memory is called “host physical address”.

The IO table storage unit 532 is a storage device that stores an IO table used when the IOMMU 531 converts a guest physical address and a host physical address. For example, a cache memory or the like is used as the IO table storage unit 532. The IO table will be described with reference to FIG.

The IO adapters 540, 541, and 542 are interfaces with IO devices connected to the information processing device 500. A DMA circuit that performs DMA is provided in each of the IO adapters 540, 541, and 542.

The IO device is, for example, an input / output device such as storages # 0 and # 1 that store data shown in FIG. 5 or a network device connected to a network.
The PCIe switch 550 is connected to the north bridge 530 and the IO adapters 540, 541, and 542.

The PCIe switch 550 performs switching control of a data transfer path connecting between the north bridge 530 and the IO adapters 540, 541, and 542. The information processing apparatus 500 uses “PCI Express 2.0” which is a serial transfer interface standard for the data transfer path.

The PCIe switch 550 includes a dirty page management unit 551 and a dirty page storage unit 552.
The dirty page management unit 551 detects a packet as data transferred from the IO adapter 540, 541, or 542 to the memory 520 by DMA write. Then, the dirty page management unit 551 stores information regarding data added, changed, or updated by the DMA write in the dirty page storage unit 552 in units of pages.

Hereinafter, page unit data added, changed, or updated by the DMA write is referred to as “dirty page”. Information about dirty pages is referred to as “dirty page information”.

The dirty page storage unit 552 is a storage device that stores dirty page information. For example, a static RAM or the like is used as the dirty page storage unit 552.
The information processing apparatus 500 is connected to the information processing apparatus 501 via a network in order to perform live migration. Further, the information processing apparatus 500 and the information processing apparatus 501 share the storages # 0 and # 1.

The information processing apparatus 501 includes CPUs 560 and 561, a memory 570, a north bridge 580, IO adapters 590, 591 and 592, and a PCIe switch 600.

CPUs 560, 561, memory 570, north bridge 580, IO adapters 590, 591, 592 and PCIe switch 600 are CPU 510, 511, memory 520, north bridge 530, IO adapters 540, 541, 542 and PCIe switch 550, It is the same.

However, the north bridge 580 may not include the IOMMU 531 and the IO table storage unit 532. Similarly, the PCIe switch 600 may not include the dirty page management unit 551 and the dirty page storage unit 552.

The information processing apparatus 501 connected to the information processing apparatus 500 only needs to be able to execute the general live migration shown in FIG.
The configuration illustrated in FIG. 5 is an example of the information processing apparatus 500 according to the present embodiment. Therefore, the configuration of the information processing apparatus 500 is not limited to the configuration illustrated in FIG. For example, the number of CPUs, the number of IO adapters, the arrangement of units constituting the information processing apparatus 500, and the like are not limited to those illustrated in FIG.

Further, the information processing apparatus 500 may include a medium driving device that drives a portable storage medium such as a CD or a DVD. When the live migration according to the present embodiment is stored in a portable storage medium such as a CD or DVD, the CPUs 510 and 511 read and execute a predetermined program from the portable storage medium via the medium driving device. The live migration according to the present embodiment is realized.

FIG. 6 is a diagram for explaining the DMA processing operation in the north bridge 530.
The DMA packets 610 and 611 are packets as data transferred from the IO adapter 540, 541, or 542 to a predetermined transfer destination by DMA processing.

The IO table 620 indicates an IO table stored in the IO table storage unit 532. The IO table 620 is an address conversion table between guest physical addresses and host physical addresses. The IO table 620 defines a host physical address corresponding to an address composed of a guest physical address and a source ID.

The IO table 620 caches (temporarily holds) some entries that are frequently used among entries in a page table 630 described later.
Here, the source ID is an ID (Identification Data) that identifies a packet transfer source unit.

Hereinafter, in the drawings, a guest physical address is abbreviated as “GPA (Guest Physical Address)”, a host physical address is abbreviated as “HPA (Host Physical Address)”, and a source ID is abbreviated as “SID (Source ID)” as necessary.

The page table 630 is an address conversion table for guest physical addresses and host physical addresses. The page table 630 defines a host physical address corresponding to the guest physical address.

When detecting the DMA packet 610, the IOMMU 531 acquires the guest physical address and the source ID from the header of the detected DMA packet 610.
Then, the IOMMU 531 refers to the IO table 620. Then, the IOMMU 531 determines whether or not an address composed of the guest physical address and the source ID acquired from the DMA packet 610 is registered in the IO table 620.

When an address composed of a guest physical address and a source ID acquired from the DMA packet 610 is registered, the IOMMU 531 acquires a host physical address corresponding to an address composed of the guest physical address and the source ID.

Then, the IOMMU 531 changes the guest physical address indicating the transfer destination set in the header of the DMA packet 610 to the host physical address acquired from the IO table 620.

On the other hand, when the address composed of the guest physical address and the source ID acquired from the DMA packet 610 is not registered in the IO table 620, the IOMMU 531 refers to the page table 630 stored in the memory 520.

Then, the IOMMU 531 acquires the host physical address registered in the same address as the guest physical address acquired from the DMA packet 610, which is the address of the page table 630.

Then, the IOMMU 531 changes the guest physical address indicating the transfer destination set in the header of the DMA packet 610 to the host physical address acquired from the page table 630.

The DMA packet 611 in which the transfer destination address is converted from the guest physical address to the host physical address as described above is output to the memory 520.
FIG. 7 is a diagram for explaining the operation of DMA processing in the PCIe switch 550.

The dirty page management unit 551 includes a control I / F (Interface) unit 710, a packet detection unit 720, a write pointer 730, and a read pointer 740.

In order to facilitate understanding of the operation of the dirty page management unit 551, in FIG. 7, a dirty page storage unit 552 is described in the dirty page management unit 551.

The dirty page storage unit 552 includes memories # 0 and # 1 that realize a FIFO (First In First Out). Hereinafter, the memories # 0 and # 1 realizing the FIFO are referred to as “FIFO # 0” and “FIFO # 1”, respectively.

The control I / F unit 710 is an interface of the dirty page management unit 551. The VMM 102 acquires dirty page information stored in the FIFO # 0 or # 1 via the control I / F unit 710.

The control I / F unit 710 includes a status register 711, a data register 712, and a control register 713.
The status register 711 is a register that can be read / written by software such as the VMM 102. The status register 711 includes the following status display bits.

Overflow information bit # 0: indicates that FIFO # 0 has overflowed.
Overflow information bit # 1: indicates that FIFO # 1 has overflowed.

Valid data count # 0: In FIFO # 0, a value obtained by dividing the difference between the address indicated by the write pointer 730 and the address indicated by the read pointer 740 by the size of the dirty page information. It shows how much dirty page information that has not yet been read remains in FIFO # 0.

Valid data count # 1: In FIFO # 1, a value obtained by dividing the difference between the address indicated by the write pointer and the address indicated by the read pointer by the size of the dirty page information. This indicates how much dirty page information that has not yet been read remains in FIFO # 1.

The data register 712 is a register that can be read from software such as the VMM 102. This register is used to read data stored in the FIFO # 0 or # 1 from the VMM 102.

In the data register 712, the dirty page information read from the FIFO # 0 or # 1 designated by the read select bit of the control register 713 is set. When the dirty page information stored in the data register 712 is read, the dirty page management unit 551 sets the read pointer 740 to the address where the next dirty page information is stored.

The control register 713 is a register that can be read / written by software such as the VMM 102. The control register 713 includes the following control bits.

Start bit: Starts registration of dirty page information in FIFO # 0 or # 1.
Stop bit: Stops registration of dirty page information in FIFO # 0 or # 1.

Clear bit: The pointer of FIFO # 0 or # 1 is returned to 0, and FIFO # 0 or # 1 is made empty. The empty state is a state where there is no dirty page information registered in the FIFO.

Write select bit: Selects whether to register dirty page information in two FIFOs # 0 or # 1.
Read select bit: Selects whether dirty page information is read from two FIFOs # 0 or # 1.

The packet detection unit 720 detects the DMA write packet 750 input to the PCIe switch 550 while the start bit of the control register 713 is set to “1”. Then, the packet detection unit 720 acquires the guest physical address and the source ID from the header of the detected DMA write packet 750. Further, the packet detection unit 720 writes the data including the acquired guest physical address and source ID as dirty page information in the currently selected FIFO # 0 or # 1. At the same time, the packet detection unit 720 sets the address of the area where dirty page information is stored next in the write pointer 730.

The write pointer 730 indicates the address of dirty page information last written to FIFO # 0 or # 1. In the present embodiment, as shown in FIG. 7, write pointers 730 are provided for FIFO # 0 and for FIFO # 1, respectively.

The read pointer 740 indicates the address of dirty page information last read from FIFO # 0 or # 1. In the present embodiment, as shown in FIG. 7, read pointers 740 are provided for FIFO # 0 and for FIFO # 1, respectively.

When FIFO # 0 is selected, the write pointer and read pointer for FIFO # 0 are stored in write pointer 730 and read pointer 740.

Similarly, when FIFO # 1 is selected, the write pointer and read pointer for FIFO # 1 are stored in write pointer 730 and read pointer 740.

FIG. 8 is a diagram illustrating a configuration example of FIFO # 0. FIG. 8 shows only FIFO # 0, but FIFO # 1 has the same configuration.
The FIFO # 0 includes a source ID that indicates an IO adapter that is a transmission source of a DMA packet, and a guest physical address [63:12] that indicates a transfer destination of the DMA packet. In the case of PCI Express, the source ID is composed of a DMA packet transmission source bus number and a function number of the DMA packet transmission source.

In this embodiment, the size of one page is, for example, 4 KB. In this case, the lower 12 bits of the 64-bit address data represent an address in the same page. Therefore, in this embodiment, only the data of bits 63 to 12 in the guest physical address is registered in the FIFO # 0.

When using 32-bit data for the guest physical address, register the 64-bit address data in which “0” is complemented to bits 63 to 32 of the guest physical address in FIFO # 0.

The above-mentioned source ID and guest physical address can be obtained from the header of the DMA packet.
FIG. 9 shows a configuration example of the header of the DMA packet. In this embodiment, since the PCIe switch 550 is used, the header of the DMA packet conforms to the standard “PCI Express 2.0”.

The header 1101 of the DMA packet shown in (1) of FIG. 9 shows a configuration example of the header of the DMA packet when address data having a 32-bit width is used for the guest physical address. A DMA packet header 1102 shown in (2) of FIG. 9 shows a configuration example of a DMA packet header when address data having a 64-bit width is used as a guest physical address.

“R” is a reserved area. “0” is always set.
“Fmt (Format)” is 2-bit data indicating the presence / absence of data stored in the payload and the header length.

“Type” is 5-bit wide data indicating the type of packet.
“TC (Transaction Class)” is 3-bit width data indicating the priority of a packet.

“TD (TLP (Transaction Layer Packet) Digest)” is 1-bit data indicating the presence or absence of an ECRC (Extended Cyclic Redundancy Check) that is an error check code.

“EP (Error Poisoned)” is 1-bit data indicating the possibility that the data stored in the payload is corrupted.
“Attr (Attributes)” is 2-bit data indicating supplementary information related to the packet order and protocol.

“AT (Address Translation)” is 2-bit data indicating supplementary information related to address translation.
“Length” is 10-bit width data indicating the data length of data stored in the payload.

The “bus number” is 8-bit data indicating the bus number of the transmission source of the DMA packet.
“Function number” is 8-bit data indicating the function number of the transmission source of the DMA packet.

“Tag” is 8-bit width data indicating the management number of the DMA packet.
“Last DW BE” is 4-bit width data indicating validity / invalidity of the Last DW Byte.

“1st DW BE” is 4-bit data indicating the validity / invalidity of the 1st DW Byte.
“Address” is 30-bit or 62-bit address data.

FIG. 10 is a flowchart showing the registration process of dirty page information by the dirty page management unit 551. 10 indicates the FIFO set in the write select bit of the control register 713 in the FIFO # 0 or # 1.

When software such as the VMM 102 sets the start bit of the control register 713 included in the dirty page management unit 551 to “1”, the dirty page management unit 551 starts the registration process of dirty page information (step S1000). .

In step S1001, the packet detection unit 720 refers to the header of the DMA packet input to the PCIe switch 550 from an IO adapter such as the IO adapter 540, 541, or 542.

Then, the packet detection unit 720 acquires “Fmt” and “Type” from the header of the DMA packet.
When the packet detection unit 720 detects a DMA write packet that is transferred from the IO adapter to the memory 520 and has Fmt = 10 (binary number) or Fmt = 11 (binary number) and Type = 00000 (binary number), step S1002 (Step S1001 YES).

Note that, when Fmt = 10 (binary number), it indicates a DMA packet that uses 32-bit address data as a guest physical address. Further, when Fmt = 11 (binary number), it indicates that the packet is a DMA packet using 64-bit address data as a guest physical address.

Type = 00000 (binary number) indicates a DMA write packet.
If the packet detection unit 720 determines that the DMA packet input to the PCIe switch 550 is not a DMA write packet, the packet detection unit 720 executes step S1001 again (NO in step S1001).

In step S1002, the packet detection unit 720 determines whether the difference between the address pointed to by the write pointer 730 and the address pointed to by the read pointer 740 is equal to a value obtained by dividing the size of the FIFO by the size of the dirty page information. Is determined to be Full.

When the difference between the address pointed to by the write pointer 730 and the address pointed to by the read pointer 740 is 0, the packet detection unit 720 determines that the FIFO is full (YES in step S1002). In this case, the packet detection unit 720 moves the process to step S1003.

In step S1003, when the packet detection unit 720 sets “1” in the overflow information bit of the status register 711, the process proceeds to step S1008. Then, the dirty page management unit 551 ends the dirty page information registration process.

In step S1002, when the difference between the address pointed to by the write pointer 730 and the address pointed to by the read pointer 740 is not equal to the value obtained by dividing the size of the dirty page information by the size of the FIFO, the packet detection unit 720 indicates that the FIFO is not Full. (Step S1002 NO). In this case, the packet detection unit 720 moves the process to step S1004.

In step S1004, the packet detection unit 720 acquires a source ID and a guest physical address from the header of the DMA write packet.
In step S1005, the packet detection unit 720 registers data including the source ID and guest physical address acquired in step S1004 as dirty page information in the FIFO address indicated by the write pointer 730.

In step S1006, the packet detection unit 720 increments the address stored in the write pointer 730 by the area size for storing the dirty page information.

In step S1007, the packet detection unit 720 refers to the control register 713 provided in the control I / F unit 710. If “1” is set in the stop bit of the control register 713 (YES in step S1007), the packet detection unit 720 moves the process to step S1008. Then, the dirty page management unit 551 ends the dirty page information registration process.

In step S1007, when the stop bit of the control register 713 is not “1”, the packet detection unit 720 moves the process to step S1001 (NO in step S1007).

FIG. 11 is a flowchart showing a process for outputting dirty page information by the dirty page management unit 551. Note that the FIFO described in FIG. 11 indicates the FIFO set in the read select bit of the control register 713 in the FIFO # 0 or # 1.

In step S1101, when software such as the VMM 102 reads dirty page information from the data register 712 included in the dirty page management unit 551, the control I / F unit 710 shifts the processing to step S1102.

In step S1102, the control I / F unit 710 increments the address stored in the read pointer 740 by the area size for storing dirty page information.

In step S1103, the control I / F unit 710 acquires dirty page information from the FIFO address indicated by the read pointer 740 and latches it in the data register 712.

When the above process ends, the control I / F unit 710 shifts the process to step S1104. Then, the control I / F unit 710 ends the dirty page information output process.
(Live migration according to this example)
FIG. 12 is a diagram for explaining the outline of live migration according to the present embodiment.

The information processing system 1200 illustrated in FIG. 12 includes a server # 0, a server # 1 that is communicably connected to the server # 0 via a network, and storages # 0 and # that are shared by the server # 0 and the server # 1. 1.

Server # 0 is the information processing apparatus 500 shown in FIG. The server # 0 is the information processing apparatus 501 shown in FIG.
On the server # 0, the VMM # 0 operates. VMM # 0 implements VM # 0 and VM # 1. On the other hand, VMM # 1 operates on the server # 1. Also, VM # 0 occupies storage # 0, and VM # 1 occupies storage # 1.

A case will be described below in which live migration is performed in which VM # 0 realized by VMM # 0 operating on server # 0 is moved to VMM # 1 operating on server # 1 in the above state.

FIG. 13 is a flowchart illustrating an outline of live migration according to the present embodiment.
In step S1301, the VMM # 0 requests the VMM # 1 that is the migration destination of the VM # 0 to secure a memory area to be allocated to the new VM. Hereinafter, the memory area allocated to the VM is referred to as “VM area”.

On the other hand, when a VM area securing request is received from VMM # 1, VMM # 1 secures a VM area in a predetermined area of the memory provided in server # 1. Then, the server # 1 notifies the server # 0 that the reservation of the VM area has been completed.

Upon receiving a notification from the server # 1 that the reservation of the VM area has been completed, the server # 0 moves the process to step S1302.
In step S1302, the VMM # 0 obtains data stored in the area of the host physical memory allocated to the migration target VM # 0 at a certain point in time. This data stored in the area of the host physical memory allocated to the VM at a certain point (check point) is called “snapshot”.

In step S1303, the VMM # 0 sets the start bit of the control register 713 of the dirty page management unit 551 to “1”, and starts recording dirty page information.

In step S1304, the VMM # 0 stores the snapshot acquired in step S1302 in a predetermined area of the memory provided in the migration destination server # 1 of the VM # 0, and stores the snapshot in the VM area secured in step S1301. make a copy.

In step S1305, the VMM # 0 acquires the remaining number of dirty page information from the dirty page management unit 551 through the valid data count of the status register 711 of the control I / F unit 710. Then, VMM # 0 compares the number of remaining dirty page information with a predetermined threshold value.

If the number of remaining dirty page information is greater than the threshold (YES in step S1305), the VMM # 0 proceeds to step S1306. If the number of remaining dirty page information is less than or equal to the threshold (NO in step S1305), the VMM # 0 moves the process to step S1307.

In step S1306, the VMM # 0 receives all the dirty page information registered in the dirty page storage unit 552 from the dirty page management unit 551 through the data register 712 of the control I / F unit 710 to date. To get.

Then, the VMM # 0 converts the guest physical address of the acquired dirty page information into a host physical address.
Then, the VMM # 0 acquires data stored in the converted host physical address from the memory provided in the server # 0. The VMM # 0 then copies the acquired data to the VM area secured in step S1301, which is the memory area of the migration destination server # 1 of the VM # 0.

When the memory copy is executed, the VMM # 0 moves the process to step S1305.
The processing in steps S1302 to S1306 described above is “pre-copy”. The pre-copy is performed while VM # 0 continues the business. Therefore, VMM # 0 performs read / write processing for storage # 0 via IOMMU 531 even during pre-copy. Therefore, the following stop and copy is required.

In step S1307, the VMM # 0 stops the operation of the VM # 0.
In step S1308, the VMM # 0 executes the same process as in step S1306.

The processing in steps S1307 to S1308 described above is “stop and copy”.
In step S1309, the VMM # 0 notifies the VMM # 1 operating on the migration destination server # 1 of the VM # 0 that the memory copy has been completed.

In step S1310, upon receiving a memory copy completion notification from the VMM # 0, the VMM # 1 starts the VM # 0 operation on the VMM # 1. When VM # 0 resumes the business on VMM # 1, VMM # 1 moves the process to step S1311. Then, the VMMs # 0 and # 1 end the live migration.

FIG. 14 is a flowchart showing specific pre-copy processing.
In step S1401, the VMM # 0 initializes the FIFOs # 0 and # 1 by setting “1” to the clear bit of the control register 713 provided in the control I / F unit 710.

Also, the VMM # 0 sets the FIFO for recording dirty page information to FIFO # 0 by setting the write select bit of the control register 713 to “0”. Similarly, the VMM # 0 sets the FIFO for reading dirty page information to FIFO # 0 by setting the read select bit of the control register 713 to “0”.

In step S1402, the VMM # 0 starts recording dirty page information by setting the start bit of the control register 713 included in the dirty page management unit 551 to “1”.

In step S1403, the VMM # 0 executes memory copy. In step S1403, the memory copy process described in steps S1302 to S1304 shown in FIG. 13 is performed.

In step S1404, the VMM # 0 shifts to a wait state for a certain time. When the predetermined time has elapsed, the VMM # 0 moves the process to step S1405.
In step S1405, the VMM # 0 sets “0” or “1” in the write select bit of the control register 713 provided in the control I / F unit 710 to change the FIFO for recording dirty page information.

For example, if the currently selected FIFO is FIFO # 0, VMM # 0 changes the FIFO for recording dirty page information to FIFO # 1 by setting the write select bit of the control register 713 to “1”. To do.

Similarly, if the currently selected FIFO is FIFO # 1, VMM # 0 sets the write select bit of the control register 713 to “0” to change the FIFO for recording dirty page information to FIFO # 0. change.

In step S1406, the VMM # 0 reads the status register 711 provided in the control I / F unit 710.
In step S1407, the VMM # 0 determines whether or not the overflow information bit of the currently selected FIFO is set to “1”.

If the overflow information bit of the currently selected FIFO is set to “1” (step S1407: YES), the VMM # 0 moves the process to step S1408. In this case, the VMM # 0 interrupts the live migration being executed or starts over from the beginning.

In step S1407, if the overflow information bit of the currently selected FIFO is set to “0” (NO in step S1407), VMM # 0 proceeds to step S1409.

In step S1409, the VMM # 0 reads the data register 712 by the number set in the valid data count of the status register 711, and acquires dirty page information.

In step S1410, the VMM # 0 extracts only the dirty page information whose source ID indicates the storage # 0 from the dirty page information read in step S1409.

The dirty page information extracted in step S1410 or the processing of S1507 described later is referred to as “copyable dirty page information”.
In step S1411, the VMM # 0 converts the guest physical address of the copy target dirty page information into a host physical address according to the page table 630.

In step S1412, the VMM # 0 acquires the data stored in the host physical address of the copy target dirty page information from the memory 520 provided in the server # 0. Then, the VMM # 0 transfers the acquired data to the VMM # 1 that is the migration destination of the VM # 0.

On the other hand, when the VMM # 1 receives data from the VMM # 0, the VMM # 1 stores the received data in the VM area secured by the process of step S1301 shown in FIG.
In step S1413, the VMM # 0 changes the FIFO for reading dirty page information by setting “0” or “1” to the read select bit of the control register 713 provided in the control I / F unit 710.

For example, if the currently selected FIFO is FIFO # 0, VMM # 0 sets the read select bit of the control register 713 to “1” to change the FIFO for reading dirty page information to FIFO # 1. .

Similarly, when the currently selected FIFO is FIFO # 1, VMM # 0 changes the FIFO for reading dirty page information to FIFO # 0 by setting the read select bit of the control register 713 to “0”. To do.

In step S1414, the VMM # 0 determines whether or not the pre-copy completion condition is satisfied. In the process of step S1414, the process of step S1305 shown in FIG. 13 is performed.

If it is determined that the pre-copy completion condition is satisfied (YES in step S1414), the VMM # 0 proceeds to step S1415. In this case, the VMM # 0 ends the pre-copy.

If it is determined that the pre-copy completion condition is not satisfied (NO in step S1414), the VMM # 0 proceeds to step S1404.
FIG. 15 is a flowchart showing specific processing of stop-and-copy.

When the pre-copy shown in FIG. 14 is completed, VMM # 0 starts the stop-and-copy process. When the VMM # 0 stops the operation of the VM # 0, the process proceeds to step S1501 (step S1500).

In step S1501, VMM # 0 shifts to a wait state for a certain time. This is to wait until the DMA write process is completely stopped. When the predetermined time has elapsed, the VMM # 0 moves the process to step S1502.

In step S1502, the VMM # 0 changes the FIFO for recording dirty page information by setting “0” or “1” to the write select bit of the control register 713 provided in the control I / F unit 710.

For example, if the currently selected FIFO is FIFO # 0, VMM # 0 changes the FIFO for recording dirty page information to FIFO # 1 by setting the write select bit of the control register 713 to “1”. To do.

Similarly, if the currently selected FIFO is FIFO # 1, VMM # 0 sets the write select bit of the control register 713 to “0” to change the FIFO for recording dirty page information to FIFO # 0. change.

In step S1503, the VMM # 0 reads the status register 711 provided in the control I / F unit 710. Then, the VMM # 0 determines whether or not the overflow information bit of the currently selected FIFO is set to “1”.

In step S1504, if the overflow information bit of the currently selected FIFO is set to “1” (YES in step S1504), VMM # 0 proceeds to step S1505. In this case, the VMM # 0 interrupts the live migration or starts over from the beginning.

In step S1504, if the overflow information bit of the currently selected FIFO is set to “0” (NO in step S1504), the VMM # 0 moves the process to step S1506.

In step S1506, the VMM # 0 reads the data register 712 by the number set in the valid data count of the status register 711, and acquires dirty page information.

In step S1507, the VMM # 0 extracts from the dirty page information read in step S1506, the copy target dirty page information in which the source ID of the dirty page information indicates the storage # 0.

In step S1508, the VMM # 0 converts the guest physical address of the copy target dirty page information into a host physical address according to the page table 630.

In step S1509, the VMM # 0 determines whether there is data page information in the FIFO from the valid data count bit of the status register 711 provided in the control I / F unit 710.

If the valid data count in the status register 711 is “0” (YES in step S1509), the VMM # 0 determines that the FIFO is empty, and the process proceeds to step S1510.

In step S1510, the VMM # 0 acquires the data stored in the host physical address of the copy target dirty page information from the memory 520 provided in the server # 0. Then, the VMM # 0 transfers the acquired data to the VMM # 1 that is the migration destination of the VM # 0.

On the other hand, when the VMM # 1 receives data from the VMM # 0, the VMM # 1 stores the received data in the VM area secured by the process of step S1301 shown in FIG.
When the data transfer to the VMM # 1 is completed, the VMM # 0 moves the process to step S1511. Then, the VMM # 0 ends the stop and copy (step S1511).

On the other hand, if the valid data count of the status register 711 is not “0” in step S1509 (NO in step S1509), the VMM # 0 determines that the FIFO is not empty, and the process proceeds to step S1512.

In step S1512, the VMM # 0 changes the FIFO for reading dirty page information by setting “0” or “1” to the read select bit of the control register 713 provided in the control I / F unit 710.

For example, if the currently selected FIFO is FIFO # 0, VMM # 0 sets the read select bit of the control register 713 to “1” to change the FIFO for reading dirty page information to FIFO # 1. .

Similarly, when the currently selected FIFO is FIFO # 1, VMM # 0 changes the FIFO for reading dirty page information to FIFO # 0 by setting the read select bit of the control register 713 to “0”. To do.

When the FIFO that reads the dirty page information is changed, the VMM # 0 moves the process to step S1501.
As described above, when the DMA packet transferred from the IO adapters 540, 541, and 542 to the memory 520 passes through the PCIe switch 550, the dirty page management unit 551 detects the DMA packet by the packet detection unit 720. .

If the detected DMA packet is a DMA write packet, the dirty page management unit 551 acquires the guest physical memory and the source ID from the header of the DMA packet and registers them in the FIFO # 0 or # 1 as dirty page information. To do.

The dirty page management unit 551 includes a CPU 510 or 511, that is, a control I / F unit 710 with the VMM realized by the CPU 510 or 511, and is registered in the FIFO # 0 or # 1 in response to a request from the VMM. Output dirty page information.

As a result, the VMM operating in the information processing apparatus 500 can acquire data on the memory 520 added, changed, or updated by the DMA write using the IOMMU 531 from the dirty page information.

As described above, since the data on the memory 520 added, changed, or updated by the DMA write can be acquired, the pre-copy and stop-and-copy shown in FIGS. Is possible. As a result, live migration can be performed even when IOMMU is used.

In addition, the information processing apparatus 500 includes two FIFOs, FIFO # 0 and # 1, as the dirty page storage unit 552. When one of the FIFOs is a write target, it is possible to avoid contention between the read process and the write process for one FIFO by making the other FIFO a read target. As a result, the dirty page information registration process to the FIFO and the dirty page information read process from the FIFO can be performed efficiently.

Claims (12)

1. Virtual machine implementation means for managing one or more virtual machines by managing hardware resources;
   By mutually converting the address of the first memory area allocated to the virtual machine and the address of the second memory area that is the real memory of the first memory area, data input to the external device is performed. Data transfer means for directly transferring data from the input / output device for controlling output to the first memory area allocated to the virtual machine;
   Transfer data detection means for detecting data directly transferred from the input / output device to the first memory area allocated to the virtual machine;
   Update information registration means for generating update information related to the first memory area changed by the data detected by the transfer data detection means, and storing the update information in an update information storage means;
   Update information output means for outputting update information stored in the update information storage means;
   An information processing apparatus comprising:
2. The data detected by the transfer data detecting means is directly transferred from the input / output device to the first memory area allocated to the virtual machine and written to the first memory area in accordance with an instruction from the virtual machine. Data
   The information processing apparatus according to claim 1.
3. The update information is information including identification information for identifying the input / output device that has transferred the data, and an address of the first memory area that is a transfer destination of the data.
   The information processing apparatus according to claim 1.
4. The update information storage means includes a first storage means and a second storage means,
   Update information can be stored or read out independently of each other.
   The information processing apparatus according to claim 1.
5. Implementing one or more virtual machines by managing hardware resources;
   By mutually converting the address of the first memory area allocated to the virtual machine and the address of the second memory area that is the real memory of the first memory area, data input to the external device is performed. Direct data transfer from an input / output device controlling output to the first memory area assigned to the virtual machine;
   Detecting data transferred directly from the input / output device to the first memory area allocated to the virtual machine;
   Generating update information related to the first memory area changed by the detected data, and storing the update information in an update information storage means;
   Outputting update information stored in the update information storage means;
   An information processing apparatus virtualization method comprising:
6. Virtual machine implementation means for managing one or more virtual machines by managing hardware resources;
   By mutually converting the address of the first memory area allocated to the virtual machine and the address of the second memory area that is the real memory of the first memory area, data input to the external device is performed. Data transfer means for directly transferring data from the input / output device for controlling output to the first memory area allocated to the virtual machine;
   Memory area transfer means for acquiring data stored in a first memory area allocated to the virtual machine to be moved and transferring the data to a destination;
   Transfer data detection means for detecting data directly transferred from the input / output device to the first memory area allocated to the virtual machine;
   Update information registration means for generating update information related to the first memory area changed by the data detected by the transfer data detection means, and storing the update information in an update information storage means;
   Update information acquisition means for acquiring update information from the update information storage means;
   Based on the update information acquired by the update information acquisition unit, update data changed by the data detected by the transfer data detection unit in the first memory area allocated to the virtual machine to be moved is transferred to the destination. Update data transfer means for transferring;
   An information processing apparatus comprising:
7. The data detected by the transfer data detecting means is directly transferred from the input / output device to the first memory area allocated to the virtual machine and written to the first memory area in accordance with an instruction from the virtual machine. Data
   The information processing apparatus according to claim 6.
8. The update information is information including identification information for identifying the input / output device that has transferred the data, and an address of the first memory area that is a transfer destination of the data.
   The information processing apparatus according to claim 6.
9. The update information storage means includes a first storage means and a second storage means,
   Update information can be stored or read out independently of each other.
   The information processing apparatus according to claim 6.
10. The update data transfer means includes
    First update data transfer means for transferring the update data to the destination until the number of update information stored in the update information storage means becomes a predetermined value or less;
    Second update data transfer means for transferring the update data to the destination after stopping the operation of the virtual machine to be moved;
    Comprising
    The information processing apparatus according to claim 6.
11. In a migration method for moving a virtual machine realized by a first information processing apparatus to a second information processing apparatus that is communicably connected to the first information processing apparatus,
    Implementing one or more virtual machines by managing hardware resources;
    By mutually converting the address of the first memory area allocated to the virtual machine and the address of the second memory area that is the real memory of the first memory area, data input to the external device is performed. Direct data transfer from an input / output device controlling output to the first memory area assigned to the virtual machine;
    Acquiring data stored in a first memory area allocated to the virtual machine to be moved and transferring the data to the second information processing apparatus;
    Detecting data transferred directly from the input / output device to the first memory area allocated to the virtual machine;
    Generating update information related to the first memory area changed by the detected data, and storing the update information in an update information storage means;
    Obtaining update information from the update information storage means;
    Transferring update data changed by the detected data in the first memory area allocated to the virtual machine to be moved to the second information processing device based on the update information;
    A migration method comprising:
12. In a migration program for moving a virtual machine realized by a first information processing apparatus having an arithmetic processing apparatus to a second information processing apparatus that is communicably connected to the first information processing apparatus,
    Virtual machine realization processing for realizing one or more virtual machines by managing hardware resources;
    By mutually converting the address of the first memory area allocated to the virtual machine and the address of the second memory area that is the real memory of the first memory area, data input to the external device is performed. A data transfer process for directly transferring data from the input / output device for controlling output to the first memory area allocated to the virtual machine;
    Memory area transfer processing for acquiring data stored in a first memory area allocated to the virtual machine to be moved and transferring the data to the first information processing apparatus;
    A transfer data detection process for detecting data directly transferred from the input / output device to the first memory area allocated to the virtual machine;
    An update information registration process for generating update information related to the first memory area to be changed by the data detected by the transfer data detection process, and storing the update information in an update information storage unit;
    Update information acquisition processing for acquiring update information from the update information storage means;
    Based on the update information acquired by the update information acquisition process, the update data changed by the data detected by the transfer data detection process in the first memory area allocated to the virtual machine to be moved is the first data Update data transfer processing to be transferred to the information processing device;
    Is executed by the arithmetic processing unit.

Images