US20180173646A1 - Memory management in virtualized computing systems having processors with more than two hierarchical privilege levels - Google Patents
Memory management in virtualized computing systems having processors with more than two hierarchical privilege levels Download PDFInfo
- Publication number
- US20180173646A1 US20180173646A1 US15/383,572 US201615383572A US2018173646A1 US 20180173646 A1 US20180173646 A1 US 20180173646A1 US 201615383572 A US201615383572 A US 201615383572A US 2018173646 A1 US2018173646 A1 US 2018173646A1
- Authority
- US
- United States
- Prior art keywords
- address
- translation scheme
- address translation
- hypervisor
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013519 translation Methods 0.000 claims abstract description 147
- 230000014616 translation Effects 0.000 claims abstract description 147
- 238000000034 method Methods 0.000 claims abstract description 34
- 238000013507 mapping Methods 0.000 description 18
- 238000010586 diagram Methods 0.000 description 12
- 238000007726 management method Methods 0.000 description 10
- 230000008569 process Effects 0.000 description 9
- 238000004590 computer program Methods 0.000 description 5
- 239000008187 granular material Substances 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 208000000785 Invasive Pulmonary Aspergillosis Diseases 0.000 description 2
- 238000007792 addition Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000002955 isolation Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000001427 coherent effect Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000000116 mitigating effect Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/14—Protection against unauthorised use of memory or access to memory
- G06F12/1458—Protection against unauthorised use of memory or access to memory by checking the subject access rights
- G06F12/1491—Protection against unauthorised use of memory or access to memory by checking the subject access rights in a hierarchical protection system, e.g. privilege levels, memory rings
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/10—Address translation
- G06F12/1009—Address translation using page tables, e.g. page table structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/10—Address translation
- G06F12/1027—Address translation using associative or pseudo-associative address translation means, e.g. translation look-aside buffer [TLB]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/10—Address translation
- G06F12/109—Address translation for multiple virtual address spaces, e.g. segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/45583—Memory management, e.g. access or allocation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/10—Providing a specific technical effect
- G06F2212/1052—Security improvement
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/15—Use in a specific computing environment
- G06F2212/151—Emulated environment, e.g. virtual machine
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/65—Details of virtual memory and virtual address translation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/65—Details of virtual memory and virtual address translation
- G06F2212/651—Multi-level translation tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
- G06F2212/68—Details of translation look-aside buffer [TLB]
Definitions
- Computer virtualization is a technique that involves encapsulating a physical computing machine platform into virtual machine(s) executing under control of virtualization software on a hardware computing platform or “host” (collectively referred to as a “virtualized computing system”).
- a virtual machine (VM) provides virtual hardware abstractions for processor, memory, storage, and the like to a guest operating system (OS) and guest application(s) that run on the guest OS.
- the virtualization software also referred to as a “hypervisor,” includes one or more virtual machine monitors (VMMs) to provide execution environment(s) for the virtual machine(s).
- the hypervisor itself can be an OS having a kernel and user applications that run on the kernel.
- a virtualized computing system can include various software components executing on the hardware, including the hypervisor kernel, hypervisor user application(s), guest OS(s), and guest application(s).
- Some central processing unit(s) execute code at multiple hierarchical privilege levels, each of which provides a different set of constraints. Examples include CPUs compatible with the ARM®v7 and ARM®v8 (Instruction Set Architecture versions 7 and 8) hardware architectures, which are commercially available from ARM Holdings of Cambridge, England. The various software components of a virtualized computing system can execute at different privilege levels of the hardware architecture.
- the hypervisor performs a context switch, i.e., storing and restoring component-specific state, such as memory and processor state.
- Context switches have a performance impact that requires careful optimization and mitigation. Efficient management of page tables, exception vectors (i.e., fixed memory addresses to which execution is directed in response to interrupts and other processor exception events), and the like to optimize context switches across various software components noticeably improves the performance of the virtualized computing system.
- a strategy that judiciously assigns and manages hypervisor components across the privilege levels is desirable.
- the method includes: generating a page table hierarchy that includes address translations to first pages of the memory that store the kernel software and second pages of the memory that store the user software; configuring the processor to: 1) implement a first address translation scheme, which uses a first virtual address width, for a hypervisor privilege level; 2) implement a second address translation scheme, which uses a second virtual address width, for supervisor and user privilege levels, where the first virtual address width is larger than the second virtual address width; and 3) use the page table hierarchy for each of the first and second address translation schemes; and executing the kernel software at the hypervisor privilege level and the user software at the user privilege level.
- FIG. 1 is a block diagram depicting a virtualized computing system according to an embodiment.
- FIG. 2 is a block diagram depicting example fields in one or more of registers according to an embodiment.
- FIG. 3 is a block diagram depicting a page table descriptor according to an embodiment.
- FIG. 4 is a flow diagram depicting a method of memory management in a virtualized computing system according to an embodiment.
- FIG. 5 is a block diagram depicting a memory map according to an embodiment.
- FIG. 6 is a block diagram depicting a shared page table hierarchy according to an embodiment.
- FIG. 1 is a block diagram depicting a virtualized computing system 100 according to an embodiment.
- Virtualized computing system 100 includes a host computer 102 having a software platform 104 executing on a hardware platform 106 .
- Hardware platform 106 may include conventional components of a computing device, such as a central processing unit (CPU) 108 and system memory 110 , as well as a storage system (storage), input/output devices, and the like (not shown).
- CPU 108 is configured to execute instructions, for example, executable instructions that perform one or more operations described herein and may be stored in system memory 110 and the storage system.
- System memory 110 is a device allowing information, such as executable instructions, virtual disks, configurations, and other data, to be stored and retrieved.
- System memory 110 may include, for example, one or more random access memory (RAM) modules.
- RAM random access memory
- CPU 108 includes one or more cores 112 , various registers 114 , and a memory management unit (MMU) 116 .
- Each core 112 is a microprocessor or like type processor element.
- Registers 114 include program execution registers for use by code executing on cores 112 and system/control registers for use by code to configure CPU 108 .
- Code is executed by CPU 108 on a core 112 at a particular privilege level (PL) of a hierarchy of privilege levels.
- PL privilege level
- CPU 108 is compliant with the ARM®v8 architecture or the like that includes four exception levels (ELs), which are defined as EL0, EL1, EL2, and EL3 in order of increasing code-execution privilege.
- ELs exception levels
- EL0 is referred to as “unprivileged execution” and execution at any of EL1, EL2, and EL3 is referred to as “privileged execution.”
- EL0 is an example of a “user PL;”
- EL1 is an example of a “supervisor PL;”
- EL2 is an example of a “hypervisor PL;” and
- EL3 is an example of a “secure PL.”
- CPU 108 supports a hierarchy of at least three hierarchical privilege levels, including the user PL, the supervisor PL, and the hypervisor PL in order of increasing execution privilege.
- Various examples described herein refer to a CPU having the ARM®v8 hardware architecture and executing in the 64-bit execution state (referred to as AArch64). It is to be understood that the memory management techniques described herein can be employed with CPUs having similar hardware architectures.
- MMU 116 implements memory management in the form of paging of system memory 110 .
- MMU 116 controls address translation and access permissions for memory accesses made by cores 112 .
- MMU 116 implements a plurality of address translation schemes based on privilege level (also referred to as “translation schemes”). Each translation scheme generally takes an input address (IA) and, if permitted based on the defined access permissions, returns an output address (OA). If an address translation cannot be performed (e.g., due to violation of the access permissions), MMU 116 generates an exception.
- MMU 116 is controlled by a plurality system registers in registers 114 .
- One type of translation scheme includes a single stage of address translation that receives a virtual address (VA) in a virtual address space and outputs a physical address (PA) in a physical address space.
- VA virtual address
- PA physical address
- the virtual address space is a logical address space managed by software and the physical address space includes the physical memory map of system memory 110 .
- Another type of translation scheme includes two stages of address translation. The first stage of address translation receives a VA and outputs an intermediate physical address (IPA) in an intermediate physical address space. The second stage of address translation receives an IPA and outputs a PA.
- the IPA address space is a logical address space managed by software.
- each translation scheme maps a VA to a PA using one or two stages of address translation.
- MMU 116 can include a translation lookaside buffer (TLB) 118 that caches address translations.
- TLB translation lookaside buffer
- MMU 116 implements different translation schemes depending on privilege level.
- CPUs compliant with the ARM®v8 architecture include a single stage translation scheme for code executing at EL2 (referred to herein as the “EL2 translation scheme”).
- the EL2 translation scheme includes a single stage that maps VAs to PAs and is controlled from EL2.
- Such CPUs also include a two-stage translation scheme for code executing at EL1 and EL0 (referred to herein as the “EL1/EL0 translation scheme”).
- the first stage of the EL1/EL0 translation scheme maps VAs to IPAs and is controlled from EL1.
- the second stage of the EL1/EL0 translation scheme maps IPAs to PAs and is controlled from EL2.
- MMU 116 implements a hypervisor PL translation scheme having a single stage and a supervisor/user PL translation scheme having one or two stages.
- the translation scheme stages can be enabled/disabled by setting fields in particular registers 114 .
- FIG. 2 is a block diagram depicting example fields in one or more of registers 114 according to an embodiment.
- the fields include a hypervisor PL stage 1 translation enable field 202 that enables/disables stage 1 of the hypervisor PL translation scheme (the only stage in that scheme).
- the fields include a supervisor/user PL stage 1 translation enable field 204 that enables/disables stage 1 of the supervisor/user PL translation scheme.
- the fields include a supervisor/user PL stage 2 translation enable field 206 that enables/disables stage 2 of the supervisor/user PL translation scheme.
- Code executing at the hypervisor PL can manipulate any of the fields 202 through 206 .
- Code executing at the supervisor PL can manipulate only the supervisor/user PL stage 1 translation enable field 204 .
- Code executing at the user PL cannot manipulate any of the fields 202 through 206 .
- the ARM®v8 architecture describes a field M in a register SCTLR_EL2 that enables/disables EL2 stage 1 address translation; a field M in a register SCTLR_EL1 that enables/disables EL1/EL0 stage 1 address translation; and a field VM in a register HCR_EL2 that enables/disables EL1/EL0 stage 2 address translation.
- MMU 116 divides system memory 110 into pages 120 .
- a “page” is the smallest block of memory for which an IA-to-OA mapping can be specified.
- Each page (also referred to herein as a “memory page”) includes a plurality of separately addressable data words, each of which in turn includes one or more bytes.
- Each address includes a set of most significant bits (MSBs) that specifies a memory page and a set of least significant bits that specifies an offset into the memory page.
- MSBs most significant bits
- Each address translation involves translating a set of MSBs of the IA into a set of MSBs of an OA.
- CPU 108 can support one or more page sizes.
- CPUs compliant with the ARM®v8 architecture can support 4 kilobyte (KB), 16 KB, and 64 KB translation granules, which are the minimum page sizes that can be translated (software configurable).
- CPUs compliant with the ARM®v8 architecture can also translate large pages (regions or blocks), such as 2 megabyte (MB) and 1 gigabyte (GB) regions for the 4 KB granule size, 32 MB regions for the 16 KB granule size, and 512 MB regions for the 64 KB granule size.
- Other CPUs may support other page sizes.
- the width of the IA is configurable for each address translation scheme. In embodiments, the VA address width for stage 1 of the supervisor/user PL translation scheme is less than that of stage 1 of the hypervisor PL translation scheme, as described further below.
- Each enabled stage of address translation in a translation scheme uses memory mapped tables referred to as page tables 122 .
- a given address translation requires one or more lookups of page tables 122 (referred to as one or more levels of lookup).
- a page table walk is the set of lookups required to translate a VA to a PA.
- Page tables 122 are organized into hierarchies, where each page table hierarchy includes a base table and a plurality of additional tables corresponding to one or more additional levels.
- the ARM®v8 architecture specifies up to four levels of page tables referred to as level 0 through level 3 tables. The number of levels in a page table hierarchy depends on the page size and the width of the IA.
- the fields of registers 114 include a hypervisor PL page table base field 208 that stores a physical address of a base page table for use with the hypervisor PL translation scheme.
- the fields also include a supervisor/user PL page table base field 210 that stores a physical address of a base page table for use with the supervisor/user PL translation scheme.
- the fields further include a hypervisor PL VA width field 212 that specifies the width of VAs for the hypervisor PL translation scheme.
- the fields further include a supervisor/user PL VA width field 214 that specifies the width of VAs for the supervisor/user PL translation scheme.
- Code executing at the hypervisor PL can manipulate any of the fields 208 through 214 .
- Code executing at the supervisor PL can manipulate only the fields 210 and 214 .
- Code executing at the user PL cannot manipulate any of the fields 210 through 214 .
- the ARM®v8 architecture specifies a register TTBR0_EL2 that stores an address of a base page table for EL2 stage 1 address translations and a register TTBR0_EL1 that stores an address of a base page table for EL1/EL0 stage 1 address translations.
- the ARM®v8 architecture further specifies a register VTTBR_EL2 that stores an address of a base page table for EL1/EL0 stage 2 address translations.
- the ARM®v8 architecture further specifies a field T0SZ in a register TCR_EL2 that dictates the VA width for the EL2 address translation scheme.
- the ARM®v8 architecture specifies a field T0SZ in a register TCR_EL1 that dictates the VA width for the EL1/EL0 address translation scheme.
- the maximum width of any IA is 48 bits. Other CPUs can have other maximum IA widths.
- the 48-bit virtual address space ranges from 0x0000_0000_0000_0000 to 0x0000_FFFF_FFFF_FFFF (where the prefix “0x” denotes a hexadecimal number).
- the virtual address space for the EL1/EL0 address translation scheme is split into two 48-bit subranges within a full 64-bit address range: the bottom 48-bit VA subrange is between 0x0000_0000_0000 and 0x0000_FFFF_FFFF_FFFF and the top 48-bit VA subrange is between 0xFFFF_0000_0000 and 0xFFFF_FFFF_FFFF.
- the register TTBR0_EL1 stores an address of a base page table for the bottom 48-bit subrange of the EL1/EL0 stage 1 translation scheme.
- a register TTBR1_EL1 stores an address of a base page table for the top 48-bit subrange of the EL1/EL0 stage 1 translation scheme.
- each page table 122 includes a set of descriptors.
- Each descriptor generally includes an address and access permissions associated with the address.
- each descriptor can describe another page table, a block of memory (e.g., a block of multiple pages), or a page.
- FIG. 3 is a block diagram depicting a page table descriptor 300 according to an embodiment.
- Page table descriptor 300 includes access permissions field 302 and an address field 304 .
- Access permissions field 302 can include read, write, and execute permissions.
- Address field 302 includes an OA that references another page table, a block of pages, or a page, depending on the level of the corresponding page table.
- Page table descriptor 300 can include various other field(s) 306 , such as memory attribute fields (e.g., memory attributes that control the memory type, access to caches, whether the memory is coherent, etc.).
- the format of page table descriptor 300 can differ depending on the particular translation scheme.
- access permissions 302 have a different format in the hypervisor PL translation scheme versus the supervisor/user PL translation scheme.
- the supervisor/user PL translation scheme access permissions 302 can specify read, write, and execute access differently for unprivileged execution versus privileged execution.
- access permissions 302 can specify read, write, and execute access regardless of privilege.
- access permissions 302 include one or more reserved bits 308 that effectively disable any distinction between unprivileged and privileged execution.
- software platform 104 includes a virtualization layer that abstracts processor, memory, storage, and networking resources of hardware platform 106 into one or more virtual machines (“VMs”) 132 that run concurrently on host computer 102 .
- VMs 132 run on top of the virtualization layer, referred to herein as a hypervisor 130 , which enables sharing of the hardware resources by VMs 132 .
- hypervisor 130 is a VMware ESXiTM hypervisor provided as part of the VMware vSphere® solution made commercially available from VMware, Inc. of Palo Alto, Calif. (although it should be recognized that any other virtualization technologies, including Xen® and Microsoft Hyper-V® virtualization technologies may be utilized consistent with the teachings herein).
- Each VM 132 supported by hypervisor 130 includes guest software (also referred to as guest code) that runs on the virtualized resources supported by hardware platform 106 .
- the guest software of each VM 132 includes a guest OS 134 and one or more applications (apps) 136 .
- Guest OS 134 can be any commodity operating system known in the art, such as such as Linux®, Microsoft Windows®, Mac OS®, or the like.
- Hypervisor 130 includes a boot loader 138 , a kernel 140 , one or more user programs 142 , and virtual machine monitors (VMMs) 144 .
- hypervisor 130 can include any number of components and the functionality implemented by boot loader 138 , kernel 140 , user program(s) 142 , and VMMs 144 may be distributed in any technically feasible manner between the hypervisor components.
- Kernel 140 provides operating system functionality (e.g., process creation and control, file system, process threads, etc.), as well as CPU scheduling and memory scheduling across guest software in VMs 132 , VMMs 144 , and user program(s) 142 .
- VMMs 144 implement the virtual system support needed to coordinate operations between hypervisor 130 and VMs 132 .
- Each VMM 144 manages a corresponding virtual hardware platform that includes emulated hardware, such as virtual CPUs (vCPUs) and guest physical memory.
- vCPUs virtual CPUs
- Each virtual hardware platform supports the installation of guest software in a corresponding VM 132 .
- Each user program 142 is a “native” process that runs on kernel 140 .
- User program(s) 142 execute in an environment provided by kernel 140 , but outside of environments provided for VMs 132 .
- each VMM 140 is paired with a corresponding process, known as a VMX process, which executes as a user program 142 .
- each VMM may be created by, entered into, and exited via the paired VMX application.
- a direct console user interface (DCUI) process which can be used to configure hypervisor 130 , executes as a user program 142 .
- DCUI direct console user interface
- Boot loader 138 includes code that is executed upon power on or reset of hardware platform 106 .
- Boot loader 138 can include one or more stages that setup CPU 108 and load kernel 140 into system memory 110 .
- boot loader 138 can configure MMU 116 to manage memory, as described further herein.
- some or all of the functions implemented by boot loader 138 to configure MMU 116 can be performed by kernel 140 after boot loader 138 loads and executes kernel 140 .
- boot loader 138 configures CPU 108 to implement the hypervisor, supervisor, and user PLs.
- Boot loader 138 configures kernel 140 to execute at the hypervisor PL (e.g., EL2).
- Kernel 140 configures VMMs 144 to execute at the hypervisor PL.
- VMMs 144 configure guest OS 134 in each VM 132 to execute at the supervisor PL (e.g., EL1).
- Guest OS 134 in each VM 132 configures applications 136 to execute at the user PL (e.g., EL0).
- Kernel 140 configures user program(s) 142 to execute at the user PL (e.g., EL0).
- kernel 140 executing at the hypervisor PL (e.g., EL2) and user program(s) 142 executing at the user PL (e.g., EL0).
- Kernel 140 includes kernel code and data (kernel code/data 126 ) that are stored in pages 120 of system memory 110 .
- user program(s) 142 include user code and data (user code/data 128 ) that are stored in pages 120 of system memory 110 .
- Kernel code/data 126 is generally referred to as “kernel software,” and user code/data 128 is generally referred to as “user software.”
- Guest software executing in VMs 132 e.g., guest OS 132
- VMMs 144 e.g., boot loader 138 , and/or kernel 140 can generate page tables 122 , which are also stored in pages 120 .
- page tables 122 include shared page tables 124 .
- kernel 140 can create and manipulate shared page tables 124 that are shared between kernel 140 and user program(s) 142 .
- MMU 116 uses two different translation schemes for code executing at the hypervisor PL versus code executing at the user PL.
- MMU 116 implements a single-stage translation scheme that maps VAs to PAs and requires page table descriptors having a first format (e.g., page table descriptor 300 with reserved bit(s) 308 ).
- MMU 116 implements a single-stage or two-stage translation scheme that maps VAs to PAs.
- Stage one of the supervisor/user PL translation scheme requires page table descriptors similar to those of stage one of the hypervisor PL translation scheme, but without reserving reserved bit(s) 308 .
- Stage two of the supervisor/user PL translation scheme requires a different format for page table descriptors that is incompatible with stage one address translations.
- stage one of the hypervisor PL translation scheme e.g., stage one of the EL2 translation scheme
- a second type of page table hierarchy for stage one of the supervisor/user PL translation scheme e.g., stage one of the EL1/EL0 translation scheme
- a third type of page table hierarchy for stage two of the supervisor/user PL translation scheme e.g., stage 2 of the EL1/EL0 translation scheme
- stage one of the EL1/EL0 translation scheme can be disabled (by disabling EL1), which forces EL0 translations to be done using stage 2 of the EL1/EL0 translation scheme.
- Such scenarios are inefficient. Since user program(s) 142 do not execute within the context of a VM there is no concept of guest physical memory. Thus, there is no need for the second stage of the supervisor/user PL translation scheme.
- the second stage of the supervisor/user PL translation scheme is disabled.
- a first type of page table hierarchy is established for stage one of the hypervisor PL translation scheme and a second type of page table hierarchy is established for stage one of the supervisor/user PL translation scheme.
- Each stage-one translation scheme maps VAs to PAs and uses page table descriptors having a similar format (with the exception of reserved bit(s) 308 ).
- kernel 140 must replicate the mappings associated with user/code data 128 across both page table hierarchies for the hypervisor PL (e.g., EL2) and the supervisor/user PL (e.g., EL1/EL0). Otherwise, kernel 140 would not be able to access user code/data 128 .
- the duplicate mappings still result in an inefficient memory management scheme.
- the third scenario described above can be modified to share a portion of the supervisor/user PL page table hierarchy with the hypervisor PL page table hierarchy.
- hypervisor PL page table base register 208 and supervisor/user PL page table base register 210 each point to different base page tables.
- the base page table for the supervisor/user PL page table hierarchy includes only descriptors for mappings to user code/data 128 .
- the base page table for the hypervisor PL page table hierarchy includes descriptors for mappings to kernel code/data 126 , as well as replicated descriptors for mappings to user code/data 128 .
- the replicated user code/data descriptors point to the supervisor/user PL page table hierarchy.
- kernel 140 must ensure that none of the mappings to kernel code/data 126 make it to the base page table of the supervisor/user PL page table hierarchy. Otherwise, user program(s) 142 would be able to access kernel code/data 126 . Thus, extra code must be written and maintained for kernel 140 in order to perform this software-based management of the page tables.
- FIG. 4 is a flow diagram depicting a method 400 of memory management in a virtualized computing system according to an embodiment.
- Method 400 may be performed by hypervisor 130 executing in virtualized computing system 100 of FIG. 1 . Steps in method 400 may be performed by kernel 140 .
- Method 400 begins at step 402 , where hypervisor 130 stores kernel code/data 126 in pages 120 of a kernel address space and user code/data 128 in pages 120 of a user address space. Additional steps of method 400 are described further below. An example memory map is now described with respect to FIG. 5 .
- FIG. 5 is a block diagram depicting a memory map 500 according to an embodiment.
- Memory map 500 includes a kernel address space 502 .
- Kernel address space 502 ranges from a minimum VA address to a maximum VA address in a VA space defined by a kernel VA width.
- the kernel VA width can be 48 bits
- the minimum VA address can be 0x0000_0000_0000_0000
- the maximum address can be 0x0000_FFFF_FFFF_FFFF.
- Kernel VA width can be other values resulting in a smaller or larger VA address range.
- Kernel address space 502 includes a user address space 504 .
- User address space 504 ranges from the minimum VA address to a maximum user VA address in a VA address space defined by a user VA width.
- the user VA width can be 46 bits and the maximum user VA address can be 0x0000_3FFF_FFFF_FFFF.
- the user VA width can be other values resulting in a smaller or larger user VA address range as constrained by the kernel VA address range.
- Kernel code/data 126 is stored within kernel address space 502 , but outside of user address space 504 . In the examples above, kernel code/data 126 can be stored starting at 0x0000_4000_0000_0000.
- User code/data 128 is stored within user address space 504 . In the example above, user code/data 128 is stored below at or below 0x0000_3FFF_0000_0000.
- hypervisor 130 generates a shared page table hierarchy for kernel and user software.
- method 400 is described in terms of a single shared page table hierarchy.
- kernel 140 can generate multiple shared page table hierarchies corresponding to multiple user programs 142 (e.g., one shared hierarchy per user program 142 ).
- the shared page table includes both mappings to kernel code/data 126 and mappings to user code/data 128 .
- the shared page table hierarchy includes descriptors having a format dictated by the hypervisor PL translation scheme (e.g., the stage 1 EL2 translation scheme for ARM®v8) (step 406 ). Additional steps of method 400 are described further below.
- An example shared page table hierarchy is now described with respect to FIG. 6 .
- FIG. 6 is a block diagram depicting a shared page table hierarchy 600 according to an embodiment.
- Shared page table hierarchy 600 includes a base page table 602 .
- Base page table 602 includes descriptors for VAs between the minimum VA address and the maximum VA address of kernel address space 502 .
- the descriptors are in the format dictated by the hypervisor PL translation scheme (e.g., the descriptor 300 having reserved bits 308 ).
- the descriptors in base page table 602 provide kernel mappings 604 and user mappings 606 .
- Kernel mappings 604 correspond to VAs between the maximum user VA address (exclusive) and the maximum VA address (inclusive).
- User mappings 606 correspond to VAs between the minimum VA address (inclusive) and the maximum user VA address (inclusive).
- Kernel mappings 604 point to additional page tables in additional kernel page table level(s) 608 .
- User mappings 606 point to additional page tables in additional user page table level(s) 610 .
- the page tables in additional kernel page table level(s) 608 include mappings that result in address translations to kernel code/data 126 .
- the page tables in additional user page table level(s) 610 include mappings that result in address translations to user code/data 128 .
- hypervisor 130 configures CPU 108 to implement a hypervisor PL translation scheme, which uses the kernel VA width, for code executing at the hypervisor PL.
- Hypervisor 130 performs such configuration by manipulating one or more registers 114 , e.g., hypervisor PL stage 1 translation enable 202 and hypervisor PL VA width 212 .
- hypervisor 130 can configure CPU 108 to implement stage 1 of the EL2 translation scheme with a VA width of 48 bits for code executing at EL2 (e.g., kernel 140 ).
- hypervisor 130 configures CPU 108 to implement a supervisor/user translation scheme, which uses a second VA width that is less than the first VA width, for code executing at the supervisor and user PLs.
- Hypervisor 130 performs such configuration by manipulating one or more registers 114 , e.g., supervisor/user PL stage 1 translation enable 204 and supervisor/user PL VA width 214 .
- hypervisor 130 disables second stage address translation (e.g., by manipulating supervisor/user PL stage 2 translation enable 206 ).
- hypervisor 130 can configure CPU 108 to implement stage 1 of the EL1/EL0 translation scheme with a VA width of 46 bits for code executing at EL0 (e.g., user programs(s) 142 ). Hypervisor 130 can configure CPU 108 to disable stage 2 of the EL1/EL0 translation scheme.
- hypervisor 130 configures CPU 108 to use the shared page table hierarchy for each of the hypervisor PL and supervisor/user PL translation schemes.
- Hypervisor 130 performs such configuration by manipulating one or more registers 114 , e.g., hypervisor PL page table base 208 and supervisor/user PL page table base 210 (step 416 ).
- Each page table base register stores an address of the base page table of the shared page table hierarchy (e.g., base page table 602 ).
- hypervisor 130 executes kernel 140 at the hypervisor PL (e.g., EL2) and user program(s) 142 at the user PL (e.g., EL0).
- hypervisor 130 does not execute code at the supervisor PL (e.g., EL1).
- user program(s) 142 access shared page tables 124 using the user VA address width and kernel 140 accesses shared page tables 124 using the kernel VA address width.
- MMU 116 issues a fault, since kernel code/data 126 is stored at VAs outside of the user VA address range.
- Kernel 140 can translate VAs mapped to the entire VA address space (e.g., to access both kernel code/data 126 and user code/data 128 ).
- hypervisor 130 efficiently manages shared page tables for both kernel 140 and user program(s) 142 using hardware features of CPU 108 .
- Kernel 140 does not require extra code to ensure that kernel mappings are not accessible by user program(s) 142 . Rather, MMU 116 will issue a fault if user program(s) 142 attempt to translate a VA that is mapped to kernel code/data 126 .
- hypervisor 130 stores kernel code/data 126 outside of the user VA address space.
- hypervisor 130 can generate any number of shared page table hierarchies for any number of user program(s) 142 .
- Hypervisor 130 can manipulate base page table registers of CPU 108 for context switches between user programs 142 to refer to the different shared page tables. For each context switch, the base page table registers refer to the same base page table in a given shared page table hierarchy. Hypervisor 130 does not need to maintain separate base page tables for kernel and user programs.
- the various embodiments described herein may employ various computer-implemented operations involving data stored in computer systems. For example, these operations may require physical manipulation of physical quantities—usually, though not necessarily, these quantities may take the form of electrical or magnetic signals, where they or representations of them are capable of being stored, transferred, combined, compared, or otherwise manipulated. Further, such manipulations are often referred to in terms, such as producing, identifying, determining, or comparing. Any operations described herein that form part of one or more embodiments of the invention may be useful machine operations.
- one or more embodiments of the invention also relate to a device or an apparatus for performing these operations.
- the apparatus may be specially constructed for specific required purposes, or it may be a general purpose computer selectively activated or configured by a computer program stored in the computer.
- various general purpose machines may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.
- One or more embodiments of the present invention may be implemented as one or more computer programs or as one or more computer program modules embodied in one or more computer readable media.
- the term computer readable medium refers to any data storage device that can store data which can thereafter be input to a computer system—computer readable media may be based on any existing or subsequently developed technology for embodying computer programs in a manner that enables them to be read by a computer.
- Examples of a computer readable medium include a hard drive, network attached storage (NAS), read-only memory, random-access memory (e.g., a flash memory device), a CD (Compact Discs)—CD-ROM, a CD-R, or a CD-RW, a DVD (Digital Versatile Disc), a magnetic tape, and other optical and non-optical data storage devices.
- the computer readable medium can also be distributed over a network coupled computer system so that the computer readable code is stored and executed in a distributed fashion.
- Virtualization systems in accordance with the various embodiments may be implemented as hosted embodiments, non-hosted embodiments or as embodiments that tend to blur distinctions between the two, are all envisioned.
- various virtualization operations may be wholly or partially implemented in hardware.
- a hardware implementation may employ a look-up table for modification of storage access requests to secure non-disk data.
- Certain embodiments as described above involve a hardware abstraction layer on top of a host computer.
- the hardware abstraction layer allows multiple contexts to share the hardware resource.
- these contexts are isolated from each other, each having at least a user application running therein.
- the hardware abstraction layer thus provides benefits of resource isolation and allocation among the contexts.
- virtual machines are used as an example for the contexts and hypervisors as an example for the hardware abstraction layer.
- each virtual machine includes a guest operating system in which at least one application runs.
- OS-less containers see, e.g., www.docker.com).
- OS-less containers implement operating system-level virtualization, wherein an abstraction layer is provided on top of the kernel of an operating system on a host computer.
- the abstraction layer supports multiple OS-less containers each including an application and its dependencies.
- Each OS-less container runs as an isolated process in userspace on the host operating system and shares the kernel with other containers.
- the OS-less container relies on the kernel's functionality to make use of resource isolation (CPU, memory, block I/O, network, etc.) and separate namespaces and to completely isolate the application's view of the operating environments.
- resource isolation CPU, memory, block I/O, network, etc.
- By using OS-less containers resources can be isolated, services restricted, and processes provisioned to have a private view of the operating system with their own process ID space, file system structure, and network interfaces.
- Multiple containers can share the same kernel, but each container can be constrained to only use a defined amount of resources such as CPU, memory and I/O.
- virtualized computing instance as used herein is meant to encompass both
- the virtualization software can therefore include components of a host, console, or guest operating system that performs virtualization functions.
- Plural instances may be provided for components, operations or structures described herein as a single instance. Boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the invention(s).
- structures and functionality presented as separate components in exemplary configurations may be implemented as a combined structure or component.
- structures and functionality presented as a single component may be implemented as separate components.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computer Security & Cryptography (AREA)
- Memory System Of A Hierarchy Structure (AREA)
Abstract
Description
- Computer virtualization is a technique that involves encapsulating a physical computing machine platform into virtual machine(s) executing under control of virtualization software on a hardware computing platform or “host” (collectively referred to as a “virtualized computing system”). A virtual machine (VM) provides virtual hardware abstractions for processor, memory, storage, and the like to a guest operating system (OS) and guest application(s) that run on the guest OS. The virtualization software, also referred to as a “hypervisor,” includes one or more virtual machine monitors (VMMs) to provide execution environment(s) for the virtual machine(s). The hypervisor itself can be an OS having a kernel and user applications that run on the kernel. Thus, a virtualized computing system can include various software components executing on the hardware, including the hypervisor kernel, hypervisor user application(s), guest OS(s), and guest application(s).
- Some central processing unit(s) execute code at multiple hierarchical privilege levels, each of which provides a different set of constraints. Examples include CPUs compatible with the ARM®v7 and ARM®v8 (Instruction Set Architecture versions 7 and 8) hardware architectures, which are commercially available from ARM Holdings of Cambridge, England. The various software components of a virtualized computing system can execute at different privilege levels of the hardware architecture.
- As part of transitioning execution control from one software component to another, the hypervisor performs a context switch, i.e., storing and restoring component-specific state, such as memory and processor state. Context switches have a performance impact that requires careful optimization and mitigation. Efficient management of page tables, exception vectors (i.e., fixed memory addresses to which execution is directed in response to interrupts and other processor exception events), and the like to optimize context switches across various software components noticeably improves the performance of the virtualized computing system. To perform efficient context switches in architectures with multiple hierarchical privilege levels, a strategy that judiciously assigns and manages hypervisor components across the privilege levels is desirable.
- One or more embodiments provide memory management in virtualized computing systems having processors with more than two hierarchical privilege levels. In an embodiment, a method of memory management in a virtualized computing system is described. The virtualized computing system includes a hypervisor executing on a hardware platform, where the hardware platform includes a processor and a memory, and the hypervisor includes kernel code and user code. The method includes: generating a page table hierarchy that includes address translations to first pages of the memory that store the kernel software and second pages of the memory that store the user software; configuring the processor to: 1) implement a first address translation scheme, which uses a first virtual address width, for a hypervisor privilege level; 2) implement a second address translation scheme, which uses a second virtual address width, for supervisor and user privilege levels, where the first virtual address width is larger than the second virtual address width; and 3) use the page table hierarchy for each of the first and second address translation schemes; and executing the kernel software at the hypervisor privilege level and the user software at the user privilege level.
- Further embodiments include a non-transitory computer-readable storage medium comprising instructions that cause a computer system to carry out the above method, as well as a computer system configured to carry out the above method.
-
FIG. 1 is a block diagram depicting a virtualized computing system according to an embodiment. -
FIG. 2 is a block diagram depicting example fields in one or more of registers according to an embodiment. -
FIG. 3 is a block diagram depicting a page table descriptor according to an embodiment. -
FIG. 4 is a flow diagram depicting a method of memory management in a virtualized computing system according to an embodiment. -
FIG. 5 is a block diagram depicting a memory map according to an embodiment. -
FIG. 6 is a block diagram depicting a shared page table hierarchy according to an embodiment. - To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements disclosed in one embodiment may be beneficially utilized on other embodiments without specific recitation.
-
FIG. 1 is a block diagram depicting avirtualized computing system 100 according to an embodiment. Virtualizedcomputing system 100 includes ahost computer 102 having asoftware platform 104 executing on ahardware platform 106.Hardware platform 106 may include conventional components of a computing device, such as a central processing unit (CPU) 108 andsystem memory 110, as well as a storage system (storage), input/output devices, and the like (not shown).CPU 108 is configured to execute instructions, for example, executable instructions that perform one or more operations described herein and may be stored insystem memory 110 and the storage system.System memory 110 is a device allowing information, such as executable instructions, virtual disks, configurations, and other data, to be stored and retrieved.System memory 110 may include, for example, one or more random access memory (RAM) modules. -
CPU 108 includes one ormore cores 112,various registers 114, and a memory management unit (MMU) 116. Eachcore 112 is a microprocessor or like type processor element.Registers 114 include program execution registers for use by code executing oncores 112 and system/control registers for use by code to configureCPU 108. Code is executed byCPU 108 on acore 112 at a particular privilege level (PL) of a hierarchy of privilege levels. In an embodiment,CPU 108 is compliant with the ARM®v8 architecture or the like that includes four exception levels (ELs), which are defined as EL0, EL1, EL2, and EL3 in order of increasing code-execution privilege. Execution at EL0 is referred to as “unprivileged execution” and execution at any of EL1, EL2, and EL3 is referred to as “privileged execution.” EL0 is an example of a “user PL;” EL1 is an example of a “supervisor PL;” EL2 is an example of a “hypervisor PL;” and EL3 is an example of a “secure PL.” In general,CPU 108 supports a hierarchy of at least three hierarchical privilege levels, including the user PL, the supervisor PL, and the hypervisor PL in order of increasing execution privilege. Various examples described herein refer to a CPU having the ARM®v8 hardware architecture and executing in the 64-bit execution state (referred to as AArch64). It is to be understood that the memory management techniques described herein can be employed with CPUs having similar hardware architectures. - MMU 116 implements memory management in the form of paging of
system memory 110. MMU 116 controls address translation and access permissions for memory accesses made bycores 112. MMU 116 implements a plurality of address translation schemes based on privilege level (also referred to as “translation schemes”). Each translation scheme generally takes an input address (IA) and, if permitted based on the defined access permissions, returns an output address (OA). If an address translation cannot be performed (e.g., due to violation of the access permissions),MMU 116 generates an exception. MMU 116 is controlled by a plurality system registers inregisters 114. - One type of translation scheme includes a single stage of address translation that receives a virtual address (VA) in a virtual address space and outputs a physical address (PA) in a physical address space. The virtual address space is a logical address space managed by software and the physical address space includes the physical memory map of
system memory 110. Another type of translation scheme includes two stages of address translation. The first stage of address translation receives a VA and outputs an intermediate physical address (IPA) in an intermediate physical address space. The second stage of address translation receives an IPA and outputs a PA. The IPA address space is a logical address space managed by software. In general, each translation scheme maps a VA to a PA using one or two stages of address translation. MMU 116 can include a translation lookaside buffer (TLB) 118 that caches address translations. - MMU 116 implements different translation schemes depending on privilege level. For example, CPUs compliant with the ARM®v8 architecture include a single stage translation scheme for code executing at EL2 (referred to herein as the “EL2 translation scheme”). The EL2 translation scheme includes a single stage that maps VAs to PAs and is controlled from EL2. Such CPUs also include a two-stage translation scheme for code executing at EL1 and EL0 (referred to herein as the “EL1/EL0 translation scheme”). The first stage of the EL1/EL0 translation scheme maps VAs to IPAs and is controlled from EL1. The second stage of the EL1/EL0 translation scheme maps IPAs to PAs and is controlled from EL2. In general,
MMU 116 implements a hypervisor PL translation scheme having a single stage and a supervisor/user PL translation scheme having one or two stages. The translation scheme stages can be enabled/disabled by setting fields inparticular registers 114. -
FIG. 2 is a block diagram depicting example fields in one or more ofregisters 114 according to an embodiment. The fields include ahypervisor PL stage 1 translation enablefield 202 that enables/disablesstage 1 of the hypervisor PL translation scheme (the only stage in that scheme). The fields include a supervisor/user PL stage 1 translation enable field 204 that enables/disablesstage 1 of the supervisor/user PL translation scheme. The fields include a supervisor/user PL stage 2 translation enable field 206 that enables/disables stage 2 of the supervisor/user PL translation scheme. Code executing at the hypervisor PL can manipulate any of thefields 202 through 206. Code executing at the supervisor PL can manipulate only the supervisor/user PL stage 1 translation enable field 204. Code executing at the user PL cannot manipulate any of thefields 202 through 206. For example, the ARM®v8 architecture describes a field M in a register SCTLR_EL2 that enables/disablesEL2 stage 1 address translation; a field M in a register SCTLR_EL1 that enables/disables EL1/EL0 stage 1 address translation; and a field VM in a register HCR_EL2 that enables/disables EL1/EL0 stage 2 address translation. - Returning to
FIG. 1 ,MMU 116 dividessystem memory 110 intopages 120. A “page” is the smallest block of memory for which an IA-to-OA mapping can be specified. Each page (also referred to herein as a “memory page”) includes a plurality of separately addressable data words, each of which in turn includes one or more bytes. Each address includes a set of most significant bits (MSBs) that specifies a memory page and a set of least significant bits that specifies an offset into the memory page. Each address translation involves translating a set of MSBs of the IA into a set of MSBs of an OA.CPU 108 can support one or more page sizes. For example, CPUs compliant with the ARM®v8 architecture can support 4 kilobyte (KB), 16 KB, and 64 KB translation granules, which are the minimum page sizes that can be translated (software configurable). CPUs compliant with the ARM®v8 architecture can also translate large pages (regions or blocks), such as 2 megabyte (MB) and 1 gigabyte (GB) regions for the 4 KB granule size, 32 MB regions for the 16 KB granule size, and 512 MB regions for the 64 KB granule size. Other CPUs may support other page sizes. In addition, the width of the IA is configurable for each address translation scheme. In embodiments, the VA address width forstage 1 of the supervisor/user PL translation scheme is less than that ofstage 1 of the hypervisor PL translation scheme, as described further below. - Each enabled stage of address translation in a translation scheme uses memory mapped tables referred to as page tables 122. A given address translation requires one or more lookups of page tables 122 (referred to as one or more levels of lookup). A page table walk is the set of lookups required to translate a VA to a PA. Page tables 122 are organized into hierarchies, where each page table hierarchy includes a base table and a plurality of additional tables corresponding to one or more additional levels. For example, the ARM®v8 architecture specifies up to four levels of page tables referred to as level 0 through level 3 tables. The number of levels in a page table hierarchy depends on the page size and the width of the IA.
- As shown in
FIG. 2 , the fields ofregisters 114 include a hypervisor PL pagetable base field 208 that stores a physical address of a base page table for use with the hypervisor PL translation scheme. The fields also include a supervisor/user PL page table base field 210 that stores a physical address of a base page table for use with the supervisor/user PL translation scheme. The fields further include a hypervisor PLVA width field 212 that specifies the width of VAs for the hypervisor PL translation scheme. The fields further include a supervisor/user PLVA width field 214 that specifies the width of VAs for the supervisor/user PL translation scheme. Code executing at the hypervisor PL can manipulate any of thefields 208 through 214. Code executing at the supervisor PL can manipulate only thefields 210 and 214. Code executing at the user PL cannot manipulate any of the fields 210 through 214. - For example, the ARM®v8 architecture specifies a register TTBR0_EL2 that stores an address of a base page table for
EL2 stage 1 address translations and a register TTBR0_EL1 that stores an address of a base page table for EL1/EL0 stage 1 address translations. The ARM®v8 architecture further specifies a register VTTBR_EL2 that stores an address of a base page table for EL1/EL0 stage 2 address translations. The ARM®v8 architecture further specifies a field T0SZ in a register TCR_EL2 that dictates the VA width for the EL2 address translation scheme. Likewise, the ARM®v8 architecture specifies a field T0SZ in a register TCR_EL1 that dictates the VA width for the EL1/EL0 address translation scheme. In the ARM®v8 architecture, the maximum width of any IA is 48 bits. Other CPUs can have other maximum IA widths. For the EL2 address translation scheme, the 48-bit virtual address space ranges from 0x0000_0000_0000_0000 to 0x0000_FFFF_FFFF_FFFF (where the prefix “0x” denotes a hexadecimal number). Note that the virtual address space for the EL1/EL0 address translation scheme is split into two 48-bit subranges within a full 64-bit address range: the bottom 48-bit VA subrange is between 0x0000_0000_0000_0000 and 0x0000_FFFF_FFFF_FFFF and the top 48-bit VA subrange is between 0xFFFF_0000_0000_0000 and 0xFFFF_FFFF_FFFF_FFFF. The register TTBR0_EL1 stores an address of a base page table for the bottom 48-bit subrange of the EL1/EL0 stage 1 translation scheme. A register TTBR1_EL1 stores an address of a base page table for the top 48-bit subrange of the EL1/EL0 stage 1 translation scheme. - Returning to
FIG. 1 , each page table 122 includes a set of descriptors. Each descriptor generally includes an address and access permissions associated with the address. Depending on the level of the page table, each descriptor can describe another page table, a block of memory (e.g., a block of multiple pages), or a page.FIG. 3 is a block diagram depicting apage table descriptor 300 according to an embodiment.Page table descriptor 300 includesaccess permissions field 302 and anaddress field 304. Access permissions field 302 can include read, write, and execute permissions.Address field 302 includes an OA that references another page table, a block of pages, or a page, depending on the level of the corresponding page table.Page table descriptor 300 can include various other field(s) 306, such as memory attribute fields (e.g., memory attributes that control the memory type, access to caches, whether the memory is coherent, etc.). - The format of
page table descriptor 300 can differ depending on the particular translation scheme. In embodiments,access permissions 302 have a different format in the hypervisor PL translation scheme versus the supervisor/user PL translation scheme. In the supervisor/user PL translation scheme,access permissions 302 can specify read, write, and execute access differently for unprivileged execution versus privileged execution. In the hypervisor PL translation scheme,access permissions 302 can specify read, write, and execute access regardless of privilege. Thus, in the hypervisor PL translation scheme,access permissions 302 include one or more reserved bits 308 that effectively disable any distinction between unprivileged and privileged execution. - Returning to
FIG. 1 ,software platform 104 includes a virtualization layer that abstracts processor, memory, storage, and networking resources ofhardware platform 106 into one or more virtual machines (“VMs”) 132 that run concurrently onhost computer 102.VMs 132 run on top of the virtualization layer, referred to herein as ahypervisor 130, which enables sharing of the hardware resources byVMs 132. One example ofhypervisor 130 that may be used in an embodiment described herein is a VMware ESXi™ hypervisor provided as part of the VMware vSphere® solution made commercially available from VMware, Inc. of Palo Alto, Calif. (although it should be recognized that any other virtualization technologies, including Xen® and Microsoft Hyper-V® virtualization technologies may be utilized consistent with the teachings herein). - Each
VM 132 supported byhypervisor 130 includes guest software (also referred to as guest code) that runs on the virtualized resources supported byhardware platform 106. In the example shown, the guest software of eachVM 132 includes aguest OS 134 and one or more applications (apps) 136.Guest OS 134 can be any commodity operating system known in the art, such as such as Linux®, Microsoft Windows®, Mac OS®, or the like. -
Hypervisor 130 includes aboot loader 138, akernel 140, one ormore user programs 142, and virtual machine monitors (VMMs) 144. In alternative embodiments,hypervisor 130 can include any number of components and the functionality implemented byboot loader 138,kernel 140, user program(s) 142, andVMMs 144 may be distributed in any technically feasible manner between the hypervisor components. -
Kernel 140 provides operating system functionality (e.g., process creation and control, file system, process threads, etc.), as well as CPU scheduling and memory scheduling across guest software inVMs 132,VMMs 144, and user program(s) 142.VMMs 144 implement the virtual system support needed to coordinate operations betweenhypervisor 130 andVMs 132. EachVMM 144 manages a corresponding virtual hardware platform that includes emulated hardware, such as virtual CPUs (vCPUs) and guest physical memory. Each virtual hardware platform supports the installation of guest software in acorresponding VM 132. - Each
user program 142 is a “native” process that runs onkernel 140. User program(s) 142 execute in an environment provided bykernel 140, but outside of environments provided forVMs 132. For example, in some embodiments, eachVMM 140 is paired with a corresponding process, known as a VMX process, which executes as auser program 142. In some embodiments, each VMM may be created by, entered into, and exited via the paired VMX application. In another embodiment, a direct console user interface (DCUI) process, which can be used to configurehypervisor 130, executes as auser program 142. -
Boot loader 138 includes code that is executed upon power on or reset ofhardware platform 106.Boot loader 138 can include one or more stages thatsetup CPU 108 andload kernel 140 intosystem memory 110. In particular,boot loader 138 can configureMMU 116 to manage memory, as described further herein. Alternatively, some or all of the functions implemented byboot loader 138 to configureMMU 116 can be performed bykernel 140 afterboot loader 138 loads and executeskernel 140. - In an embodiment,
boot loader 138 configuresCPU 108 to implement the hypervisor, supervisor, and user PLs.Boot loader 138 configureskernel 140 to execute at the hypervisor PL (e.g., EL2).Kernel 140 configuresVMMs 144 to execute at the hypervisor PL.VMMs 144 configureguest OS 134 in eachVM 132 to execute at the supervisor PL (e.g., EL1).Guest OS 134 in eachVM 132 configures applications 136 to execute at the user PL (e.g., EL0).Kernel 140 configures user program(s) 142 to execute at the user PL (e.g., EL0). Thus, outside of the context of a VM,virtualized computing system 100 includeskernel 140 executing at the hypervisor PL (e.g., EL2) and user program(s) 142 executing at the user PL (e.g., EL0). -
Kernel 140 includes kernel code and data (kernel code/data 126) that are stored inpages 120 ofsystem memory 110. Likewise, user program(s) 142 include user code and data (user code/data 128) that are stored inpages 120 ofsystem memory 110. Kernel code/data 126 is generally referred to as “kernel software,” and user code/data 128 is generally referred to as “user software.” Guest software executing in VMs 132 (e.g., guest OS 132),VMMs 144,boot loader 138, and/orkernel 140 can generate page tables 122, which are also stored inpages 120. In particular, page tables 122 include shared page tables 124. As discussed further below,kernel 140 can create and manipulate shared page tables 124 that are shared betweenkernel 140 and user program(s) 142. - As discussed above,
MMU 116 uses two different translation schemes for code executing at the hypervisor PL versus code executing at the user PL. At the hypervisor PL,MMU 116 implements a single-stage translation scheme that maps VAs to PAs and requires page table descriptors having a first format (e.g.,page table descriptor 300 with reserved bit(s) 308). At the user PL,MMU 116 implements a single-stage or two-stage translation scheme that maps VAs to PAs. Stage one of the supervisor/user PL translation scheme requires page table descriptors similar to those of stage one of the hypervisor PL translation scheme, but without reserving reserved bit(s) 308. Stage two of the supervisor/user PL translation scheme requires a different format for page table descriptors that is incompatible with stage one address translations. - In one scenario, there are three types of page table hierarchies: a first type page table hierarchy for stage one of the hypervisor PL translation scheme (e.g., stage one of the EL2 translation scheme); a second type of page table hierarchy for stage one of the supervisor/user PL translation scheme (e.g., stage one of the EL1/EL0 translation scheme); and a third type of page table hierarchy for stage two of the supervisor/user PL translation scheme (e.g., stage 2 of the EL1/EL0 translation scheme). Alternatively, stage one of the EL1/EL0 translation scheme can be disabled (by disabling EL1), which forces EL0 translations to be done using stage 2 of the EL1/EL0 translation scheme. Such scenarios are inefficient. Since user program(s) 142 do not execute within the context of a VM there is no concept of guest physical memory. Thus, there is no need for the second stage of the supervisor/user PL translation scheme.
- In a third scenario, the second stage of the supervisor/user PL translation scheme is disabled. A first type of page table hierarchy is established for stage one of the hypervisor PL translation scheme and a second type of page table hierarchy is established for stage one of the supervisor/user PL translation scheme. Each stage-one translation scheme maps VAs to PAs and uses page table descriptors having a similar format (with the exception of reserved bit(s) 308). However, in this scenario,
kernel 140 must replicate the mappings associated with user/code data 128 across both page table hierarchies for the hypervisor PL (e.g., EL2) and the supervisor/user PL (e.g., EL1/EL0). Otherwise,kernel 140 would not be able to access user code/data 128. The duplicate mappings still result in an inefficient memory management scheme. - The third scenario described above can be modified to share a portion of the supervisor/user PL page table hierarchy with the hypervisor PL page table hierarchy. In such a modified scenario, hypervisor PL page
table base register 208 and supervisor/user PL page table base register 210 each point to different base page tables. The base page table for the supervisor/user PL page table hierarchy includes only descriptors for mappings to user code/data 128. The base page table for the hypervisor PL page table hierarchy includes descriptors for mappings to kernel code/data 126, as well as replicated descriptors for mappings to user code/data 128. The replicated user code/data descriptors point to the supervisor/user PL page table hierarchy. However, in this modified scenario,kernel 140 must ensure that none of the mappings to kernel code/data 126 make it to the base page table of the supervisor/user PL page table hierarchy. Otherwise, user program(s) 142 would be able to access kernel code/data 126. Thus, extra code must be written and maintained forkernel 140 in order to perform this software-based management of the page tables. -
FIG. 4 is a flow diagram depicting amethod 400 of memory management in a virtualized computing system according to an embodiment.Method 400 may be performed byhypervisor 130 executing invirtualized computing system 100 ofFIG. 1 . Steps inmethod 400 may be performed bykernel 140.Method 400 begins atstep 402, wherehypervisor 130 stores kernel code/data 126 inpages 120 of a kernel address space and user code/data 128 inpages 120 of a user address space. Additional steps ofmethod 400 are described further below. An example memory map is now described with respect toFIG. 5 . -
FIG. 5 is a block diagram depicting amemory map 500 according to an embodiment.Memory map 500 includes akernel address space 502.Kernel address space 502 ranges from a minimum VA address to a maximum VA address in a VA space defined by a kernel VA width. For example, the kernel VA width can be 48 bits, the minimum VA address can be 0x0000_0000_0000_0000, and the maximum address can be 0x0000_FFFF_FFFF_FFFF. Kernel VA width can be other values resulting in a smaller or larger VA address range.Kernel address space 502 includes a user address space 504. User address space 504 ranges from the minimum VA address to a maximum user VA address in a VA address space defined by a user VA width. For example, the user VA width can be 46 bits and the maximum user VA address can be 0x0000_3FFF_FFFF_FFFF. The user VA width can be other values resulting in a smaller or larger user VA address range as constrained by the kernel VA address range. Kernel code/data 126 is stored withinkernel address space 502, but outside of user address space 504. In the examples above, kernel code/data 126 can be stored starting at 0x0000_4000_0000_0000. User code/data 128 is stored within user address space 504. In the example above, user code/data 128 is stored below at or below 0x0000_3FFF_0000_0000. - Returning to
FIG. 4 , atstep 404,hypervisor 130 generates a shared page table hierarchy for kernel and user software. For purposes of clarity by example,method 400 is described in terms of a single shared page table hierarchy. However, it should be understood thatkernel 140 can generate multiple shared page table hierarchies corresponding to multiple user programs 142 (e.g., one shared hierarchy per user program 142). The shared page table includes both mappings to kernel code/data 126 and mappings to user code/data 128. In an embodiment, the shared page table hierarchy includes descriptors having a format dictated by the hypervisor PL translation scheme (e.g., thestage 1 EL2 translation scheme for ARM®v8) (step 406). Additional steps ofmethod 400 are described further below. An example shared page table hierarchy is now described with respect toFIG. 6 . -
FIG. 6 is a block diagram depicting a sharedpage table hierarchy 600 according to an embodiment. Sharedpage table hierarchy 600 includes a base page table 602. Base page table 602 includes descriptors for VAs between the minimum VA address and the maximum VA address ofkernel address space 502. The descriptors are in the format dictated by the hypervisor PL translation scheme (e.g., thedescriptor 300 having reserved bits 308). The descriptors in base page table 602 providekernel mappings 604 anduser mappings 606.Kernel mappings 604 correspond to VAs between the maximum user VA address (exclusive) and the maximum VA address (inclusive).User mappings 606 correspond to VAs between the minimum VA address (inclusive) and the maximum user VA address (inclusive).Kernel mappings 604 point to additional page tables in additional kernel page table level(s) 608.User mappings 606 point to additional page tables in additional user page table level(s) 610. The page tables in additional kernel page table level(s) 608 include mappings that result in address translations to kernel code/data 126. The page tables in additional user page table level(s) 610 include mappings that result in address translations to user code/data 128. - Returning to
FIG. 4 , atstep 408,hypervisor 130 configuresCPU 108 to implement a hypervisor PL translation scheme, which uses the kernel VA width, for code executing at the hypervisor PL.Hypervisor 130 performs such configuration by manipulating one ormore registers 114, e.g.,hypervisor PL stage 1 translation enable 202 and hypervisorPL VA width 212. For example,hypervisor 130 can configureCPU 108 to implementstage 1 of the EL2 translation scheme with a VA width of 48 bits for code executing at EL2 (e.g., kernel 140). - At
step 410,hypervisor 130 configuresCPU 108 to implement a supervisor/user translation scheme, which uses a second VA width that is less than the first VA width, for code executing at the supervisor and user PLs.Hypervisor 130 performs such configuration by manipulating one ormore registers 114, e.g., supervisor/user PL stage 1 translation enable 204 and supervisor/userPL VA width 214. Atstep 412,hypervisor 130 disables second stage address translation (e.g., by manipulating supervisor/user PL stage 2 translation enable 206). For example,hypervisor 130 can configureCPU 108 to implementstage 1 of the EL1/EL0 translation scheme with a VA width of 46 bits for code executing at EL0 (e.g., user programs(s) 142).Hypervisor 130 can configureCPU 108 to disable stage 2 of the EL1/EL0 translation scheme. - At
step 414,hypervisor 130 configuresCPU 108 to use the shared page table hierarchy for each of the hypervisor PL and supervisor/user PL translation schemes.Hypervisor 130 performs such configuration by manipulating one ormore registers 114, e.g., hypervisor PLpage table base 208 and supervisor/user PL page table base 210 (step 416). Each page table base register stores an address of the base page table of the shared page table hierarchy (e.g., base page table 602). - At
step 418,hypervisor 130 executeskernel 140 at the hypervisor PL (e.g., EL2) and user program(s) 142 at the user PL (e.g., EL0). In an embodiment, outside of the context ofVMs 132,hypervisor 130 does not execute code at the supervisor PL (e.g., EL1). In this manner, user program(s) 142 access shared page tables 124 using the user VA address width andkernel 140 accesses shared page tables 124 using the kernel VA address width. If user program(s) 142 attempt to translate a VA mapped to kernel code/data 126,MMU 116 issues a fault, since kernel code/data 126 is stored at VAs outside of the user VA address range.Kernel 140 can translate VAs mapped to the entire VA address space (e.g., to access both kernel code/data 126 and user code/data 128). - Accordingly,
hypervisor 130 efficiently manages shared page tables for bothkernel 140 and user program(s) 142 using hardware features ofCPU 108.Kernel 140 does not require extra code to ensure that kernel mappings are not accessible by user program(s) 142. Rather,MMU 116 will issue a fault if user program(s) 142 attempt to translate a VA that is mapped to kernel code/data 126. The only requirement is thathypervisor 130 stores kernel code/data 126 outside of the user VA address space. Note thathypervisor 130 can generate any number of shared page table hierarchies for any number of user program(s) 142.Hypervisor 130 can manipulate base page table registers ofCPU 108 for context switches betweenuser programs 142 to refer to the different shared page tables. For each context switch, the base page table registers refer to the same base page table in a given shared page table hierarchy.Hypervisor 130 does not need to maintain separate base page tables for kernel and user programs. - The various embodiments described herein may employ various computer-implemented operations involving data stored in computer systems. For example, these operations may require physical manipulation of physical quantities—usually, though not necessarily, these quantities may take the form of electrical or magnetic signals, where they or representations of them are capable of being stored, transferred, combined, compared, or otherwise manipulated. Further, such manipulations are often referred to in terms, such as producing, identifying, determining, or comparing. Any operations described herein that form part of one or more embodiments of the invention may be useful machine operations. In addition, one or more embodiments of the invention also relate to a device or an apparatus for performing these operations. The apparatus may be specially constructed for specific required purposes, or it may be a general purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general purpose machines may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.
- The various embodiments described herein may be practiced with other computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like.
- One or more embodiments of the present invention may be implemented as one or more computer programs or as one or more computer program modules embodied in one or more computer readable media. The term computer readable medium refers to any data storage device that can store data which can thereafter be input to a computer system—computer readable media may be based on any existing or subsequently developed technology for embodying computer programs in a manner that enables them to be read by a computer. Examples of a computer readable medium include a hard drive, network attached storage (NAS), read-only memory, random-access memory (e.g., a flash memory device), a CD (Compact Discs)—CD-ROM, a CD-R, or a CD-RW, a DVD (Digital Versatile Disc), a magnetic tape, and other optical and non-optical data storage devices. The computer readable medium can also be distributed over a network coupled computer system so that the computer readable code is stored and executed in a distributed fashion.
- Although one or more embodiments of the present invention have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications may be made within the scope of the claims. Accordingly, the described embodiments are to be considered as illustrative and not restrictive, and the scope of the claims is not to be limited to details given herein, but may be modified within the scope and equivalents of the claims. In the claims, elements and/or steps do not imply any particular order of operation, unless explicitly stated in the claims.
- Virtualization systems in accordance with the various embodiments may be implemented as hosted embodiments, non-hosted embodiments or as embodiments that tend to blur distinctions between the two, are all envisioned. Furthermore, various virtualization operations may be wholly or partially implemented in hardware. For example, a hardware implementation may employ a look-up table for modification of storage access requests to secure non-disk data.
- Certain embodiments as described above involve a hardware abstraction layer on top of a host computer. The hardware abstraction layer allows multiple contexts to share the hardware resource. In one embodiment, these contexts are isolated from each other, each having at least a user application running therein. The hardware abstraction layer thus provides benefits of resource isolation and allocation among the contexts. In the foregoing embodiments, virtual machines are used as an example for the contexts and hypervisors as an example for the hardware abstraction layer. As described above, each virtual machine includes a guest operating system in which at least one application runs. It should be noted that these embodiments may also apply to other examples of contexts, such as containers not including a guest operating system, referred to herein as “OS-less containers” (see, e.g., www.docker.com). OS-less containers implement operating system-level virtualization, wherein an abstraction layer is provided on top of the kernel of an operating system on a host computer. The abstraction layer supports multiple OS-less containers each including an application and its dependencies. Each OS-less container runs as an isolated process in userspace on the host operating system and shares the kernel with other containers. The OS-less container relies on the kernel's functionality to make use of resource isolation (CPU, memory, block I/O, network, etc.) and separate namespaces and to completely isolate the application's view of the operating environments. By using OS-less containers, resources can be isolated, services restricted, and processes provisioned to have a private view of the operating system with their own process ID space, file system structure, and network interfaces. Multiple containers can share the same kernel, but each container can be constrained to only use a defined amount of resources such as CPU, memory and I/O. The term “virtualized computing instance” as used herein is meant to encompass both VMs and OS-less containers.
- Many variations, modifications, additions, and improvements are possible, regardless the degree of virtualization. The virtualization software can therefore include components of a host, console, or guest operating system that performs virtualization functions. Plural instances may be provided for components, operations or structures described herein as a single instance. Boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of the invention(s). In general, structures and functionality presented as separate components in exemplary configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements may fall within the scope of the appended claim(s).
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/383,572 US10002084B1 (en) | 2016-12-19 | 2016-12-19 | Memory management in virtualized computing systems having processors with more than two hierarchical privilege levels |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/383,572 US10002084B1 (en) | 2016-12-19 | 2016-12-19 | Memory management in virtualized computing systems having processors with more than two hierarchical privilege levels |
Publications (2)
Publication Number | Publication Date |
---|---|
US10002084B1 US10002084B1 (en) | 2018-06-19 |
US20180173646A1 true US20180173646A1 (en) | 2018-06-21 |
Family
ID=62554478
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/383,572 Active US10002084B1 (en) | 2016-12-19 | 2016-12-19 | Memory management in virtualized computing systems having processors with more than two hierarchical privilege levels |
Country Status (1)
Country | Link |
---|---|
US (1) | US10002084B1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220086226A1 (en) * | 2021-01-15 | 2022-03-17 | Intel Corporation | Virtual device portability |
Families Citing this family (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10162655B2 (en) * | 2014-06-23 | 2018-12-25 | Vmware, Inc. | Hypervisor context switching using TLB tags in processors having more than two hierarchical privilege levels |
US10255090B2 (en) | 2014-06-23 | 2019-04-09 | Vmware, Inc. | Hypervisor context switching using a redirection exception vector in processors having more than two hierarchical privilege levels |
US11119939B2 (en) * | 2017-08-21 | 2021-09-14 | Alibaba Group Holding Limited | Methods and systems for memory management of kernel and user spaces |
US10599835B2 (en) | 2018-02-06 | 2020-03-24 | Vmware, Inc. | 32-bit address space containment to secure processes from speculative rogue cache loads |
JP7056391B2 (en) * | 2018-06-08 | 2022-04-19 | 富士通株式会社 | Control method of arithmetic processing unit, information processing unit and arithmetic processing unit |
WO2020000335A1 (en) * | 2018-06-29 | 2020-01-02 | Intel Corporation | Systems and methods of restricting access to kernel memory |
EP3814913A1 (en) * | 2018-08-08 | 2021-05-05 | Huawei Technologies Co., Ltd. | Apparatus and method for providing one time programmable memory features in a hypervisor of a computing device |
CN110928737B (en) * | 2018-09-19 | 2021-05-18 | 华为技术有限公司 | Method and device for monitoring memory access behavior of sample process |
US10705983B1 (en) | 2019-03-01 | 2020-07-07 | International Business Machines Corporation | Transparent conversion of common virtual storage |
US11144472B2 (en) | 2019-03-27 | 2021-10-12 | Intel Corporation | Memory management apparatus and method for managing different page tables for different privilege levels |
KR20210044086A (en) * | 2019-10-14 | 2021-04-22 | 에스케이하이닉스 주식회사 | Controller and data storage system having the same |
US10997086B1 (en) | 2020-03-03 | 2021-05-04 | Intel Corporation | Systems and methods in a graphics environment for providing shared virtual memory addressing support for a host system |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7330942B2 (en) * | 2004-07-31 | 2008-02-12 | Hewlett-Packard Development Company, L.P. | Method for efficient virtualization of physical memory in a virtual-machine monitor |
US9720717B2 (en) * | 2013-03-14 | 2017-08-01 | Sandisk Technologies Llc | Virtualization support for storage devices |
US9530000B2 (en) * | 2013-06-14 | 2016-12-27 | Microsoft Technology Licensing, Llc | Secure privilege level execution and access protection |
-
2016
- 2016-12-19 US US15/383,572 patent/US10002084B1/en active Active
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220086226A1 (en) * | 2021-01-15 | 2022-03-17 | Intel Corporation | Virtual device portability |
Also Published As
Publication number | Publication date |
---|---|
US10002084B1 (en) | 2018-06-19 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10002084B1 (en) | Memory management in virtualized computing systems having processors with more than two hierarchical privilege levels | |
US11995462B2 (en) | Techniques for virtual machine transfer and resource management | |
US7467381B2 (en) | Resource partitioning and direct access utilizing hardware support for virtualization | |
US10768962B2 (en) | Emulating mode-based execute control for memory pages in virtualized computing systems | |
US8127098B1 (en) | Virtualization of real mode execution | |
TWI471727B (en) | Method and apparatus for caching of page translations for virtual machines | |
US11042485B2 (en) | Implementing firmware runtime services in a computer system | |
US20090228882A1 (en) | Method and apparatus for supporting heterogeneous virtualization | |
US10620985B2 (en) | Transparent code patching using a hypervisor | |
US10387184B2 (en) | Address based host page table selection | |
US10162657B2 (en) | Device and method for address translation setting in nested virtualization environment | |
US10489185B2 (en) | Hypervisor-assisted approach for locating operating system data structures based on attribute matching | |
US10642751B2 (en) | Hardware-assisted guest address space scanning in a virtualized computing system | |
US20180267818A1 (en) | Hypervisor-assisted approach for locating operating system data structures based on notification data | |
US11995459B2 (en) | Memory copy during virtual machine migration in a virtualized computing system | |
US10185664B1 (en) | Method for switching address spaces via an intermediate address space | |
US20210382747A1 (en) | Efficient userspace driver isolation by shallow virtual machines | |
US10754796B2 (en) | Efficient user space driver isolation by CPU page table switching | |
WO2017131914A1 (en) | Sharing a guest physical address space among virtualized contexts | |
US11573904B2 (en) | Transparent self-replicating page tables in computing systems | |
US20140208034A1 (en) | System And Method for Efficient Paravirtualized OS Process Switching | |
US11748136B2 (en) | Event notification support for nested virtual machines | |
US11550609B2 (en) | Unified hypercall interface across processors in virtualized computing systems | |
US11169838B2 (en) | Hypercall implementation in a virtualized computer system | |
US11210222B2 (en) | Non-unified cache coherency maintenance for virtual machines |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: VMWARE, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WARKENTIN, ANDREI;LAPLACE, CYPRIEN;LI, YE;REEL/FRAME:040672/0452 Effective date: 20161219 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
AS | Assignment |
Owner name: VMWARE LLC, CALIFORNIA Free format text: CHANGE OF NAME;ASSIGNOR:VMWARE, INC.;REEL/FRAME:067102/0395 Effective date: 20231121 |