CN116745752A

CN116745752A - Migrating memory pages accessible by an input-output device

Info

Publication number: CN116745752A
Application number: CN202180086751.8A
Authority: CN
Inventors: 菲利普·恩杰; 尼蓬·拉瓦尔
Original assignee: ATI Technologies ULC
Current assignee: ATI Technologies ULC
Priority date: 2020-12-29
Filing date: 2021-12-09
Publication date: 2023-09-12
Also published as: US11726693B2; KR20230123985A; WO2022144646A1; JP2024503259A; US20220206700A1; EP4272081A1

Abstract

An electronic device includes a memory, an input-output memory management unit (IOMMU), a processor executing a software entity, and a page migration engine. The software entity and the page migration engine perform operations for preparing to migrate a memory page in the memory that is accessible by the at least one IO device, the software entity and the page migration engine setting migration status information in a page table entry for the memory page based on the operations being performed. When the operation for preparing to migrate the memory page is completed, the page migration engine migrates the memory page in the memory. The IOMMU uses the migration status information in the page table entry to control one or more operations of the IOMMU.

Description

Migrating memory pages accessible by an input-output device

Background

Prior Art

Some electronic devices include processors (e.g., central processing units, etc.) that perform computations, control, memory access, and other operations, and memories in which data is stored (where "data" is used as a generic term for information such as program code instructions, input or result data from a computing operation, constant values, etc.). Many such electronic devices also include input-output (IO) devices, such as network interface devices, disk controllers, and the like. The IO devices may interact with processors and memory for performing various operations. For example, the network interface device may store data received via the network in memory and then signal the processor that the data stored in memory is awaiting processing.

In some cases, data stored in memory in an electronic device may be stored in the memory in a manner that makes data access inefficient. For example, memory may be implemented using a combination of two different types of memory, one of which is a faster but more expensive type of memory and the other of which is a slower but less expensive type of memory. Inefficiencies may occur in slower types of memory when frequently accessed, high priority, and/or time critical data is stored in such memory, or vice versa. As another example, inefficiency may occur when data is stored in a remote portion of memory, such as a portion of memory in a remote socket of a non-uniform memory access (NUMA) electronic device.

Because inefficiency in accessing data in memory can affect the overall performance of an electronic device, some electronic devices include a migration mechanism for dynamically repositioning or "migrating" the data in memory so that the data is stored in a more appropriate location in memory. For example, for purposes of memory management and migration, data in memory may be organized into contiguous blocks of data or "memory pages" of a specified size (e.g., 4KiB, 2MiB, etc.). The migration mechanism may monitor access of the memory pages and selectively migrate the memory pages based on a specified pattern or characteristic of access of the memory pages. For example, in an electronic device having two different types of memory as described above, a migration mechanism may migrate a memory page from a lower speed type of memory to a higher speed type of memory when the memory page stored in the lower speed type of memory is accessed frequently enough. In some electronic devices, the migration mechanism automatically and proactively migrates memory pages—and thus may migrate memory pages without transferring information about the migration to each of the access entities (e.g., processors, IO devices, etc.) that are capable of accessing the memory pages. In other words, the migration mechanism migrates the memory pages "behind the scenes" without requiring each of the access entities to be directly aware of the migration, authorized to approve the migration, and/or the participants of the migration.

In some electronic devices, some or all of the IO devices do not provide support for migrating memory pages. For example, an IO device may be an older or simpler IO device that does not include software and/or hardware to support handling errors (e.g., page faults, etc.), which may occur when a memory page is migrated and found to be temporarily unavailable. If a memory page that such an IO device is capable of accessing were to be migrated and the IO device attempted to access the memory page or information associated with the memory page (e.g., address translation information, etc.) during the migration, the IO device and/or other functional blocks and devices (e.g., processors, etc.) in the electronic device may crash, experience errors, or operate in an unexpected or unexpected manner.

Drawings

Fig. 1 presents a block diagram illustrating an electronic device, according to some embodiments.

Fig. 2 presents a block diagram illustrating an operating system and functional blocks and devices in an electronic device, in accordance with some embodiments.

Fig. 3 presents a block diagram illustrating a hypervisor in an electronic device and functional blocks and devices, in accordance with some embodiments.

FIG. 4 presents a block diagram illustrating a page table in accordance with some embodiments.

FIG. 5 presents a block diagram illustrating metadata in a page table entry, in accordance with some embodiments.

Fig. 6 presents a state diagram illustrating migration states, according to some embodiments.

FIG. 7 presents a flowchart illustrating a process for migrating memory pages, in accordance with some embodiments.

Like reference numerals refer to like elements throughout the drawings and description.

Detailed Description

The following description is presented to enable one of ordinary skill in the art to make and use the described embodiments, and is provided in the context of a particular application and its requirements. Various modifications to the described embodiments will be readily apparent to those skilled in the art, and the generic principles described herein may be applied to other embodiments and applications. Thus, the described embodiments are not limited to the embodiments shown, but are to be accorded the widest scope consistent with the principles and features described herein.

Terminology

In the following description, various terms are used to describe embodiments. The following is a simplified general description of one of these terms. It is noted that the term may have additional aspects that are important, and for the sake of clarity and brevity, these aspects are not recited herein, and thus the description is not intended to limit the term.

Functional block: functional blocks are meant to be a group of interrelated circuits, such as integrated circuit circuits, discrete circuits, etc. The circuits are "interrelated" in that circuit elements in the circuits share at least one attribute. For example, circuitry may be included in, fabricated on, or otherwise coupled to a particular integrated circuit chip, substrate, circuit board, or portion thereof, may involve the performance of specified operations (e.g., computing operations, control operations, memory operations, etc.), may be controlled by a common control element and/or a common clock, and so forth. The circuitry in the functional block may have any number of circuit elements, from a single circuit element (e.g., a single integrated circuit logic gate or discrete circuit element) to millions or billions of circuit elements (e.g., integrated circuit memory). In some implementations, the functional blocks use circuitry that performs the operations without executing program code to perform the operations "in hardware.

Data: data is a general term that indicates information that may be stored in memory and/or used in computing, controlling, and/or other operations. The data includes information such as actual data (e.g., results of computing or control operations, outputs of processing circuitry, inputs to computing or control operations, variable values, sensor values, etc.), files, program code instructions, control values, and/or other information. The data may be organized into logical blocks (e.g., 4KiB or 2MiB memory pages, etc.) that include one or more of the types of information described above for operations (such as loading data into memory, migrating, etc.).

Accessing: when applied to interactions with data stored in memory (e.g., main memory, cache memory, etc.), access or "memory access" indicates all forms of interactions that may be performed with, on, using, and/or with the data and corresponding metadata and/or control values. For example, accessing may include reading or loading data from memory, writing or storing data to memory, invalidating or deleting data in memory, reading or writing metadata of data in memory, changing a state of data in memory, coherency state or permissions, and the like. This data is "accessible" to a given functional block or device when a particular type of memory access is allowed/granted to the given functional block or device.

Virtual memory

In the described embodiments, the electronic device uses virtual memory technology to handle data access by software entities (e.g., applications, operating systems, device drivers, virtual machines, etc.) executing in the electronic device or by input-output (IO) devices (e.g., network interface devices, peripheral component interface express (PCIe) bus devices, disk controllers, etc.). Generally, when data is initially accessed by a software entity or IO device, a memory block or page comprising the given size of data (e.g., 4KiB, 2MiB, etc.) is copied from mass storage (e.g., disk drive or nonvolatile semiconductor memory) to an available physical location in memory (e.g., main memory) in the electronic device, or a memory page is newly created in memory (e.g., for storing results of calculations or other operations, etc.). To avoid the need for the software entity and the IO device to track the physical location of the page in memory, the electronic device tracks the physical location of the page for the software entity or IO device. Instead of accessing memory using addresses based on physical locations (or physical addresses) of data in memory, the software entities and IO devices access memory using "virtual" addresses in a virtual address space, where a virtual address space is a local address space dedicated to the corresponding software entity or IO device. From the perspective of the software entity or IO device, the virtual address indicates the actual physical location where the data is stored in memory, and the virtual address is used by the software entity and IO device accordingly for memory access. However, the virtual address may not map directly to a physical address of a physical location in a page where data is stored in memory. As part of tracking the physical location of the page, the electronic device translates virtual addresses used by the software entity and the IO device in the memory access request to physical addresses where the data entity is located. The electronic device then performs memory accesses for the software entity and the IO device using the physical address.

To achieve the virtual address to physical address translation described above, the electronic device includes a page table. A page table is a record stored in the memory of an electronic device that includes an entry or "page table entry" having virtual address to physical address translation information as well as other information for a page of data stored in the memory. In other words, the page table includes a mapping of virtual addresses to corresponding physical addresses for each data page present in memory. Upon receiving a request from a software entity or IO device to access memory at a given virtual address, the electronic device retrieves corresponding physical address information from the page table by performing a page table walk during which the page table is searched for page table entries providing physical addresses associated with the virtual addresses. For example, upon receiving a request from an operating system to access memory at a given virtual address, a Memory Management Unit (MMU) in a Central Processing Unit (CPU) core in the electronic device may perform a page table walk to obtain corresponding physical address information from the page table and may use the physical address information to process the memory access. As another example, upon receiving a request from an IO device to access memory at a given virtual address, an input-output memory management unit (IOMMU) in the electronic device may perform a page table walk to obtain corresponding physical address information from the page table and may process the memory access using the physical address information.

Because the page table walk described above is relatively slow, it is desirable to avoid performing the page table walk. Thus, the electronic device includes a Translation Lookaside Buffer (TLB) that is a local cache for storing a limited number of copies of address translation information (i.e., information based on page table entries) acquired during a page table walk. For example, the CPU may include a TLB for locally storing copies of the page table entry-based information for memory accesses by the software entity (or copies of the page table entry itself). For another example, the IOMMU may include a TLB for locally storing copies of the information for memory accesses of the IO devices based on the page table entries (or copies of the page table entries themselves). During operation, the CPU core or IOMMU first attempts to fetch cached page table entries from the corresponding TLB for performing virtual address to physical address translations. When a copy of the corresponding page table entry does not exist in the TLB (i.e., when a "miss" occurs), the CPU core or IOMMU performs a page table walk to obtain the desired page table entry. The CPU core or IOMMU may then cache a copy of the fetched page table entries in the corresponding TLB for subsequent use.

In some embodiments, the IOMMU and IO devices support communication protocols for address translation, such as PCI Express (PCIe) Address Translation Services (ATS). For ATS, input-output (IO) devices may include corresponding local caches or "address translation caches" (ATCs), which are used by each IO device to store page table entry-based copy information (or page table entries themselves). Generally, when an IO device desires a local copy of the page table entry-based information (or, more simply, a copy of the entire page table entry), the IO device requests the page table entry-based information from the IOMMU. The IOMMU checks to ensure that the IO device is permitted to have this information and then provides the information based on the page table entry to the IO device. The IO device then stores the information based on the page table entries in the ATC for subsequent use. Thus, using ATC, the IO device may avoid the need to repeatedly request page table entry based information from the IOMMU.

Migrating memory pages

In the described implementations, memory pages may be migrated within memory. For migration, a memory page is physically moved within memory such that the page is relocated from a first location in memory to a second location in memory. For example, if the memory page is an 8KiB memory page and is initially located at addresses A through A+8KiB-1, the page may be migrated to addresses B through B+8KiB-1. In some embodiments, migrating a memory page to relocate the memory page to a location in memory more appropriate for the memory page and/or the memory itself. For example, frequently accessed or higher priority memory pages may be migrated from a first type of memory having a slower access speed to a second type of memory having a faster access speed, or vice versa. As another example, memory pages may be migrated within memory to avoid wear on memory circuitry in the memory. For another example, a memory page may be migrated in memory to locate the memory page closer to other memory pages being accessed. For another example, in some embodiments, memory is configured for non-uniform memory access (NUMA), where each of a plurality of portions of memory are located near a respective processing node and memory pages are migrated between portions of memory in the node.

In some embodiments, the decision to migrate a memory page is made by a software entity and/or a functional block or device in the electronic device based on one or more migration rules (i.e., criteria, thresholds, etc.) that identify conditions under which the memory page is to be migrated. For example, an operating system or hypervisor executed by a processor in an electronic device may determine that a memory page is to be migrated based on the frequency or type of access to the memory page. As another example, a memory management unit function in the processor may determine that a memory page is to be migrated from slower access memory to faster access memory based on a priority identified in a memory access request for the memory page.

Although a processor is able to migrate a memory page in memory, i.e., physically move individual bytes of the memory page in memory, performing the migration using the processor may consume processor resources that are more suitable for other operations (e.g., computing operations, other types of memory accesses, etc.). Thus, in some embodiments, migration of memory pages is offloaded from a processor to a page migration engine function or device (which may include a Direct Memory Access (DMA) function that performs operations for physically moving memory pages within memory). Given the identity of the memory page and the destination location in memory, the page migration engine may execute and may be optimized to perform the migration of the memory page.

SUMMARY

In the described embodiments, the electronic device includes a processor and one or more input-output (IO) devices (where the IO devices may include functional blocks or actual devices such as network interfaces, disk controllers, graphics processing units, etc.). The electronic device further comprises a memory for storing data for use by the processor and/or the IO device. The electronic device further includes at least one page migration engine function block that performs operations for copying, moving, and otherwise processing data stored in the memory. As part of these operations, the page migration engine performs operations associated with migrating memory pages (e.g., 4KiB memory pages), including repositioning memory pages within memory according to migration rules. The electronic device also includes an input-output memory management unit (IOMMU) that manages memory accesses of the IO device. In the described embodiment, the IOMMU performs the following operations: controlling access by an IO device to memory pages that are accessible by the IO device and that are to be migrated or are being migrated, and controlling use of information from page table entries for such memory pages. Operations performed by the IOMMU are intended to ensure that the IO device (and more generally the electronic device) does not experience errors or faults, which may occur if the IO device accesses a memory page being migrated (or encounters a missing memory page) and/or uses incorrect information from a page table entry for such a memory page.

In the described embodiments, the page table includes migration information in the page table entries that is used to help control access to the memory pages being migrated and the corresponding page table entries. The page table entry includes migration status information along with the above-described conversion information and metadata. The migration status information includes an indication of whether preparation for migrating the memory page associated with the page table entry is underway or whether the memory page is being migrated (or not). For example, in some embodiments, the migration state information includes two bits, each possible combination of which identifies a corresponding migration state from a set of migration states. In a first or "non-migrated" migration state identified by 00, the memory page is not migrated. In the second and third migration states identified by 01 and 10, respectively, corresponding preparations for migrating memory pages are ongoing. In a fourth migration state, identified by 11, final preparation of migrating the memory page is ongoing, or the memory page is in the process of being migrated (i.e., actually being relocated in memory).

In some embodiments, the processor executes a software entity that, in combination with the page migration engine, performs at least some of the following operations: memory pages to be migrated are prepared and migration of memory pages is handled. For example, the software entity may be or include a hypervisor, an operating system, an application program, and the like. In an operation performed by a software entity and/or a page migration engine for preparing to migrate a memory page, migration status information of the memory page is set. For example, as functional blocks and devices in an electronic device pass through each of the migration states described above, a software entity or page migration engine may set the migration state information accordingly. Another operation performed by the software entity and/or the page migration engine is to invalidate an existing copy of information from the page table entry for the migrated page stored in a cache in the IOMMU or IO device. A further operation performed by the page migration engine is to, after migrating the memory page, update a page table entry for the memory page to identify a new location in memory for the memory page, and set migration status information in the page table entry to indicate a non-migration status.

In some implementations, the IOMMU enforces access restrictions for memory pages and page table entries based on migration status information in the page table entries. For example, when the migration status information indicates that preparation of a memory page is being migrated or that a memory page is being migrated, the IOMMU may prevent the IOMMU itself and the IO device from storing or using copies of information from page table entries in the local cache. For another example, when the migration information in the page table entry indicates that a memory page is ready to be migrated or is being migrated, the IOMMU may ensure that accesses to the memory page are handled accordingly. That is, the IOMMU may ensure that an existing memory access of a memory page is completed before the memory page is migrated, and lay aside (i.e., suspend, buffer, etc.) the newly encountered memory access of the memory page until after the memory page is migrated. In contrast, when the migration information in the page table entry indicates that the memory page is in a non-migrated migration state (and thus not migrated), the IOMMU may allow normal access of the memory page and page table entry.

In some embodiments, some or all of the IO devices support an address translation service, such as PCI Express (PCIe) Address Translation Service (ATS), which is a standard according to which the IO devices may request information from page table entries from the IOMMU to be cached locally in an entry in the local cache or "Address Translation Cache (ATC)" in the IO devices. As part of the above operations for preventing storing copies of information from page table entries in a cache in an IO device, the IOMMU may refuse to accept/reject ATS requests for information from page table entries associated with memory pages to be migrated or being migrated. For example, in some embodiments, the IOMMU may respond to the ATS request by informing the requesting IO device that the memory page is not present, which causes the IO device to proceed with operations for causing the memory page to be present in the IO page table. For another example, the IOMMU may inform the requesting IO device that the untranslated address should be used to access the memory page-and thus, information from the IO page table entry will not be provided to the IO device.

While the above operations are performed with respect to memory pages that the IO device is able to access and is ready to migrate or being migrated, other types of memory pages may be handled differently during migration. For example, in some embodiments, memory pages that are inaccessible to IO devices but only accessible to a processor may be migrated by: the page is marked as "not present" in the corresponding page table entry and any copies of the page table information for such memory pages in the cache (i.e., the translation look-aside buffer) in the processor are invalidated. In this case, the processor includes a mechanism for handling missing pages (e.g., via page faults, etc.). Furthermore, memory pages and their corresponding page table entries that are not ready for migration or not migrated, even when they are accessible by IO devices, are normally accessible, although in view of memory access ordering rules and the like.

By handling migration of memory pages that are accessible by IO devices, the described embodiments enable such memory pages to migrate more freely within memory, regardless of whether there are IO devices that themselves may incorrectly handle migrating memory pages. Migrating these memory pages may help ensure that the memory pages are in a desired location in memory. This may improve the operation of the memory, which in turn improves the operation of the processor and the IO devices using memory pages in the memory.

Electronic equipment

Fig. 1 presents a block diagram illustrating an electronic device 100, according to some embodiments. As shown in FIG. 1, electronic device 100 includes a processor 102, a memory 104, a mass storage 106, input-output (IO) devices 108-110, an input-output (IO) hub 112, and a page migration engine 114. Processor 102, memory 104, mass storage 106, IO devices 108-110, IO hub 112, and page migration engine 114 are implemented in "hardware," i.e., using corresponding integrated circuits, discrete circuits, and/or devices. For example, in some embodiments, processor 102, memory 104, mass storage 106, IO devices 108-110, IO hub 112, and page migration engine 114 are implemented in integrated circuits on one or more semiconductor chips, in a combination of integrated circuits on one or more semiconductor chips and discrete circuits and/or devices, or in discrete circuits and/or devices. In fig. 1, the electronic device 100 is partially colored to enable various graphic elements to be more easily discerned.

Processor 102 is a functional block in electronic device 100 that performs computations, memory access, control, and/or other operations. The processor 102 includes cores 116-118, where each core includes one or more Central Processing Unit (CPU) cores, graphics Processing Unit (GPU) cores, embedded processors, application Specific Integrated Circuits (ASICs), and/or other computing functional blocks.

Processor 102 includes a cache memory, or "cache," which is a functional block for storing copies of data that may be used by cores 116-118 to perform various operations. As can be seen in FIG. 1, the caches in processor 102 include level one (L1) caches 120-122 (L1 120 and L1 122) in cores 116-118, respectively. Each of the L1 caches 120-122 includes memory circuitry for storing data, and control circuitry for handling access to data stored in the memory circuitry. The processor 102 also includes a shared level two (L2) cache 124 and level three (L3) cache 126, each of which includes memory circuitry for storing data, and control circuitry for handling access to data stored in the memory circuitry.

Processor 102 includes a Platform Security Processor (PSP) 128, which is a functional block in electronic device 100 for performing security-related operations. For example, in some embodiments, the PSP 128 includes a CPU core, an ASIC, and/or a microcontroller. The PSP 128 includes circuitry designed to protect other functional blocks and devices in the processor 102 and/or software entities executed by the processor 102 from specified malicious or errant behaviors. Thus, the PSP 128 may be used to protect the operation of other functional blocks, devices, and/or software entities susceptible to such behavior. In other words, PSP 128 may perform operations associated with enabling a trusted execution environment in electronic device 100. To this end, the PSP 128 may perform operations for encryption/decryption (e.g., key generation, data encryption/decryption, etc.), registration and/or authentication of hardware and/or software entities, access permission verification, etc. In some embodiments, PSP 128 performs at least some of the operations described herein for handling migration of pages accessible to IO devices 108-110. For example, the software entity may communicate a request to migrate a memory page to the PSP 128, which then verifies that movement of the memory page is permitted (i.e., the software entity is allowed to move a particular memory page to a destination location in memory, etc.) before forwarding the request to migrate the memory page to the page migration engine 114 for subsequent processing.

The memory 104 is a functional block in the electronic device 100 for storing data for other functional blocks. For example, in some embodiments, memory 104 is a "main" memory in electronic device 100. The memory 104 includes memory circuitry for storing data, and control circuitry for handling access to the data stored in the memory circuitry.

In some embodiments, the memory 104 includes two or more different types of memory arranged such that different portions of a set of addressable locations in the memory 104 are in each of the types of memory. For example, in some embodiments, half of the addressable locations are in a first type of memory and are therefore implemented using a first type of memory circuit, and half of the addressable locations are in a second type of memory and are therefore implemented using a second type of memory circuit. The use of both types of memory is illustrated in fig. 1 via memory type 130 and memory type 132. For example, memory type 130 may be a fifth generation double data rate synchronous dynamic random access memory (DDR 5 DRAM) or another type of memory, and memory type 132 may be a 3D cross point (3D XPoint) memory or another type of memory. Each type of memory has corresponding memory characteristics. For example, memory characteristics may include some or all of access speed/latency, access bandwidth, data protection/verification/error correction, reliability/mean time to failure, electrical power consumption, heat generation, data addressability (e.g., byte or block addressability), data compatibility/specification, monetary cost, design complexity, and the like.

The mass storage 106 is a functional block and/or device that stores data used by other functional blocks and devices in the electronic device 100. For example, mass storage device 106 may be or include semiconductor memory, a magnetic disk drive, an optical drive, and the like. The mass storage 106 includes circuitry and/or devices that retain stored data even if power to the mass storage 106 is turned off (or otherwise reduced), and thus serves as a non-volatile "long-term" storage for the data. At run-time (i.e., when the electronic device 100 is operating), a copy of the data is retrieved from the mass storage 106 and stored in the volatile memory 104 (and possibly one or more caches) for subsequent use by functional blocks in the electronic device 100. For example, data may be retrieved/read from the mass storage 106 in blocks or pages of a given size (e.g., 4KiB, 2MiB, 1GiB, etc.), as well as from pages stored in the memory 104 as "memory pages. In addition, memory pages created by other functional blocks and devices in the electronic device 100 may be stored in the memory 104 (e.g., for storing computing results, files, configuration values, etc.) and ultimately written out to the mass storage 106. When stored in the memory 104, the memory pages may be accessible by other functional blocks and devices according to access rules and permissions of the memory pages and/or data therein.

Returning to the processor 102, the Memory Management Unit (MMU) 134 is a functional block that handles memory access requests and requests for information from the page table. When data is to be accessed by a functional block in the processor 102, the functional block sends a memory access request to the MMU 134. For example, a software entity (e.g., program, operating system, etc.) executed by the kernel 116 may cause a load/store unit in processing circuitry in the kernel 116 to send memory access requests (e.g., for data loads or stores, etc.) to the MMU 134.MMU 134 then sends the corresponding memory access request to one or more of L2 cache 124, L3 cache 126, and memory 104 to satisfy or resolve the memory access request. For example, if data is to be loaded, the MMU 134 may retrieve the data from the L2 cache 124, the L3 cache 126, or the memory 104 and forward the data to the requesting functional block. This may mean that one or more memory pages including the data are to be loaded from mass storage 106 to memory 104 if the data is not already present in memory 104.

MMU 134 includes a Table Walker (TW) 136, which is a functional block that performs operations related to obtaining address translation information and other information from a page table via page table walks. For example, as described in more detail above, in some embodiments, electronic device 100 uses virtual memory such that software entities (e.g., that are executed by cores 116-118) can access memory using virtual addresses that are translated to physical addresses. Upon receiving a memory access request from a software entity having a virtual address, the table walker 136 performs a page table walk operation to translate the virtual address into a physical address of a page whose data is located in the memory 104. During a page table walk operation, the table walker 136 may also examine page table entries and/or other records to ensure that the functional blocks and/or software entities that are requesting each memory access are permitted to perform such accesses, i.e., are permitted to access memory at physical addresses, etc., and that the page tables and other records for translation have not been tampered with or modified without permission. In some embodiments, the table walker 136 also examines the migration status information in the page table entries as part of the operations described herein for handling the migration of pages.

An example of the page table described above is shown as page table 148 in FIG. 1. Generally, page table 148 is a record having an entry or "page table entry" with virtual address to physical address translation information as well as other information for each page of data currently stored in memory 104. Page table 148 includes in each page table entry corresponding migration status information for handling the migration of memory pages as described herein. Although shown as a single entity in fig. 1, in some embodiments, page table 148 is or includes a plurality of separate page sub-tables that are used in combination to perform virtual address to physical address translations. Further, in some embodiments, page table 148 includes or is included in a page table hierarchy (e.g., host page table, IOMMU page table, etc.) for performing virtual address to physical address translations for virtual machines and IO devices, etc., i.e., for performing multiple translations of guest virtual addresses, host virtual addresses, and host physical addresses. For example, in some embodiments, page table 148 includes a host page table used by processor 102 to perform virtual-to-physical address translations (and other operations) for software entities executing on processor 102, and an IOMMU page table used by IOMMU 140 to perform virtual-to-physical address translations (and other operations) for IO devices 108-110. For clarity, a single page table is used in the examples herein-although multiple page tables, page table hierarchies, and/or individual page tables may be used in some embodiments with corresponding adjustments to the operations described herein.

MMU 134 includes a Translation Lookaside Buffer (TLB) 138, which is a functional block for storing copies of information retrieved from page table entries. The TLB 138 includes memory circuitry for a cache that stores copies of page table entries, or portions thereof, obtained during page table walks by the table walker 136 (or from other sources). To avoid performing a page table walk, the MMU 134 retrieves address translation information and/or other page table entry information from the TLB 138, when possible. However, if address translation information and/or other page table entry information is not present in the TLB 138, the table walker 136 performs a table walk to obtain the information. In some implementations, the specified information in the TLB 138 is invalidated before the corresponding page is migrated, as described herein.

Input-output (IO) devices 108-110 are functional blocks or devices that interact with processor 102 and other functional blocks and devices in electronic device 100. Accordingly, processor 102 and other functional blocks may receive "input" data from IO devices 108-110 or send "output" data to IO devices 108-110. IO devices 108-110 may also interact with functional blocks or devices external to electronic device 100. For example, the input-output devices 108-110 may include network interface devices, disk controllers, devices coupled to corresponding wired or wireless buses or interfaces (e.g., a peripheral component interconnect express (PCIe) bus, universal Serial Bus (USB), wiFi network, etc.), graphics processing units, and so forth. The particular operations performed by each of the IO devices 108-110 depend on the nature of each IO device. For example, if IO device 108 is a disk controller, IO device 108 may retrieve data from a disk (e.g., mass storage 106) and write the data to memory 104, or vice versa. As another example, if IO device 110 is a network interface device, IO device 110 may store data received via a network (not shown) in memory 104, retrieve data from memory 104 to be transmitted to a receiving device over the network, provide data to processor 102, or retrieve data from processor 102, etc.

IO hub 112 is a functional block or device that performs operations for connecting between IO devices 108-110 and other functional blocks in electronic device 100. In other words, IO hub 112 performs operations for routing communications and other data between IO devices 108-110 and functional blocks such as processor 102, memory 104, and the like. The particular communications routed by IO hub 112 and the operations performed to route the communications depend on the nature of IO devices 108-110, but may include memory access, data communications, control and configuration communications, and the like.

IO hub 112 includes an input-output memory management unit (IOMMU) 140, which is a functional block that performs operations for handling memory accesses of IO devices 108-110. When data in memory 104 is to be accessed by an IO device, the IO device transmits a corresponding memory access request. IOMMU 140 intercepts a memory access request from an IO device, processes the request internally to determine whether the memory access request is resumable, and then sends a corresponding request to memory 104 to access the data (assuming the memory access request is resumable).

In the operations performed by IOMMU 140 to process memory access requests from IO devices 108-110, address translation for memory accesses by IO devices 108-110 is performed (i.e., to obtain physical addresses in memory associated with virtual addresses used by IO devices 108-110 in the memory access requests). The IOMMU 140 includes an input-output table walker (iomw) 142, which is a functional block that performs operations related to obtaining address translations from a page table (e.g., page table walks, etc.). The operations performed by the input-output table walker 142 are similar to those performed by the table walker 136, but it is used for memory accesses from the IO devices 108-110 (rather than the processor 102). In some embodiments, in conjunction with performing address translation, input-output table walker 142 also examines metadata and page information (such as migration status information) to determine whether memory accesses from IO devices 108-110 are permitted in memory pages and/or whether memory pages are being migrated, as described herein. In these embodiments, the input-output table walker 142 may block or otherwise process memory accesses to avoid incorrect memory accesses or memory accesses to pages being migrated. In some embodiments, the input-output table walker 142 uses and thus executes a table walker in a page table hierarchy that includes page tables associated with the IOMMU 140, host page tables, and/or nested page tables associated with software entities (e.g., with hypervisors, virtual machines, etc.).

The IOMMU 140 also includes a Translation Lookaside Buffer (TLB) 144, which is a functional block for storing copies of information obtained from page table entries. The TLB 144 includes memory circuitry for a cache that stores copies of page table entries, or portions thereof, acquired during page table walks by the table walker input-output table walker 142 (or from other sources). To avoid performing a page table walk, the IOMMU 140 retrieves address translation information and/or other page table entry information from the TLB 144 when possible. However, if address translation information and/or other page table entry information is not present in the TLB 144, the input-output table walker 142 performs a table walk to obtain the information. In some implementations, the specified information in the TLB 144 is invalidated before the corresponding page is migrated, as described herein.

In some embodiments, some or all of IO devices 108-110 include a local cache for storing copies of information retrieved from page table entries. For example, in some embodiments, some or all of IO devices 108-110 and IOMMU 140 provide support for an Address Translation Service (ATS) according to which IOMMU 140 may provide copies of information obtained from page table entries to IO devices 108-110 for caching therein. An example of such a cache is shown as an Address Translation Cache (ATC) 146 in IO device 108. ATC 146 includes memory circuitry for a cache that stores copies of page table entries, or portions thereof, acquired during page table walks by input-output table walker 142 (or from other sources). To avoid performing page table walks, IO device 108 obtains address translation information and/or other page table entry information from ATC 146 when possible. However, if address translation information and/or other page table entry information is not present in the ATC 146, the input-output table walker 142 performs table walks to obtain that information (or uses cache information from the TLB 144). In some embodiments, the specified information in ATC 146 is invalidated before the corresponding page is migrated, as described herein.

In some embodiments, although shown as a single entity in fig. 1, the IOMMU 140 includes multiple separate IOMMU (e.g., multiple instances of IOMMU functional blocks). For example, in some embodiments, electronic device 100 includes at least two IOMMU functional blocks, each IOMMU functional block controlling memory accesses by a subset of a set of IO devices in electronic device 100. In these embodiments, the IOMMU works together, or by a "master" IOMMU or other functional block, to ensure that the IOMMU is properly using the information in the page table and accessing the memory of the IO device—including the case where there is a migrated memory page as described herein. For example, the IOMMU may broadcast requests and acknowledgements of TLB invalidations, memory access completion or stalls, etc. over the communication links to which all IOMMU are connected to ensure that all IOMMU are synchronized before a given IOMMU performs operations (such as accessing information in page table entries, allowing memory accesses from the IO devices to proceed, etc.).

Page migration engine 114 is a functional block that performs operations associated with accessing data in memory 104. The page migration engine 114 may be used to offload memory access operations from other functional blocks in the electronic device 100. For example, in some embodiments, page migration engine 114 offloads and thus receives and executes memory accesses on behalf of cores 116-118, which enables cores 116-118 to avoid performing memory access operations themselves. In some embodiments, page migration engine 114 includes one or more Direct Memory Access (DMA) functional blocks and other functional blocks that perform operations for moving memory pages within memory 104. In a memory access operation performed by the page migration engine 114, memory pages are "migrated" in memory 104. For migration, upon receiving a request from one of the cores 116-118 (e.g., from an operating system executed by the core 116, etc.) or from another source (such as the PSP 128) to identify a memory page and a new location where the memory page is to be stored in memory, the page migration engine 114 moves the memory page to the new location (i.e., physically moves individual bits/bytes in the memory page to the new location in one or more data movement operations). In conjunction with migrating memory pages, page migration engine 114 performs operations as described herein for handling migration of memory pages accessible to IO devices.

In some embodiments, although shown as a single entity in fig. 1, page migration engine 114 includes multiple separate page migration engines (e.g., multiple instances of a page migration engine functional block). For example, in some embodiments, page migration engine 114 includes at least two page migration engines, each arranged to perform memory accesses on behalf of other functional blocks in electronic device 100, including migrating memory pages as described herein. In these embodiments, page migration tasks/requests are distributed among two or more page migration engines according to a distribution scheme (e.g., round robin, by memory address or location in memory, etc.), and the page migration engines may coordinate accesses to avoid specified memory accesses, etc.

In some embodiments, the communication path is coupled between functional blocks in the electronic device 100, as indicated by the arrowed lines between the functional blocks. The communication path includes one or more buses, wires, leads, and/or other connections, possibly along with controllers, structural elements (e.g., switches, routers, etc.). Communication paths may be used to route commands, data, control signals, and/or other information between functional blocks. For example, in some implementations, an interconnect fabric is coupled between the IO hub 112, the processor 102 (e.g., MMU 134), and the memory 104. It is noted that for clarity, some communication paths are not shown in fig. 1.

Although electronic device 100 is shown in fig. 1 as having a particular number and arrangement of functional blocks and devices, in some embodiments electronic device 100 includes a different number and/or arrangement of functional blocks and devices. For example, in some embodiments, processor 102 includes a different number of cores. As another example, in some embodiments, different numbers and/or arrangements of caches are present in the processor 102. As another example, in some embodiments, IO device 108 does not include an address translation cache, or IO devices 108-110 both include translation caches. Generally, in the described embodiments, the electronic device 100 includes a sufficient number and/or arrangement of functional blocks to perform the operations described herein.

The electronic device 100 and the processor 102 as shown in fig. 1 are simplified for illustration purposes. However, in some embodiments, the electronic device 100 and/or the processor 102 include additional or different elements and mechanisms for performing the operations and other operations described herein. For example, the electronic device 100 and/or the processor 102 may include a power supply function block or device, a human interface function block or device, and the like.

In some embodiments, the functional blocks shown separately in fig. 1 are implemented together. For example, in some embodiments, some or all of IO hub 112 and IO devices 108-110 are incorporated in/with processor 102, such as fabricated on the same integrated circuit chip. In other words, for such embodiments, IO hub 112 and IO devices 108-110 may be integrated with processor 102 (e.g., as a "system on a chip" or in another form). Alternatively, in some embodiments, the functional blocks shown as part of other functional blocks may be separate. For example, in some embodiments, the platform security processor 128 is implemented separately from the processor 102.

The electronic device 100 may be or be included in any electronic device that performs computing operations. For example, the electronic device 100 may be or be included in a desktop computer, a laptop computer, a wearable electronic device, a tablet computer, a smart phone, a server, an artificial intelligence device, virtual or augmented reality equipment, a network device, a toy, an audiovisual device, a household appliance, a controller, a vehicle, and/or the like, and/or combinations thereof.

Operating system

In some embodiments, one or more software entities interact with functional blocks in an electronic device to perform operations for handling migration of memory pages accessible to IO devices in memory. In some of these embodiments, the one or more software entities are or include an operating system. Fig. 2 presents a block diagram illustrating an operating system and functional blocks and devices in an electronic device, in accordance with some embodiments. Note that fig. 2 is simplified and generally illustrates the relationship of the operating system to the electronic device hardware. In some embodiments, some or all of the elements shown in fig. 2 are absent and/or differently arranged.

As can be seen in FIG. 2, an operating system 200 is connected between electronic device hardware 202 (i.e., between the functional blocks and devices of electronic device 100 (such as processor 102) and a set of programs 204-208. For example, operating system 200 may be an operating system such as that from Microsoft (Redmond, WA)From Apple, inc. (Cupertino, CA)Etc., and programs 204-208 may each be productivity applications, scientific computing applications, web browsers, etc. IOMMU 210 is coupled between IO device 212 and electronic device hardware 202 (e.g., memory 104, etc.). In some embodiments, the functional blocks and devices in FIG. 2, i.e., electronic device hardware 202, IOMMU 210, and IO device 212, are similar to the hardware (e.g., processor 102, memory 104, page migration engine 114, etc.), IOMMU 140, and IO devices 108-110, respectively, in electronic device 100 as shown in FIG. 1.

Virtual machine and hypervisor

In some embodiments, one or more software entities interact with functional blocks in an electronic device to perform operations for handling migration of memory pages accessible to IO devices in memory. In some of these embodiments, the one or more software entities are or include a hypervisor. Generally, in some embodiments, a processor in an electronic device executes one or more virtual machines. A virtual machine is a software entity that emulates or otherwise interfaces with a processor and other functional blocks and devices (e.g., memory, IO devices, etc.) in an electronic device in order to provide support for executing software programs. For example, a virtual machine may provide support for running one or more instances of an operating system known as a guest operating system. The guest operating system then provides support for executing other software programs (such as applications, databases, etc.). In some embodiments, the processor also executes a hypervisor, which is a software entity that performs operations for controlling/managing the execution of the virtual machine. For example, the hypervisor may start and initialize virtual machines, assist in controlling access by the virtual machines to functional blocks and devices in the electronic device (e.g., indicating which areas of memory and/or IO devices the virtual machines are permitted to access, etc.), terminate or shut down virtual machines, and so forth. Fig. 3 presents a block diagram illustrating a hypervisor in an electronic device and functional blocks and devices, in accordance with some embodiments. Note that fig. 3 is simplified and generally illustrates the relationship of the hypervisor to the electronic device hardware. In some embodiments, some or all of the elements shown in fig. 3 are absent and/or differently arranged.

As can be seen in fig. 3, there are three Virtual Machines (VMs) 300-304, each executing a respective guest operating system (guest OS) 306-310 and one or more Programs (PRGRMS) 312-316 thereon. Hypervisor 318 interfaces between host operating system 320 and virtual machines 300-304. The host operating system 320 is connected between the electronic device hardware 322 and the hypervisor 318. IOMMU 324 is coupled between IO device 326 and electronic device hardware 322. In some embodiments, the functional blocks and devices in FIG. 3, i.e., electronic device hardware 322, IOMMU 324, and IO devices 326, are similar to the hardware (e.g., processor 102, memory 104, page migration engine 114, etc.), IOMMU 140, and IO devices 108-110, respectively, in electronic device 100 as shown in FIG. 1.

Page table entry

In the described embodiments, functional blocks and devices in the electronic device (e.g., processor 102, IOMMU 140, etc.) use one or more page tables to perform virtual address to physical address translations, as well as other operations. Fig. 4 presents a block diagram illustrating a page table 400, in accordance with some embodiments. As can be seen in fig. 4, the page table 400 includes a plurality of page table entries 402 (one of which is highlighted using a dashed line), each of which may store Virtual Address (VA) to Physical Address (PA) translations 404 and corresponding metadata 406. In some embodiments, page table 148 in FIG. 1 is internally arranged similarly to page table 400.

Generally, when a memory page (i.e., a block, section, or portion of memory of a specified size (such as 4KiB, 2MiB, etc.) is retrieved from mass storage 106 and stored in memory 104, or a memory page is newly created in memory 104, a corresponding page table entry 402 is added to page table 400, the entry having a virtual address to physical address translation 404 and metadata 406. Thus, if a memory page is available in memory 104, page table 400 should include a corresponding page table entry 402. Page table entry 402 is added to page table 400 to enable tracking of the relationship between the physical address of a page in memory and the virtual address used by software entities (e.g., programs, firmware, operating systems, etc.) and devices to access the memory page as described above.

In some embodiments, the virtual address to physical address translation 404 in each page table entry 402 in the page table 400 includes information that may be used to determine the physical address of a corresponding page in memory associated with one or more virtual addresses. In other words, each virtual address to physical address translation 404 includes information that can be used to identify a location of a memory page in memory 104 that is addressed by a software entity and/or device using the corresponding virtual address. For example, in some embodiments, each virtual address to physical address translation 404 includes a first field having some or all of one or more virtual addresses (e.g., N bits of M bits of each virtual address, where N.ltoreq.M) and a second field having some or all of the physical addresses to which the one or more virtual addresses are mapped/matched.

Metadata 406 includes information that associates, characterizes, controls, and/or otherwise relates to corresponding virtual address to physical address translations 404. When translation information for a memory page is added to a page table entry 402 in the page table 400, metadata is retrieved, generated, etc. and added to the page table entry 402. Fig. 5 presents a block diagram illustrating metadata 406 in a page table entry, in accordance with some embodiments. As can be seen in fig. 5, metadata 406 includes validity 500, permissions 502, controls 504, and migration status information (MIG STATE INF) 506. Validity 500 includes one or more values related to a validity of a page table entry 402, a virtual address to physical address translation 404 in the page table entry 402, and/or a corresponding memory page in memory 104. For example, validity 500 can include one or more bits that indicate whether page table entry 402 itself is valid/accessible and/or whether a corresponding page in memory 104 is valid/accessible.

Permissions 502 include one or more values related to access permissions for corresponding pages in memory 104. For example, permissions 502 may include one or more bits indicating whether a corresponding page in memory 104 has read-only or read-write permission settings, whether the page is only accessible with certain permissions (administrator, user, kernel, etc.), and/or other permission information.

Control 504 includes one or more values related to the use of page table entry 402 and/or a corresponding page in memory 104. For example, the control 504 may include a page size indication identification, a dirty indication identification (for when a corresponding page in the memory 104 has been modified and thus is inconsistent with an associated memory block in the mass storage 106), an accessed indication identification (for when a page in the memory 104 has been accessed one or more times), a write-through indication identification (for when modifications to a page in the memory 104 will be immediately reflected in an associated memory block on the mass storage 106), and/or other values.

The migration status information 506 includes one or more values indicating whether preparation of the associated page is being migrated or the associated page is being migrated. For example, in some embodiments, the migration status information includes a plurality of bits sufficient to uniquely identify each possible migration status of the corresponding memory page. For example, if there are six migration states, with a first migration state for pages that are not migrated, a sixth migration state for pages that are actively migrated, and four intermediate migration states for various prepare operations to migrate a memory page, then the migration state information may be stored in three bits. The migration status information 506 may be used by a function block or device accessing the page table entry to decide what operations are permitted (or not permitted) for the memory page and/or the information in the corresponding page table entry 402. For example, in some embodiments, when the migration status indicates that preparation of a memory page is being migrated or that the memory page is being migrated, the access function block or device may determine that a copy of the information from the page table entry is not permitted to be cached in the local cache and/or that some or all of the memory accesses to the memory page are restricted.

In some embodiments, one or both of virtual address to physical address translation 404 and metadata 406 may be modified/changed, updated, etc. after being added to page table 400. For example, as a page is moved from a first location to a second location in memory 104, virtual address to physical address translation 404 in the corresponding page table entry 402 may be updated. As another example, in embodiments in which metadata 406 includes permissions 502 (e.g., read/write permissions), permissions 502 may be updated to indicate read-only, read-write, etc. permissions for corresponding pages in memory 104. For another example, the migration status information 506 may be updated as the function blocks and devices pass through the migration status for migrating memory pages.

As described above, MMU 134, IOMMU 140, or another entity in electronic device 100 may perform a page table walk when attempting to obtain physical address information or metadata specifying a virtual address. During a page table walk, MMU 134, IOMMU 140, etc. searches page table 400 for page table entry 402, if such page table entry 402 exists, with corresponding virtual address-to-physical address translation 404 maintained therein. Upon encountering such a page table entry 402, the MMU 134, IOMMU 140, etc., obtains the physical address or metadata of the specified virtual address from the page table entry 402 (i.e., from the virtual address to physical address translation 404 in the page table entry 402). If the MMU 134, IOMMU 140, etc. cannot find the corresponding page table entry 402, then an error handling operation (e.g., issue and then handle a page fault, etc.) is performed. In the described embodiments, functional blocks and devices (e.g., IOMMU, page migration engine, etc.) in an electronic device perform operations to prevent IO devices from encountering problems of missing or unavailable page table entries or corresponding memory pages due to migration of the corresponding memory pages. For example, in some embodiments, when determining a virtual address to physical address translation for a memory access request, input-output table walker 142 examines migration status information 506 in the corresponding page table entry 402 to determine whether the virtual address to physical address translation 404 is allowed to be pending by IOMMU 140 and/or IO devices 108-110 caches and/or memory accesses due to the current migration status.

Although page table 400 is shown with a particular number of page table entries 402, some embodiments include a different number of page table entries 402 (as indicated by the ellipses in fig. 4). Further, although page table entry 402 and metadata 406 are shown as having particular arrangements and types of information, in some embodiments, one or both of page table entry 402 and metadata 406 include different information. For example, in some embodiments, at least some of the above metadata is stored in a different location than the page table entry 402. For example, in some embodiments, a separate in-memory data structure is used to store migration status information. In general, page table 400 includes enough information to enable functional blocks and devices in an electronic device to store and retrieve virtual address to physical address translation information and metadata as described herein.

Although page table 400 is shown as a single page table, in some embodiments page table 400 is organized differently. For example, in some embodiments, page table 400 is implemented using two or more sub-tables arranged in a hierarchy. In these embodiments, the search of each sub-table directs the search function block or device to the next sub-table in the hierarchy until the final sub-table is reached, where virtual address to physical address translations 404 and metadata 406 are stored in the corresponding page table entry 402.

Migration state

In some embodiments, when preparation for migrating a memory page is made and then the memory page is migrated, functional blocks, devices, and/or software entities in the electronic device (e.g., IOMMU 140, page migration engine 114, processor 102, operating system 200, and/or hypervisor 318) pass through the migration states in a set of migration states. In addition to the initial non-migrated migration states, in each migration state, respective operations for preparing to migrate a memory page and/or for migrating a memory page are performed. Requests for access to information in memory pages and/or corresponding page table entries to be migrated or being migrated are processed by functional blocks and devices in the electronic device (e.g., IOMMU 140, page migration engine 114, etc.) based on the current migration state of the memory pages. Generally, as operations for each migration state are performed, the requests are processed in such a way that the function blocks and devices continue to operate correctly. For example, IO devices (e.g., IO devices 108-110) and/or processors (e.g., processor 102) are protected from errors and errors that may be caused by migrating memory pages and/or access to corresponding page table entries.

Fig. 6 presents a state diagram illustrating migration states, according to some embodiments. Generally, the migration states shown in fig. 6 are examples of migration states through which functional blocks, devices, and/or software entities in an electronic device (e.g., electronic device 100) may pass when preparing to migrate a memory page and subsequently migrate the memory page. Although a particular number and arrangement of migration states and transitions between migration states are shown in fig. 6, in some embodiments, different numbers and/or arrangements of migration states may be used and/or different transitions between migration states may be used.

In fig. 6, transitions between migration states are shown using arrowed lines between migration states. The first set of transitions between the migration states are shown using thick lines, while the other transitions between the migration states are shown using thin lines. In the following description, a first set of transitions between migration states is described, and further transitions between migration states are described subsequently. It should be noted, however, that the different line thicknesses are used only to clearly describe the migration states and transitions between migration states.

As can be seen in fig. 6, the migration states include Migration States (MSs) 600-606. In the migration state 600 (which is a "non-migration" migration state), the memory pages are not migrated and will therefore remain in the current location in memory. Migration state 600 is thus a migration state in which memory pages are maintained until they are to be migrated (and more generally, memory pages are maintained until they are to be migrated). In the migration state 600, migration state information (e.g., migration state information 506) in a page table entry for a memory page is set to represent or identify the memory page as a bit sequence, value, etc. in the migration state 600. For example, assuming that the migration status information includes two bits, the migration status information (MS) may be set to 00 to represent or identify the migration status 600.

In the migration state 600, because the migration state information in the page table entry identifies the memory page as being in the migration state 600, the IOMMU allows access to the corresponding information in the memory page and page table entry from the IO device to proceed normally. That is, because preparation to migrate the memory page is not performed and the memory page is not migrated, access to the information in the memory page and page table entries is not blocked, stalled, or otherwise restricted by the IOMMU. In other words, in the migration state 600, the IOMMU normally uses information from the page table entries and normally performs memory accesses of memory pages.

However, in some embodiments, although access to memory pages and corresponding page table entries is not restricted due to migration of memory pages in migration state 600, other restrictions regarding access to information in memory pages and page table entries may be implemented. For example, if a memory page is inaccessible to an IO device because the memory page is a private memory page that is not normally accessible to the IO device, then in the migration state 600 such access restrictions are not changed.

Upon encountering a request to migrate a memory page from a software entity (e.g., hypervisor, operating system, etc.) or forwarded from a platform security processor (e.g., PSP 128), the functional blocks, devices, and software entity proceed to migration state 602 (as indicated via the thick arrow lines between migration states 600 and 602). When entering the migration state 602, the software entity sets the migration state information in the page table entry for the memory page to a bit sequence, value, etc. that represents or identifies the migration state 602. For example, assuming that the migration state information includes two bits, the migration state information may be set to 01 to represent or identify the migration state 602.

In the migration state 602, an initial preparation is made to migrate a memory page in memory. Preparing the IOMMU to include prevents new copies of information from page table entries for memory pages from being cached (i.e., stored) in the local cache in the IOMMU and/or IO devices (e.g., TLB 144, ATC 146, etc.). Generally, because subsequent migration of the memory page will cause a copy of the information from the page table entry to be incorrect because the memory page is located at a new physical address in memory, storing a copy of the information from the page table entry in a local cache in the IOMMU or IO device is prevented until the memory page has been migrated and the page table entry has been updated. Thus, in the migration state 602, because the migration state information in the page table entries identifies the memory page as being in the migration state 602, the IOMMU will not obtain copies of the information from the page table entries in order to cache such copies in a local cache in the IOMMU itself and/or provide such copies to the IO devices for storage in the local cache in the IO devices.

In some embodiments, when a copy of the information from the page table entry is not provided to the IO device, the IOMMU denies the ATS request from the IO device for the copy of the information from the page table entry. For example, in some embodiments, the IOMMU may respond to the ATS request by informing the requesting IO device that a memory page is not present, which causes the IO device to proceed with the operation for causing the page to be present. For another example, the IOMMU may inform the requesting IO device that the untranslated address should be used to access the memory page—so that information from the page table entry is not provided to the requesting IO device.

In migration state 602, the software entity also causes the IOMMU to invalidate any existing copies of information from page table entries in local caches in the IO devices (e.g., in Address Translation Caches (ATCs)). By invalidating these entries, subsequent invalidation and/or modification of copies of information from page table entries in local caches in the IO devices is avoided. Note, however, that copies of the information from the page table entries in the local caches in the IOMMU (which may still be needed to perform memory accesses in the migration state 602 and in the subsequent migration state) are preserved in the migration state 602-and eventually invalidated in the subsequent migration state (i.e., migration state 604).

In migration state 602, and in contrast to the subsequent migration states (i.e., migration states 604 and 606), the IOMMU grants memory access from the IO device to the memory page to proceed unrestricted. In other words, both existing/outstanding memory accesses from the IO device and newly encountered memory accesses are granted to proceed. Memory access is allowed to continue with memory access to clean/complete memory pages in preparation for migrating the memory pages (because the memory pages should not be migrated without the memory access being outstanding) without unnecessarily delaying operation of the IO device.

Functional blocks, devices, and software entities proceed from migration state 602 to migration state 604 (as shown via the thick arrow lines between migration states 602 and 604). For example, after performing the operations of migration state 602, the software entity may request that the page migration engine migrate the memory page to cause entry into migration state 604. When entering the migration state 604, the page migration engine sets the migration state information in the page table entry for the memory page to a bit sequence, value, etc. that represents or identifies the migration state 604. For example, assuming that the migration state information includes two bits, the migration state information may be set to 10 to represent or identify the migration state 604.

In the migration state 604, preparation for migrating memory pages in memory continues. The preparation includes the IOMMU quiescing a newly encountered first type of memory access to the memory page from the IO device. For example, in some embodiments, a first type of memory access includes a memory access that causes a response (e.g., returned data, etc.) from memory to a requesting IO device, such as a read, compare, and swap, etc. -in contrast to a second type of memory access (e.g., write, invalidate, etc.) that does not cause a response. Thus, in the migration state 604, because the migration state information in the page table entry identifies the memory page as being in the migration state 604, the IOMMU stalls the newly encountered first type of memory access until after the migration of the memory page is complete (e.g., until returning to the migration state 600). For example, the IOMMU may suspend a newly encountered first type of memory access in a buffer, memory element, etc. (e.g., by storing information from the memory access request of the IO device), and then release the suspended memory access from the buffer, memory element, etc. after the migration of the memory page is completed. Note, however, that existing/outstanding first type of memory accesses performed prior to entering the migration state 604 are permitted to complete, and that existing/outstanding and newly encountered second type of memory accesses are permitted to continue. In some embodiments, newly encountered memory accesses of a first type are shelved separately from newly encountered memory accesses of a second type (which are shelved in the migration state 606, as described below) to avoid conflicts, deadlock conditions, and/or other problems that may occur between the memory accesses of the first and second types.

In migration state 604, the page migration engine invalidates any existing copies of information from page table entries in the local cache in the IOMMU (recall that existing copies of information from page table entries in the local cache in the IO device were previously invalidated in migration state 602). For example, the page migration engine may issue an invalidation request to the IOMMU that causes the IOMMU to invalidate any existing copies of information from page table entries in the local cache. In migration state 604, the IOMMU also continues to prevent the IOMMU and/or IO device from storing copies of information from page table entries for memory pages in the local cache, as described above for migration state 602.

Functional blocks, devices, and software entities proceed from migration state 604 to migration state 606 (as shown via the thick arrow lines between migration states 604 and 606). For example, the page migration engine may enter the migration state 606 after completing invalidation of any existing copy of information from page table entries in a local cache in the IOMMU in the migration state 604. When entering the migration state 606, the page migration engine sets the migration state information in the page table entry for the memory page to a bit sequence, value, etc. that represents or identifies the migration state 606. For example, assuming that the migration status information includes two bits, the migration status information may be set to 11 to represent or identify the migration status 606.

In the migration state 606, preparation for migrating the memory page in memory is completed, and then the memory page is migrated. The operations include the IOMMU quiescing all newly encountered memory accesses from the IO device to the memory page. Thus, in the migration state 606, because the migration state information in the page table entry identifies the memory page as being in the migration state 606, the IOMMU stalls both the newly encountered first type and second type of memory accesses until after the migration of the memory page is complete (e.g., until returning to the migration state 600). For example, the IOMMU may suspend both the newly encountered first type and second type of memory accesses in a buffer, memory element, etc. (e.g., by storing information from the memory access requests of the IO devices), and then release the suspended memory accesses from the buffer, memory element, etc. after the migration of the memory pages is completed. In migration state 606, the IOMMU also continues to prevent the IOMMU and/or IO device from storing copies of information from page table entries for memory pages in the local cache, as described above for migration state 602.

In the migration state 606, existing/outstanding memory accesses are completed despite the shelving of all newly encountered memory accesses from the IO device. In other words, all remaining memory accesses made prior to entering the migration state 606 are completed. All existing/outstanding memory accesses are completed such that no memory accesses to memory pages of the memory pages are accessed during migration (as stale/incorrect data may be accessed).

After the existing/outstanding memory accesses are completed and thus all remaining accesses to the memory page are put aside to prepare the migrated memory page, the preparation of the migrated memory page is completed. Thus, the page migration engine migrates memory pages in memory. For example, the page migration engine may move a memory page from a first location/physical address in memory to a second location/physical address in memory (e.g., a location specified in a request to migrate the memory page). In some implementations, the movement in memory is from a first type of memory (e.g., slower access type of memory) to a second type of memory (e.g., faster access type of memory), or vice versa. Although migration state 606 is shown as a single migration state, in another example, the migration states may be divided into migration states 606A and 606B, with an existing/unfinished second type of memory access completed in migration state 606A, with a memory page migrated/moved in memory in migration state 606B and a cleanup operation (described below) performed in preparation for returning the memory page to migration state 600.

After migrating the memory page, the page migration engine updates (or causes another functional block or software entity to update) the page table entry to identify the physical address at which the memory page is stored (i.e., the physical address of the second location in memory). By updating the page table entries, the page migration engine updates virtual address to physical address translation information for a memory page so that virtual addresses used by software entities and devices in the electronic device can be correctly mapped to physical addresses storing the memory page. The page migration engine also sets any associated metadata in the page table entry based on the second location in memory (if necessary).

Because the memory page has been successfully migrated, the functional blocks, devices, and software entities return to the migration state 600 (as shown via the thick arrow lines between the migration states 606 and 600). As part of returning to the migration state 600, the page migration engine sets the migration state information in the page table entry for the memory page to a bit sequence, value, etc. that represents or identifies the migration state 600. For example, assuming that the migration state information includes two bits, the migration state information may be set to 00 to represent or identify the migration state 600. As described above, the migration state 600 is a "non-migration" migration state. Because the migration status information in the page table entry identifies the memory page as being in the migration status 600, the IOMMU allows access to the corresponding information in the memory page and page table entry from the IO device to proceed normally. That is, because preparation to migrate the memory page is not performed and the memory page is not migrated, access to the information in the memory page and page table entries is not blocked, stalled, or otherwise restricted by the IOMMU.

Upon returning to migration state 600, the IOMMU releases the pending memory accesses of the first and second types of memory pages previously pending during migration states 604 and 606. For example, in some embodiments, the IOMMU polls/periodically reads page table entries while pending memory accesses are suspended by the IOMMU. When the IOMMU finds that the page table entry identifies the memory page as being in the migration state 600, the IOMMU begins to fetch pending memory accesses in a first-in first-out order from a buffer or memory element storing the pending memory accesses, and performs each memory access.

In some embodiments, the IOMMU performs each pending memory access (which itself may be suspended in the order in which they were received or another order and subsequently released) before other newly encountered memory accesses of the memory page. In addition, the IOMMU restores the caching of information from the page table entries in the local cache. In other words, the IOMMU itself may cache a copy of the information from the page table entry and may provide a copy of the information from the page table entry to the requesting IO device. The page migration engine also returns a response to the requesting software entity indicating that the memory page has been successfully migrated. This completes the migration of the memory page.

For the example in fig. 6, it is assumed that the migration attempt was successful—and thus all operations for preparing the migration page and then migrating the page were successful. However, if one of the operations fails or encounters an unexpected or unexpected condition or error, the migration of the page may be terminated. When the migration is terminated, the functional blocks, devices, and software entities return from the current migration state (which may be any of the migration states 602-606) to the migration state 600. This is illustrated via the corresponding thin arrowed lines between migration states 602-604 and migration state 600 and the thick arrowed lines between migration state 606 and migration state 600. After terminating the migration of the memory page, the software entity, the page migration engine, or another functional block or device restores the migration state information in the page table entry for the memory page to a bit sequence, value, etc. that represents or identifies the migration state 600. For example, assuming that the migration state information includes two bits, the migration state information may be set to 00 to represent or identify the migration state 600. When returning to the migration state 600 due to a failure to migrate a memory page, the IOMMU frees any pending memory accesses of the memory page and resumes caching of copies of information from page table entries in the local caches in the IOMMU and IO devices—and more generally, removes the impact of preparation for migrating the memory page.

In some embodiments, access to non-migrated memory pages and their respective page table entries is normal despite limited memory access to migrated memory pages and caching information from the respective page table entries in the local cache. That is, because no preparation is made to migrate the non-migrated memory pages and the non-migrated pages will not be migrated, access to the information in the non-migrated memory pages and corresponding page table entries is not blocked, shelved, or otherwise restricted by the IOMMU. In other words, the IOMMU normally uses information from such page table entries and normally performs memory accesses for such memory pages. In some embodiments, the migration status information for the non-migrated page is set to indicate that the non-migrated page is in the migration status 600. However, certain memory accesses to non-migrated memory pages may be prevented for ordering reasons and/or other reasons based on migration of memory pages-but generally may proceed as long as the non-migrated memory pages and corresponding page table entries in memory remain correct.

In some embodiments, functional blocks, devices, and software entities may transition between migration states in different orders. For example, in some embodiments, the migration states may be skipped, as indicated by thin arrow lines between the migration states. For example, in some embodiments, only memory pages that are accessible to the processor (and thus inaccessible to the IO device) may transition directly from migration state 600 to migration state 606. In these embodiments, the processor includes a page fault mechanism for handling missing memory pages, and the software entity and/or page migration engine may simply mark the memory page as missing in the corresponding page table entry and continue to migrate the memory page. In some embodiments, the processor uses a different page table than the IOMMU, and thus memory pages may be marked as "missing" in the processor's page table during migration-this causes the memory pages to be processed by the processor using the processor's page fault mechanism. However, in some embodiments, the processor uses the same page table as the IOMMU, but it has a different indication identity available to the processor and IOMMU. In these implementations, the available indication identification of the processor may be set such that the memory page is processed by the processor using the processor's page fault mechanism, while the memory page remains "available" for IOMMU processing (via pending memory accesses, etc.), as described herein.

Although for the example shown in fig. 6, only a single memory page is migrated, in some embodiments, the preparation of migrating multiple memory pages and/or migrating the multiple memory pages may occur in parallel. In other words, using the above-described migration status information for each memory page, the preparation of migrating two or more memory pages and/or the migration of the two or more memory pages may be performed at the same or similar times, simultaneously, etc. Note that in this case, the individual memory pages may be in different migration states—and thus the migration of the memory pages may begin dynamically (e.g., when a software entity determines that each memory page is to be migrated, etc.) and complete in parallel with the preparation of migrating multiple memory pages and/or the migration of other memory pages. Each memory page is thus able to be individually identified via its page table entries as being in one of the migrated states or in the non-migrated state-and operations for handling accesses of the memory page and/or the respective page table entries are performed as described herein.

Process for handling migrated memory pages

In the described embodiments, while preparation for migrating memory pages is ongoing and memory pages are being migrated in memory (e.g., memory 104), software entities (e.g., hypervisor 318, operating system 200, etc.) and page migration engines (e.g., page migration engine 114) perform operations for controlling memory accesses of memory pages and associated page table entries. FIG. 7 presents a flowchart illustrating a process for migrating memory pages, in accordance with some embodiments. Fig. 7 is presented as a general example of operations performed by a requestor in some embodiments. However, in some embodiments, the requestor performs different operations and/or performs the operations in a different order.

As can be seen in fig. 7, the process begins when an operation is performed to prepare a memory page in memory that is accessible to a migration IO device, the software entity and page migration engine are executed to set migration status information in a page table entry for the memory page based on the operation (step 700). Generally, for this operation, when the software entity and page migration engine perform respective operations for preparing to migrate a memory page (e.g., a 4KiB memory page), the software entity and page migration engine update the migration information in the page table entry for the memory page to indicate the current/present migration state. As described above, in migration states 602-606, the software entity and/or the page migration engine performs operations for preparing to migrate the memory page, and then in migration state 606 (which may be divided into migration states 606A and 606B, as described above, with the second type of memory access existing/unfinished completed in migration state 606A, with the memory page migrated/moved in memory and the cleanup operations performed in migration state 606B), the page migration engine performs operations for migrating the memory page. Setting the migration state information in the page table entry for the memory page based on the operation being performed thus includes setting the migration state to identify the migration state of the memory page. For example, after receiving a request to migrate a memory page, when (or before) performing an initial operation to prepare to migrate the memory page, the software entity may set the migration state information to the migration state 602, such as by setting the migration state to a bit sequence or value that indicates the migration state 602.

In some embodiments, the particular operations performed by the software entity and the IO device depend on the current migration state of the memory page. For example, in some embodiments, the operation performed by the software entity is an operation to invalidate (via a corresponding command to the IOMMU) any existing copies of information from page table entries in the cache in the at least one IO device. In some embodiments, the software entity performs this operation in the migration state 602. As another example, in some embodiments, the operation performed by the page migration engine is an operation to invalidate any existing copies of information from page table entries in caches in the IOMMU. In some embodiments, the page migration engine performs this operation in migration state 604. As another example, in some embodiments, the operations performed by the page migration engine are operations to complete any memory accesses to a memory page that were not pending by the IOMMU prior to migrating the memory page. In some embodiments, the page migration engine performs these operations in migration states 602-606. The specific sequence/order in which these operations are performed by the software entity and page migration engine is described above with respect to fig. 6.

The IOMMU controls the use of information from the page table entries based on the migration status information in the page table entries (step 702). For this operation, the IOMMU (or more precisely, the IOMMU table walker) acquires the migration status information after performing the table walk and checks the current value of the migration status information. The IOMMU restricts use of information from the page table when the migration status information indicates that preparation of the migrating memory page is ongoing and/or the memory page is being migrated. For example, in migration states 602-606, the IOMMU may prevent a copy of information from page table entries from being cached in the IOMMU itself or in a cache in the IO device. In contrast, when the migration status information indicates that the page is not migrated, such as in migration status 600, the IOMMU may normally use information from the page table. Thus, in the non-migration state, the IOMMU itself may cache a copy of the information from the page table entry, and may provide a copy of the information from the page table entry to the IO device (subject to any other existing access restrictions).

The IOMMU also controls execution of memory accesses to the memory pages based on the migration status information in the page table entries (step 704). For this operation, the IOMMU obtains the migration status information from the page table entry and checks the current value of the migration status information, as described above. The IOMMU may restrict memory access to the memory pages when the migration status information indicates that preparation of the memory pages for migration is ongoing and/or the memory pages are being migrated. For example, in migration states 604-606, the IOMMU may stall the designated newly encountered memory access to the memory page-although existing memory accesses are allowed to proceed to completion before the memory page is migrated. In contrast, when the migration status information indicates that the page is in other migration states (such as in migration states 600-602), the IOMMU may grant all memory accesses to the memory page to proceed normally. Thus, in these migration states, the IOMMU typically performs memory accesses to memory pages, subject to any other existing access restrictions.

Although fig. 7 presents an example of setting and using state information in preparation for migrating a memory page, in some embodiments, migration state information is typically present in a page table entry and is otherwise used (i.e., in other migration states) to "control" access of information in the memory page and corresponding page table entry. For example, for a memory page that completes the preparation described in FIG. 7 and that is actually being migrated (i.e., is in the process of moving from one location to another in memory), migration status information is used by the IOMMU to control access of the memory page and information in the corresponding page table entries. In some embodiments, migration of memory pages is performed in migration state 606—utilizing the above-described control of access to the memory pages and information in the corresponding page table entries. For another example, for memory pages that are not migrated, and therefore for which preparation of migrating memory pages is not performed, migration status information is used by the IOMMU to control access to the memory pages and information in the corresponding page table entries. In some embodiments, operations for non-migrated memory pages are performed in migration state 600—utilizing the above-described control (or lack of control) of access to memory pages and information in corresponding page table entries.

In some embodiments, at least one electronic device (e.g., electronic device 100, etc.) performs some or all of the operations described herein using code and/or data stored on a non-transitory computer readable storage medium. More specifically, at least one electronic device reads code and/or data from a computer-readable storage medium and executes the code and/or uses the data in performing the described operations. The computer readable storage medium can be any device, medium, or combination thereof that can store code and/or data for use by an electronic device. For example, computer-readable storage media may include, but are not limited to, volatile and/or nonvolatile memory including flash memory, random access memory (e.g., eDRAM, RAM, SRAM, DRAM, DDR SDRAM, etc.), nonvolatile RAM (e.g., phase change memory, ferroelectric random access memory, spin transfer torque random access memory, magnetoresistive random access memory, etc.), read Only Memory (ROM), and/or magnetic or optical storage media (e.g., disk drive, tape, CD, DVD, etc.).

In some implementations, one or more hardware modules perform the operations described herein. For example, a hardware module may include, but is not limited to, one or more Central Processing Units (CPUs)/CPU cores, graphics Processing Units (GPUs)/GPU cores, application Specific Integrated Circuit (ASIC) chips, field Programmable Gate Arrays (FPGAs), compressors or encoders, computing units, embedded processors, accelerated Processing Units (APUs), controllers, requesters, completors, network communication links, and/or other functional blocks. When circuitry (e.g., integrated circuit elements, discrete circuit elements, etc.) in such hardware modules is activated, the circuitry performs some or all of the operations. In some embodiments, the hardware modules include general-purpose circuitry, such as an execution pipeline, computing or processing unit, etc., that performs operations when executed by instructions (e.g., program code, firmware, etc.). In some implementations, a hardware module includes special purpose or dedicated circuitry that performs the operations, perhaps including circuitry that performs some or all of the operations "in hardware" without instructions.

In some embodiments, data structures representing some or all of the functional blocks and circuit elements described herein (e.g., electronic device 100 or portions thereof) are stored on a non-transitory computer-readable storage medium including a database or other data structure that can be read by an electronic device and used directly or indirectly to fabricate hardware including the functional blocks and circuit elements. For example, the data structure may be a behavioral level description or a Register Transfer Level (RTL) description of hardware functions in a high-level design language (HDL) such as Verilog or VHDL. The description may be read by a synthesis tool that may synthesize the description to produce a netlist that includes a list of transistor/circuit elements from a synthesis library representing the functions of hardware including the functional blocks and circuit elements described above. The netlist can then be placed and routed to generate a dataset describing the geometry to be applied to the mask. The mask may then be used in various semiconductor fabrication steps to produce one or more semiconductor circuits (e.g., integrated circuits) corresponding to the functional blocks and circuit elements described above. Alternatively, the database on the computer accessible storage medium may be a netlist (with or without a synthesis library) or a data set (as needed) or Graphic Data System (GDS) II data.

In this specification, variables or unspecified values (i.e., a general description of values without a specific instance of a value) are represented by letters such as N, M and X. As used herein, although similar letters may be used in different locations throughout this specification, variables and unspecified values are not necessarily the same in each case, i.e., there may be different variables and values intended for some or all of the general variables and unspecified values. In other words, the specific examples of N and any other letters used in this specification to represent variables and unspecified values are not necessarily related to each other.

The expression "etc." or "etc." as used herein is intended to mean one and/or the same, i.e. the equivalent of at least one of the elements "in the list associated with" etc. For example, in the phrase "the electronic device performs a first operation, a second operation, etc.", the electronic device performs at least one of the first operation, the second operation, and other operations. Further, the elements in the list associated with "etc" are merely examples in a set of examples, and at least some examples may not appear in some embodiments.

The foregoing description of the embodiments has been presented only for the purposes of illustration and description. They are not intended to be exhaustive or to limit the embodiments to the forms disclosed. Accordingly, many modifications and variations will be apparent to practitioners skilled in the art. In addition, the above disclosure is not intended to limit the embodiments. The scope of the embodiments is defined by the appended claims.

Claims

1. An electronic device, comprising:

a memory;

an input-output memory management unit (IOMMU);

a processor executing a software entity; and

a page migration engine, wherein:

the software entity and the page migration engine are configured to perform operations for preparing to migrate a memory page in the memory that is accessible by at least one IO device, the software entity and the page migration engine setting migration status information in a page table entry for the memory page based on the operations being performed; and is also provided with

The IOMMU is configured to control one or more operations of the IOMMU using the migration status information in the page table entry.

2. The electronic device of claim 1, wherein when using the migration status information in the page table entry to control the one or more operations of the IOMMU, the IOMMU controls at least one of:

use of information from the page table entry; and

memory access of the memory page.

3. The electronic device of claim 2, wherein when migration status information is set in the page table entry for the memory page, the software entity and the page migration engine are configured to:

Setting, by the software entity and the page migration engine, the migration status information to a value that, when read by the IOMMU, causes the IOMMU to control use of information from the page table entries by:

not storing a copy of the information from the page table entry in a cache in the IOMMU; and

a copy of the information from the page table entry is not provided to the at least one IO device for storage in a cache in the at least one IO device.

4. The electronic device of claim 3, wherein when performing the operation for preparing to migrate the memory page, the software entity and the page migration engine are configured to:

invalidating, by the software entity, any existing copy of information from the page table entry in a cache in the at least one IO device; and

any existing copies of information from the page table entries in a cache in the IOMMU are invalidated by the page migration engine.

5. The electronic device of claim 3, wherein when a copy of information from the page table entry is not provided to the at least one IO device, the IOMMU is configured to not provide a copy of information from the page table entry to the at least one IO device in response to an Address Translation Service (ATS) request from the at least one IO device.

6. The electronic device of claim 2, wherein when migration status information is set in the page table entry for the memory page, the page migration engine is configured to:

setting the migration status information to a first value that, when read by the IOMMU, causes the IOMMU to control execution of memory accesses of the memory page by:

a newly encountered first type of memory access to the memory page is pending, but existing first and second types of memory accesses to the memory page and newly encountered second type of memory access are permitted to proceed.

7. The electronic device of claim 6, wherein when migration status information is set in the page table entry for the memory page, the page migration engine is configured to:

setting the migration status information to a second value that, when read by the IOMMU, causes the IOMMU to control execution of memory accesses of the memory page by:

both the first type and the second type of memory accesses newly encountered for the memory page are pending, but existing first type and second type of memory accesses for the memory page are granted.

8. The electronic device of claim 7, wherein when performing the operation for preparing to migrate the memory page, the page migration engine is configured to:

any memory accesses of the first type and the second type that are not pending for the memory page are ensured to be completed before the memory page is migrated.

9. The electronic device of claim 2, wherein the page migration engine is further configured to perform operations for migrating the memory page, the operations comprising:

moving the memory page from a first location to a second location in the memory; and

after the movement is completed:

updating information in the page table entry to identify the memory page as being stored at the second location in the memory; and

the migration status information is set to a value that, when read by the IOMMU, causes the IOMMU to resume normal use of information from the page table entry and normal execution of memory accesses of the memory page.

10. The electronic device of claim 9, wherein:

the memory includes two or more different types of memory, each type of memory having different memory characteristics; and is also provided with

Moving the memory page from the first location in memory to the second location in memory includes moving the memory page from a first type of memory in the type to a second type of memory in the type.

11. The electronic device of claim 2, wherein the software entity is further configured to set migration status information in other page table entries for other memory pages that are not migrated to a value that, when read by the IOMMU, causes the IOMMU to normally use information from the page table entries and normally perform memory accesses for the memory pages.

12. The electronic device of claim 1, wherein the software entity is one of an operating system or a hypervisor.

13. The electronic device of claim 1, further comprising the at least one IO device coupled to or included in the electronic device.

14. A method for migrating memory pages in memory in an electronic device, the method comprising:

performing an operation for preparing to migrate a memory page in the memory accessible by at least one IO device, the performing comprising setting migration status information in a page table entry for the memory page based on the operation being performed; and

The migration status information in the page table entry is used to control one or more operations of an IOMMU in the electronic device.

15. The method of claim 14, wherein using the migration status information in the page table entry to control the one or more operations of the IOMMU comprises controlling at least one of:

use of information from the page table entry; and

memory access of the memory page.

16. The method of claim 15, wherein setting migration status information in the page table entry for the memory page comprises:

setting the migration status information to a value that, when read by the IOMMU, causes the IOMMU to control use of information from the page table entry by:

17. The method of claim 16, wherein preparing to migrate the memory page comprises:

Invalidating any existing copies of information from the page table entry in a cache in the at least one IO device; and

any existing copies of information from the page table entries in a cache in the IOMMU are invalidated.

18. The method of claim 16, wherein not providing a copy of information from the page table entry to the at least one IO device comprises: a copy of information from the page table entry is not provided to the at least one IO device in response to an Address Translation Service (ATS) request from the at least one IO device.

19. The method of claim 15, wherein setting migration status information in the page table entry for the memory page comprises:

20. The method of claim 19, wherein setting migration status information in the page table entry for the memory page comprises:

21. The method of claim 20, wherein preparing to migrate the memory page comprises:

22. The method of claim 15, further comprising:

performing operations for migrating the memory page, the operations comprising:

after the movement is completed:

23. The method according to claim 22, wherein:

24. The method of claim 15, further comprising:

the migration status information in other page table entries for other memory pages that are not migrated is set to a value that, when read by the IOMMU, causes the IOMMU to normally use information from the page table entries and normally perform memory accesses for the memory pages.

25. The method of claim 14, wherein the software entity is one of an operating system or a hypervisor.