DE102009060265A1 - Efficient use of a remapping engine - Google Patents

Abstract

An apparatus, system and method are disclosed. In one embodiment, the device includes remapping engine reallocation logic that is capable of monitoring a first amount of traffic that is translated by a first remapping engine. When the first amount of traffic hits the first remapping engine's threshold, the logic will reroute some of the traffic to be translated by a second remapping engine.

Description

FIELD OF THE INVENTION

The Invention relates to remapping engine translations in a computer platform, implemented the virtualization.

BACKGROUND OF THE INVENTION

Many computer platforms use virtualization to more efficiently manage and prioritize resources. Input / output (I / O) devices can also benefit from virtualization. Intel ^® Corporation has released a virtualization technology for direct I / O (VT-d) specification (Revision 1.0, September 2008), which describes the implementation details, Direct Memory Access (DMA) -enabled I / O Use devices in a virtualized environment.

Around virtual addresses in DMA and interrupt requests issued by a I / O device to efficiently translate into physical memory addresses, was a logic, called remapping engine, developed which the translation performs. A given computer platform may have multiple remapping engines. The VT-D specification allows a given I / O device, such as a Platform Component Interconnect (PCI) or a PCI Express device, to be within the scope of a single remapping engine. These Assignment of a device to a remapping engine is done at the time of hardware design and is a feature of the design of the computer platform.

The Assignment of an I / O device It makes for a single remapping engine for a virtual machine monitor (VMM) or an operating system (OS) inflexible. That can also be done in reduced power result.

BRIEF DESCRIPTION OF THE DRAWINGS

The The present invention is exemplary in nature and not restrictive in any way the drawings are shown in which like reference numerals are similar Denote elements and in which:

1 describes one embodiment of a system and device to reassign remapping engines to balance the entire remapping load between available remapping engines.

2 FIG. 10 is a flowchart of one embodiment of a process to migrate an I / O device from a remapping engine to another remapping engine.

DETAILED DESCRIPTION OF THE INVENTION

embodiments from a device, System and method to reassign remapping engines and thus the balance the entire remapping load between available remapping engines, are revealed. In many scenarios, a primary remapping engine can on a computer platform due to a high amount of translations, which requests a specific dedicated I / O device (through DMA or interrupt requests) be claimed. The logic within the computer platform can notice this stressful situation and a secondary remapping engine find the present not burdened. The logic can be the I / O device to the unloaded secondary remapping engine migrate to remove the load from the primary remapping engine. When the migration is complete is, all subsequent DMA and interrupt requests of the I / O device, which translation require, by the secondary Remapping engine translated.

The References in the following description and in the claims to "one embodiment" or "embodiment" of the disclosed Technologies mean that a certain in connection with the embodiment described function, structure or property at least in one embodiment contained in the disclosed technologies. For that reason, refer different occurrences of the frequently used in this specification Expression "in an embodiment "not always necessarily on the same embodiment.

In the following description and in the claims the terms "contains" and "includes" as well as whose possibly used derivatives as synonyms to understand. In addition, in the following description and in the claims Terms "coupled" and "connected" and their Derivatives are used. However, these terms are not as Synonyms for each other to understand. Instead, in certain embodiments, "connected" may be used to indicate that two or more items are in direct physical or electrical contact with each other. "Coupled" can mean that two or more elements in direct physical or electrical Contact are. However, "coupled" can also mean that two or more elements are not in direct contact with each other, but still working or interacting with each other.

1 describes an embodiment of a system and device to reassign remapping engines to balance the entire remapping load between available remapping engines. The remapping reassignment system may be part of a computer platform (ie, computer system) that includes one or more processors. The processors may each have one or more cores. The processors may be ^Intel® microprocessors or another brand of microprocessor in various embodiments. The processors are in 1 Not shown.

The system contains a physical system memory 100 , In some embodiments, system memory 100 a kind of dynamic Random Access Memory (DRAM). For example, the system memory may be a kind of double data rate (DDR) synchronous DRAM. In other embodiments, the system memory may be another type of memory, such as a flash memory.

The system includes Direct Memory Access (DMA) and interrupt remapping logic 102 , Virtualization remapping logic, such as DMA and interrupt remapping logic 102 , protects physical regions from system memory 100 by limiting the DMA of I / O devices, such as I / O device 1 (FIG. 104 ) and I / O device 2 ( 106 ) to previously assigned physical storage regions, such as domain A ( 108 ) for the I / O device 1 ( 104 ) and domain B ( 110 ) for the I / O device 2 ( 106 ). Remapping logic also restricts I / O-generated interrupts to these regions. The DMA and interrupt remapping logic 102 may be located in a processor in the system, in an I / O complex in the system or elsewhere. An I / O complex may be an integrated circuit within the computer system that is separate from the one or more processors. The I / O complex may include one or more I / O host controllers to facilitate the exchange of information between the processor / memory and one or more I / O devices in the system, such as I / O device 1 ( 104 ) and I / O device 2 ( 106 ), to facilitate. While in certain embodiments, the DMA and interrupt remapping logic 102 can be integrated into the I / O complex, the other parts of the I / O complex are in 1 Not shown. In some embodiments, such as many "on-chip system" embodiments, the I / O complex may be integrated into a processor. So if the DMA and interrupt remapping logic 102 integrated into the I / O complex, it would therefore also be integrated into a processor in these embodiments.

The DMA and interrupt remapping logic 102 may be programmed with a Virtual Machine Monitor (VVM) in some embodiments that enable a virtualized environment within the computer system. In other embodiments, the DMA and interrupt remapping logic may be 102 be programmed by an operating system (OS).

In many embodiments, I / O device 1 (FIG. 104 ) and I / O device 2 ( 106 ) DMA-capable and interrupt-capable devices. In these embodiments, the DMA and interrupt remapping logic translates 102 the address of each incoming DMA request and interrupt from the I / O devices to the correct physical memory address in system memory 100 , In many embodiments, the DMA and interrupt remapping logic checks 102 based on the information provided by the VVM or the operating system (OS), permission to access the translated physical address.

The DMA and interrupt remapping logic 102 allows the VVM or OS to have multiple DMA protection domains, such as domain A ( 108 ) for I / O device 1 ( 104 ) and domain B ( 110 ) for I / O device 2 ( 106 ). Each protection domain is an isolated environment that contains a subset of the physical host memory. The DMA and interrupt remapping logic 102 Enables the VMM or the OS to map one or more I / O devices to a protection domain. When any I / O device tries to access a specific location in system memory 100 get the DMA and interrupt remapping logic looks 102 the remapping page tables 112 after the access authorization of this I / O device to this speci fish through protection domain. If the I / O device tries to access outside the area it is allowed to access, the DMA and interrupt remapping logic blocks 102 access and reports an error to the VMM or the OS.

In many embodiments, there are two or more remapping engines, such as remapping engine 1 (FIG. 114 ) and remapping engine 2 ( 116 ) in the DMA and interrupt remapping logic 102 are integrated. Each remapping engine contains logic to handle streams of DMA requests and interrupts from one or more I / O devices. The remapping engines generally start as specific I / O devices. For example, remapping engine 1 ( 114 ), the DMA requests and interrupts to domain A ( 108 ) to be handled by I / O device 1 ( 104 ) and remapping engine 2 ( 116 ) can be assigned to the DMA requests and interrupts to domain B ( 110 ) to be handled by the I / O device 2 ( 106 ) were received.

Although the remapping engines may be originally assigned to translate DMA requests and interrupts to physical addresses for a specific I / O device, in many embodiments, remapping reassignment logic may be used 118 dynamically change these original assignments for each remapping engine based on observed workloads. In many embodiments, the DMA and interrupt remapping logic become 102 and the remapping reassignment logic 118 used together in a computer platform that uses I / O virtualization technologies. For example, I / O device 1 (FIG. 104 ) generate a very heavy DMA request workload while I / O device 2 ( 106 ) is inactive. The heavy DMA request workload of I / O device 1 ( 104 ), the capacity of remapping engine 1 ( 114 ), which results in a deterioration of the performance (ie response time) for the requests from I / O device 1 ( 104 ) and one or more additional I / O devices (not shown) that are also used by the remapping engine 1 (FIG. 114 ) could be dependent. This example can use remapping reassignment logic 118 notice the discrepancy in the workloads and decide which of I / O device 1 ( 104 ) received DMA request workload between the remapping engine 1 ( 114 ) and the otherwise unused remapping engine 2 ( 116 ). Thus, the additional capacity of remapping engine 2 (FIG. 116 ) the workload used by remapping engine 1 ( 114 ) and facilitate the responsiveness to requests for I / O device 1 (FIG. 104 ) increase.

In another example, the opposite may be the case where remapping engine 2 (FIG. 116 ) with I / O device 2 ( 106 ) is overloaded, and thus remapping reassignment logic 118 a part of the work received for the remapping engine 1 ( 114 split off). In yet another example, a third I / O device (not shown), the initial remapping engine 1 (FIG. 114 ), a lot of interrupt traffic for translation to remapping engine 1 ( 114 ). This interrupt traffic from I / O device 3 may be more traffic than the combination of DMA and interrupt requests from I / O devices 1 and 2 together. In this example, remapping reassignment logic 118 it remapping engine 1 ( 114 ) to handle the incoming requests from I / O device 3, but can handle I / O device 1 ( 104 ) to remapping engine 2 ( 116 reassign). Thus, remapping engine 2 ( 116 ) now translate the incoming requests for both I / O device 1 and I / O device 2.

Many DMA and interrupt traffic scenarios can use the remapping-reassignment logic 118 try to reassign DMA requests from one remapping engine to another to balance the received workload among all available remapping engines. In many in 1 In the embodiments not shown, there may be a pool of remapping engines that includes more than two entire remapping engines. In these embodiments, the remapping reassignment logic 118 Reassign work to each of the remapping engines in the pool to fairly balance the total number of DMA requests across the entire pool. In some embodiments, when a single remapping engine, such as remapping engine 1 (FIG. 114 ), which performs all the DMA request work, but the amount of work is small enough not to burden the capacity of the particular remapping engine, can be the remapping-reassignment logic 118 Do not reassign a portion of the DMA request workload. In some embodiments, the reallocation is generally performed when the workload for a given remapping engine has reached the threshold of the remapping engine for requests. In many embodiments, again, the DMA and interrupt remapping logic 102 and the remapping-reassignment logic 118 both used in a computer platform that uses I / O virtualization technologies.

In many embodiments, the threshold of requests is a number of requests during a given period of time, which is similar to the limit that the remapping engine can handle without performance degradation. A degradation in the performance of the remapping engine can be caused by a queue of accumulating DMA requests, since the requests are handled by the Re The mapping engine can be received at a faster rate than the remapping engine can translate queries. The remapping reassignment logic 118 can use one of several different methods to compare the current workload of DMA requests with the threshold. For example, a ratio of requests to system clock cycles may be compared to a threshold ratio. The watchdog logic can be used in the remapping reassignment logic 118 be integrated because it receives all requests from the set of I / O devices and maps each request to a remapping engine.

In many embodiments, the DMA remapping logic 102 one or more control registers for the VMM or OS ready to remap the ability to reassign DMA request workloads between remapping engines for the remapping reassignment logic 118 to activate or deactivate. In many embodiments, remapping engines may be referred to as equivalent remapping engines if the same set of I / O devices is available to each one. Thus, a remapping engine could theoretically execute DMA query translations for one set of I / O devices while a second remapping engine is idle, the opposite also being true. If an I / O device is accessible to a remapping engine, but not to another remapping engine, the remapping engines can not be considered equivalent. Equivalent remapping engines enable remapping reassignment logic 118 Free to mix and match DMA query workloads with any equivalent remapping engine.

If the equivalence between VMM or OS remapping engines is enabled by the one or more control registers, each remapping engine can use the same set of remapping page tables 112 and actively use all other remapping-related registers to participate in the DMA request translation process. In many embodiments, the one or more control registers are software-based registers residing in system memory, such as the control registers 120A , In other embodiments, the one or more control registers are hardware-based registers that are physically in the DMA remapping logic 102 located, such as the control registers 120B ,

In many embodiments, the DMA remapping logic may be 102 to the VMM or OS communicate the equivalent relationship between two or more remapping engines by using an extension to the current DMA Remapping Hardware Unit Definition (DRHD) structure defined in the ^Intel® VT-d specification.

Each remapping engine has a DRHD structure in memory. For example, the DRHD structures may reside in the remapping page tables / structures 112 which are part of the system memory 100 are. In other embodiments, the DRHD structure may be located elsewhere in the system memory 100 are located. The DRHD structure for each remapping engine contains an array of remapping engines that are equivalent to the remapping engine in question. This array is called the "equivalent DRHD Array". This array is a collection of fields. It is defined in Table 1. The array is used to tell the VMM or OS such equivalence. It is up to the VMM or OS to decide, if necessary, to use the alternative remapping engines with the remapping engine that was primarily assigned to a given I / O device. Table 1. The structural layout of an equivalent DRHD array. Number of units in the equivalent array A 16-bit field that indirectly indicates the length of this entire field. Base address of the first equivalent unit That's a 64-bit field. See the register base address in the DRHD (Section 8.3 of the VT-d Specification). Base address of the nth equivalent unit N is displayed in the first field.

In some embodiments, the remapping reassignment logic 118 report the DMA request translation workload for each remapping engine to the VMM or OS, which would allow the VMM or OS to decide to enable and use alternate remapping engines to reduce translation pressure to the primary remapping engine.

The DMA remapping logic 102 can also communicate information about the capabilities of each remapping engine in terms of migrating remapping page tables between remapping engines. Specifically, if the VMM or OS determines the mapping entries for DMA and Inter There may be a software-based or hardware-based page table copy of root requests from one remapping engine to another.

In some embodiments, the VMM or OS may set up the page tables relative to the newly assigned I / O device and then copy the remapping page tables from the old remapping engine storage for page tables to the new remapping engine storage for page tables. In other embodiments, the DMA and interrupt remapping logic may be 102 silently copy the page tables between the remapping engine storage locations. Quiet copying of these page tables allows VMM or OS software level overhead to be removed and brought to a lower hardware level. This can be done without the knowledge of software.

If the page tables once from the old remapping engine storage space to be copied to new (that is, migrated) is the new remapping engine the one who for the handling of all future translation requests of the relevant I / O device responsible for. The old remapping engine is not for the I / O device more responsible and will not receive any DMA or interrupt request received from the device translate more.

2 FIG. 10 is a flowchart of one embodiment of a process to migrate an I / O device from a remapping engine to another remapping engine. The process is performed by processing logic, which may be hardware, software, or a combination of hardware and software. The process begins by processing a DMA or interrupt request from an I / O device (processing block 200 ) receives.

The processing logic determines whether the primary remapping engine assigned to service the request reaches its threshold of requests over a certain period of time (processing block 202 ) has reached. This determination may use performance counters, timestamps, algorithms, and other methodologies to determine if the primary remapping engine currently has enough translation requests to degrade the translation responsiveness of the engine per request.

For example, the VMM or the OS can use any remapping engine either directly or through the remapping assignment logic 118 to query the current state of remapping translation printing on each remapping engine. In another example, the DMA and interrupt remapping logic 102 disrupt the VMM or OS when at least one of the remapping engines begins to experience translation pressure or constraints on its translation resources. In both examples, the DMA and interrupt remapping logic 102 also communicate more detailed information about the exact nature of the translation pressure, including the hierarchy or the exact I / O devices that are the cause of the translation pressure. The VMM or OS can decide which performance information, if any, to use when determining whether to migrate the translation entries of one I / O device to another equivalent remapping engine.

Let's go back to 2 : If this request threshold has not been reached, the processing logic causes the primary remapping engine to translate the DMA or interrupt request, and the process is complete.

Otherwise, when the threshold of requests has been reached, the Processing logic, which remapping engine of one or more other equivalents Remapping engines available is and is either present too little or at all not used. This may include the determination of whether there is enough overcapacity in one given backup remapping engine gives to the extra Pressure in the additional Traffic of the device take.

Once an available backup remapping engine is found, processing logic then migrates the remapping page tables for the I / O device from the primary remapping engine to the backup remapping engine (processing block 206 ). If the backup remapping engine has received the page tables of the I / O device that can be used for remapping, then processing logic redirects the DMA or interrupt request to the backup remapping engine (processing block 208 ), and the process is complete.

In many embodiments, processing logic, after verifying that there is an available equivalent remapping engine, may have a control register in the hardware ( 1 . 120B ) to indicate that the new backup remapping engine is equivalent to the primary remapping en gine should be considered.

To accommodate this register programming, currently occupied fields in the Global Command Register (currently defined in the ^Intel® VT-d specification) can be redefined for this command (eg, a command that "Enable equivalent remapping Engine is called). The new remapping engine can be identified by another 8-byte register for this purpose. Table 2 shows an example of the modifications made to the global command and status registers to allow for equivalence checking among remapping engines. Table 2. Global command and status register bits used for remapping engine equivalence. Bit 21 of the global command register If set to 1, a new equivalent remapping engine has been identified. If set to 0, any existing equivalence relationship is removed. Bit 21 of the global status register This bit is set to 1 after the hardware finishes executing the command.

The VMM or the OS may enable equivalence for either all current devices belonging to the scope of the remapping engine A, or only for a particular set of devices that currently belong to the scope of the remapping engine A. If the equivalence can not be performed, the DMA and interrupt remapping logic can be used 102 communicate this error status through an error register.

Consequently become embodiments from a device, System and method disclosed to reassign remapping engines and thus the entire remapping load between available To balance remapping engines. The embodiments described herein were with reference to certain exemplary embodiments explained. It is for People who have insight into this revelation, obviously, that different adjustments and changes in these embodiments must be made without going in the broader sense and scope of the herein described embodiments departing. The description and the drawings are accordingly exemplary Representation, but not in a limiting sense.

Claims (24)

Device, which includes: Remapping engine reallocation logic to monitor a first amount of traffic passing through a first Remapping engine translated becomes; and redirect some of the first amount of traffic to be translated by a second remapping engine when the first amount of traffic a first remapping engine traffic threshold reached. device The claim of claim 1, wherein the remapping engine reassignment logic further comprises causes: before redirecting the portion of the first amount of traffic querying the second remapping engine to determine an amount of available translation capacity; and the enabling the diversion, if available Amount of translation capacity in the Able to be the part of the first amount of traffic to be diverted serve. device The method of claim 2, wherein the remapping engine reassignment logic further comprises causes: determining the capacity of the first and second remapping engines; and splitting up a portion of the first amount of traffic respectively the first and the second remapping engine, so each one Engine in proportion to the maximum capacity each engine has essentially the same percentage of traffic having. device The claim of claim 3, wherein the remapping engine reassignment logic further comprises causes: redirecting part of the first amount of traffic to one or more additional ones Remapping engines, with the first, the second and one or more additional Remapping Engines in Relation to the maximum capacity each engine essentially the same percentage Have traffic. device according to claim 1, wherein the first amount of traffic at least the Traffic from a first device and a second device includes. device according to claim 5, wherein the part of the first amount of traffic, the redirected to the second remapping engine, at least the traffic from the second device includes. device The claim of claim 1, wherein the remapping engine reassignment logic further comprises causes: the monitoring the first amount of traffic passing through the first remapping engine and the second Remapping engine translated becomes; and redirecting the part of the first amount of traffic, which is translated by the second remapping engine, to the first remapping engine, if the first amount of traffic is below the first traffic threshold returns. device The method of claim 7, wherein the remapping engine reassignment logic further comprises causes: communicating to a power management logic, that the second remapping engine can be stopped after the Part of the first amount of traffic being translated by the second remapping engine will be returned to the first remapping engine. System comprising: a first device and a second device; a first remapping engine and a second remapping engine, each one Remapping engine with both the first and the second Device coupled is; and Remapping engine reassignment logic that works with both the first as well as the second remapping engine is coupled, with the remapping engine reassignment logic a monitored first amount of traffic, which translates through a first remapping engine becomes; and a part of the first amount of traffic passing through a second remapping engine to translate is, redirects when the first amount of traffic a first remapping engine traffic threshold reached. The system of claim 9, wherein the remapping engine reassignment logic further comprises causes: before redirecting the portion of the first amount of traffic querying the second remapping engine to determine an amount of available translation capacity; and the enabling the diversion, if available Amount of translation capacity the part can serve the first amount of traffic to be redirected. The system of claim 10, wherein the remapping engine reassignment logic further comprises causes: determining the capacity of the first and second remapping engines; and splitting up a portion of the first amount of traffic respectively the first and the second remapping engine, so each one Engine in proportion to the maximum capacity each engine has essentially the same percentage of traffic having. The system of claim 11, wherein the remapping engine reassignment logic further comprises causes: redirecting part of the first amount of traffic to one or more additional ones Remapping engines, with the first, the second and one or more additional Remapping Engines in Relation to the maximum capacity each engine essentially the same percentage Have traffic. The system of claim 9, wherein the first quantity Traffic comprises at least the traffic from the first device and the second device. The system of claim 13, wherein the part of the first Amount of traffic redirected to the second remapping engine at least the traffic of at least the second device comprises. The system of claim 9, wherein the remapping engine reassignment logic further comprises causes: the monitoring the first amount of traffic passing through the first remapping engine and the second Remapping engine translated becomes; and redirecting the part of the first amount of traffic, which is translated by the second remapping engine, to the first remapping engine, if the first amount of traffic is below the first traffic threshold returns. The system of claim 15, further comprising: power management logic to control the power provided to at least each of the first and second remapping engines, wherein the remapping engine reassignment logic further comprises: communicating with the power management logic to lower at least the amount of power delivered to the second remapping engine after the portion of the first amount of traffic translated by the second remapping engine is returned to the first remapping engine. A method comprising: the monitoring a first set of traffic translated by a first remapping engine becomes; and diverting a portion of the first amount of traffic, which is to be translated by a second remapping engine, if the first Amount of traffic a first remapping engine traffic threshold reached. The method of claim 17, further comprising includes: before redirecting the portion of the first amount of traffic querying the second remapping engine to determine an amount of available translation capacity; and the enabling the diversion, if available Amount of translation capacity in the Able to be the part of the first amount of traffic to be diverted serve. device according to claim 18, further comprising: the determining the capacity the first and second remapping engines; and splitting a portion of the first amount of traffic to each of the first and second remapping engine, so that each engine in relation to maximum capacity each engine has essentially the same percentage of traffic having. device according to claim 19, further comprising: the rerouting a portion of the first amount of traffic to one or more additional ones Remapping engines, with the first, the second and one or more additional Remapping Engines in Relation to the maximum capacity each engine essentially the same percentage Have traffic. device according to claim 17, wherein the first amount of traffic at least the Traffic from a first device and a second device includes. device according to claim 21, wherein the part of the first amount of traffic, the redirected to the second remapping engine, at least the traffic from the second device includes. device according to claim 17, further comprising: monitoring the first set traffic translated by the first remapping engine and the second remapping engine becomes; and redirecting the part of the first amount of traffic, which is translated by the second remapping engine, to the first remapping engine, if the first amount of traffic is below the first traffic threshold returns. device according to claim 23, further comprising: communicating a power management logic that stops the second remapping engine can be after the part of the first amount of traffic passing through the second remapping engine translated is returned to the first remapping engine.

Images