KR101444990B1 - Method and apparatus for performing energy-efficient network packet processing in a multi processor core system - Google Patents

Abstract

A method and apparatus for core affinity management for network packet processing is provided. A low-power idle state of a plurality of processing units in a system including a plurality of processing units is monitored. The network packet processing adaptively reallocates to the low-power idle processing units to increase the low-power idle state residency for the non-low power idle processing units, resulting in reduced energy consumption.

Description

[0001] METHOD AND APPARATUS FOR PERFORMING ENERGY-EFFICIENT NETWORK PACKET PROCESSING IN A MULTI PROCESSOR CORE SYSTEM [0002]

This disclosure relates to energy-efficiency in a multi-processor core system, and more particularly to energy-efficient network packet processing.

Typically, a computer system with multiple processor cores handles high workloads by distributing the workload among all of the processor cores. However, as the workload decreases, each of the plurality of processor cores may not be fully utilized.

To reduce power consumption by the plurality of processor cores when the workload is low, the operating system may adjust the number of processor cores used based on the system utilization level. Unused processor cores may be left in a low-power idle state (parked and parked) and held in a low-power idle state for long successive intervals. The operating system continues to distribute the workload among the processor cores rather than the low-power idle state.

Features of embodiments of the present invention will become apparent as the following detailed description proceeds, with reference to the drawings in which like reference numerals denote like parts.
1 is a block diagram of a system including an embodiment of a network interface controller that supports receive side scaling.
2 is a block diagram illustrating an embodiment of the network interface controller and memory shown in FIG.
Figure 3 is a flow diagram of an embodiment of a method for dynamically adjusting core affinity settings in accordance with the principles of the present invention.
4 is a block diagram illustrating another embodiment of the network interface controller and memory shown in FIG.
Although the following detailed description proceeds with reference to the illustrative embodiments of the invention, many alternatives, modifications and variations thereof will be apparent to those skilled in the art. Accordingly, the invention is intended to be broadly construed and defined only as set forth in the appended claims.

The computer system may include a network interface controller (adapter, card) that receives network packets from the network and forwards the received network packets for processing to one of the plurality of processor cores. The workloads distributed to the process cores may include processing of network packets received by the network interface controller.

For example, in a computer system, the processing of network packets may be distributed to processor cores such that processing of the same traffic flow (e.g., network packets having the same source address and destination address) is performed by the same processor core have. When the workload is low, the operating system can only use a subset of the plurality of processor cores and set the other processor cores in a low-power idle state. However, if a received network packet to be processed for a particular traffic flow is assigned to a low-power idle processor core (core affinity setting for traffic flow), then the processor core wakes up from a low-power idle state. As a result, processor cores that are low-power idle do not have the opportunity to remain in a low-power idle state for a long time.

Embodiments of the present invention dynamically adjust core affinity settings for network packet processing based on whether the operating system has set the processor cores to a low-power idle state.

Embodiments of the present invention will be described with respect to a computer system having a network interface controller that supports Receive Side Scaling (RSS) used by the Microsoft's® Windows® operating system (OS). However, the present invention is not limited to RSS only. In other embodiments, the network adapter may be a scalable input / output (I / O) device used by a Linux operating system including a scheduler with power-saving mode capability or any other operating system including power- .

1 is a block diagram of a system 100 that includes an embodiment of a network interface controller 108 that supports receive side scaling. The system 100 includes a processor 101, a memory controller hub (MCH) 102, and an input / output (I / O) controller hub (ICH) The MCH 102 includes a memory controller 106 that controls communication between the processor 101 and the memory 110. [ The processor 101 and the MCH 102 communicate via the system bus 116.

The processor 101 may be an Intel® Pentium D, an Intel® Xeon® processor, or an Intel® Core ™ Duo processor, an Intel® Core ™ i7 processor, or any other type of processor. In the illustrated embodiment, the system includes two multi-core processors 101 each having at least two processor cores ("cores") 122. In one embodiment, each multi-core processor includes four cores 122. In one embodiment,

The memory 110 may be a memory such as a Dynamic Random Access Memory (DRAM), a Static Random Access Memory (SRAM), a Synchronized Dynamic Random Access Memory (SDRAM), a Double Data Rate 2 (DDR2) RAM, a Rambus Dynamic Random Access Memory But may be of any other type.

ICH 104 may be coupled to MCH 102 using a high speed chip-to-chip interconnect 114 such as a Direct Media Interface (DMI). DMI supports 2 gigabit / second concurrent transmission rates through two unidirectional lanes.

The ICH 104 may include a storage input / output (I / O) controller for controlling communication with at least one storage device 112 coupled to the ICH 104. For example, the storage device may be a disk drive, a digital video disk (DVD) drive, a compact disk (CD) drive, a Redundant Array of Independent Disks (RAID), a tape drive, or other storage device. ICH 104 may communicate with storage device 112 via storage protocol interconnect 118 using a serial storage protocol such as Serial Attached Small Computer System Interface (SAS) or Serial Advanced Technology Attachment (SATA).

In another embodiment, network interface controller 108 in system 100 may be included in an ICH 104 that does not include a storage I / O controller 120, or may be included on a separate network interface card inserted into a system card slot have.

2 is a block diagram illustrating an embodiment of the network interface controller 108 and memory 110 shown in FIG. The network interface controller 108 includes a hash function unit 220, an indirect table 230 and a plurality of hardware receive queues 202. The memory 110 includes an operating system kernel 280, a filter driver 210, and a network device driver (miniport driver) In an embodiment, the hash function unit 220 and the indirect table 230 are included in a network interface controller ("NIC") 108. In other embodiments, some of these components or some of the components may be located outside the network interface controller 108.

In the illustrated embodiment, miniport driver 270 and filter driver 210 are components of the Microsoft® Windows® operating system model (WDM). WDM includes device function drivers that can be class drivers or miniport drivers. The miniport driver may support a specific type of device, for example, a particular network interface controller 108. [ The filter driver 210 is an optional driver that adds value or modifies the operation of the function driver (miniport driver) 270.

The Network Driver Interface Specification (NDIS) is a library of functions accessed through an application programming interface (API) to the network interface controllers 110. NDIS operates as an interface between Layer 2 (link layer) and Layer 3 (network layer) of the 7 Layer Open System Interconnect (OSI). The library function contains object identifiers (OIDs).

The network interface controller 108 may use a hash function of the hash function unit 220 to compute a hash value for each hash type in the received network packet (e.g., one or more fields in the header). Many of the least significant bits of the hash value are indirectly identified by processor core 122 (FIG. 1) that handles the processing of the received data packet and one of a plurality of receive queues 202 that store the received data packet May be used to index an entry in table 230. < RTI ID = 0.0 > The network interface controller 108 may interrupt the identified processor core 122. [

In the embodiment, as each network packet is received by the network interface controller 108, a "flow" associated with the network packet is determined. The "flow" of a Transmission Control Protocol (TCP) packet may be determined based on the Internet Protocol (IP) header included in the packet and the field value of the TCP header. For example, the flow identifier for the "flow" may depend on the combination of the IP source address and the IP destination address included in the IP header and the source port address and destination port address included in the TCP header of the received network packet.

A plurality of hardware receive queues 202 are provided to store the received network packets. Each hardware receive queue 202 may be assigned to a different flow and also to one of the plurality of processor cores 122 to ensure sequential packet delivery for a particular flow. Thus, each of the plurality of hardware receive queues 202 is associated with one of the plurality of core processors 122 via an indirect table 230.

In the illustrated embodiment, the network interface controller 108 has eight receive cues 202. However, the number of reception cues 202 is not limited to eight. In other embodiments, the number of receive cues 202 may be more or less. The received network packets are stored in one of the plurality of receive queues 202. [ The packet processing of each receive queue may be affinitized to a particular processor core 122. In an embodiment of the invention, the core affinity settings for packet processing may be dynamically adjusted based on the number of processor cores 122 that are power-saving idle ('parked').

Table 1 shows an indirect table (FIG. 1) that initially allocates the reception queues 202 to the processor cores 122 in a system having 8 cores and 8 receive queues for distributing received packet processing to all processor cores 122 230 are shown below.

3 is a flow diagram of an embodiment for dynamically adjusting core affinity settings in accordance with the principles of the present invention.

The distribution of the receive queues 202 to the core processors 122 is dynamically modified based on the OS core parking state handled by the OS kernel. For example, if the workload is low, the OS kernel can "park" them by setting some of the core processors 122 to a low-power state. For example, in an embodiment with eight core processors, six of the eight core processors may be "parked " and the remaining core processors (e.g., two of the eight) may be used. Thus, interrupts from the network interface controller are sent only to the non-parked core and the parked cores remain in a low-power idle state to reduce energy consumption.

For example, in an embodiment, when the Windows® 7 server core parking function chooses to use three cores and parks the remaining cores when the workload is low, the network interface controller performs packet processing on the selected three cores and It is modified to execute a network interrupt. This increases the lower power idle state residency for the parked cores.

In an embodiment, the 10 Gigabit network interface controller has multiple receive side scaling receive (RSS Rx) queues each associated with a particular core. The network interface controller obtains the configuration of the OS core parking configuration through the filter driver 210 and modifies the RSS indirect table 230 based on the OS core parking configuration.

At block 300, the filter driver 210 periodically requests an OS parking state from the OS kernel 280. [ For example, in an embodiment, the filter driver 210 may use an API command provided by the OS kernel 280 to query the parking status of each of the processor cores 122 to determine the OS core parking status < RTI ID = 0.0 >Lt; / RTI > In an embodiment, for example, the filter driver 210 periodically requests an OS parking state based on the number of received network packets after each about 1000 network packets are received.

In another embodiment, the filter driver 210 accesses the CPU 101 to obtain the current power state of each processor core 202 directly. For example, Intel® Core ™ i7 supports low power states at the core level for optimal power management. The core power states include C0, C1, C3, and C6. C0 is in a normal operating state and C1, C3 and C6 are in a low power state with different levels of reduced power consumption. For example, in the C3 low-power state, all clocks are stopped and the processor core maintains all its structural state except for the cache.

The current power state of each processor core 202 is stored in one or more registers of the CPU 101 accessible by the filter driver 210. To determine if a particular core has been "parked ", the filter driver 210 periodically reads a register that stores the power state for the cores and, based on the distribution of the power state type of the core read over time period, "Parked" For example, within a time period, the cores are "parked" if the registers are read n times and the cores are in each low power state. During the time period, if the core is in the high power state n times, then the core is not "parked. &Quot; During a time period, if the power state of the core is different, i. E., The number of times the power state is in the low power state is greater than the number of times the power state is in the high power state, then the core is estimated to be "parked. Processing continues at block 302.

At block 302, upon receiving the parked state of each processor core 122, the filter driver 210 determines which of the processor cores 122 has changed the parked state. If not, the processing continues at block 300, which continues to periodically request the parking status of the processor cores 122.

If so, the filter driver 210 generates new data stored in the indirect table 230 of the NIC 108 based on the parked state. The filter driver 210 uses the OID_GEN_RECEIVE_SCALE_PARAMETERS object identifier (OID) to modify the RSS parameters of the NIC based on the parked state of the processor cores. OID_GEN_RECEIVE_SCALE_PARAMETERS The OID contains NDIS_RECEIVE_SCALE_PARAMETERS specifying the RSS parameters.

In an embodiment, the structure includes a header having a type that specifies that the object includes an RSS parameter, a flag indicating whether the indirect table and associated members have changed, and the size of the indirect table. New data stored in the indirect table 230 based on the core parking state is appended after other structure members. Processing continues at block 304.

At block 304, upon detecting the receipt of the OID_GEN_RECEIVE_SCALE_PARAMETERS OID, the miniport driver 270 stores the new data received for the indirect table 230 in the indirect table 230. [ For example, in an embodiment with eight receive (RSS Rx) queues 202, if the retrieved core parking state indicates that only processor core 0 and processor core 4 are not parked, then a round-robin core allocation (The contents of the indirect table) in the indirect table 230 as shown in Table 2 below. Processor core 0 is assigned to receive cues 0, 2, 4, and 6, and processor core 4 is assigned to receive cues 1, 3, 5, and 7.

For example, the data stored in Table 2 may be denoted as data structure IndTable as shown below:

IndTable [0] = 0

IndTable [1] = 4

IndTable [2] = 0

IndTable [3] = 4

IndTable [4] = 0

IndTable [5] = 4

IndTable [6] = 0

IndTable [7] = 4

The IndTable structure is appended to the OID received by the miniport driver 270 to cause the miniport driver 270 to update the data stored in the indirect table 230.

The embodiment has been described with respect to a system including a filter driver 210. However, it is not limited to filter driver 210 to monitor core parking status and request modification of indirect table 230. In other embodiments, these functions may be included in other portions of the network driver stack.

4 is a block diagram illustrating another embodiment of the network interface controller 108 and memory 110 shown in FIG.

As shown in FIG. 4, instead of adding the filter driver 210 (FIG. 2) to the network driver stack, monitoring of the core parking state is performed by the OS kernel 280. A request to update the indirect table 230 is sent directly from the OS kernel 280 to the miniport driver 270. [ Similar to the embodiment discussed with respect to the filter driver 210 in conjunction with Figure 2, the OS kernel 280 may include modified content for the indirect table 230, as discussed with the embodiment shown in Figure 2, Create OID_GEN_RECEIVE_SCALE_PARAMETERS OID as an input parameter. This OID is sent directly to the device driver 270 that modifies the indirect table 230 based on the received modified content.

In an embodiment, the RSS alignment function is added to the OS core to adjust the RSS indirection table 230.

An embodiment of the Windows® operating system has been described. The modification of the contents of the indirect parking table 230 based on the monitoring of the core parking status and the monitored core parking status is not limited to the Windows® operating system and the method can be applied to other operating systems such as, for example, the Linux operating system .

Other embodiments of the invention may also include machine-accessible media including instructions for performing the operations of the present invention. Such embodiments may also be referred to as a program product. Such machine-accessible medium is a computer-readable storage medium having stored thereon instructions (computer-readable program code), which may be formed by a floppy disk, hard disk, CD-ROM, ROM, and RAM and a machine or device But may include, without limitation, other types of configurations of parts being manufactured. The instructions may also be used in a distributed environment and may be stored locally and / or remotely for access by a single or multi-processor machine.

While the embodiments of the present invention have been particularly shown and described with reference to the embodiments thereof, it will be appreciated that various changes in form and detail may be made therein without departing from the scope of the embodiments of the invention which are defined by the claims Will be understood by those skilled in the art.

Claims (19)

Monitoring a low power idle state of each of the plurality of processing units; And
And re-allocating the flows allocated to the processing units having a low power idle state to other processing units having a non-low power idle state such that energy consumption by the plurality of processing units And causing only network interface controller interrupts to be sent to the processing units having the non-low power idle state
≪ / RTI > The method according to claim 1,
Adaptively re-assigning flows includes reallocating the flows through an indirect table, wherein the indirect table includes a plurality of entries, each entry having a processing associated with a flow associated with a queue And identifying the unit. The method according to claim 1,
Wherein monitoring the low power idle state of the plurality of processing units is performed by an operating system. The method according to claim 1,
Wherein monitoring the low power idle state of the plurality of processing units is performed by a driver associated with the network interface controller. delete A computer-readable storage medium storing instructions,
The command, when accessed, causes the computer to:
Monitoring a low power idle state of each of the plurality of processing units; And
Allocating flows assigned to the processing units having a low power idle state to other processing units having a non-low power idle state to reduce energy consumption by the plurality of processing units, Causing only controller interrupts to be sent to the processing units having the non-low power idle state
Readable < / RTI > storage medium. The method according to claim 6,
Adaptively re-assigning flows includes reallocating the flows through an indirect table, wherein the indirect table includes a plurality of entries, each entry having a processing associated with a flow associated with a queue A computer-readable storage medium comprising an identification of a unit. The method according to claim 6,
Wherein monitoring the low power idle state of the plurality of processing units is performed by an operating system. The method according to claim 6,
Wherein monitoring the low power idle state of the plurality of processing units is performed by a driver associated with a network interface controller. A plurality of processing units; And
Monitors the low power idle state of each of the plurality of processing units and adaptively reassigns the flows assigned to the processing units having a low power idle state to other processing units having a non-low power idle state A module that reduces energy consumption by the plurality of processing units and only causes network interface controller interrupts to be sent to the processing units having the non-
/ RTI > 11. The method of claim 10,
Further comprising an indirect table comprising a plurality of entries, each entry including an identification of a processing unit to which a flow associated with the queue is assigned, the module adapted to re- allocate the flows via the indirect table A device that reallocates to a target device. 11. The method of claim 10,
Wherein the module is included in an operating system. 11. The method of claim 10,
Wherein the module is a driver associated with a network interface controller. 14. The method of claim 13,
Wherein the driver is a filter driver. 11. The method of claim 10,
And a disk drive for storing data packets associated with flows received from the network. delete delete delete delete

Images