KR20230159602A - Address hashing in multi-memory controller systems - Google Patents

Abstract

In one embodiment, the system may support programmable hashing of address bits at multiple levels of granularity to map memory addresses to memory controllers and ultimately to at least memory devices. Hashing can be programmed to distribute pages of memory across memory controllers, and consecutive blocks of a page can be mapped to physically distant memory controllers. In one embodiment, address bits can be dropped from each granularity level to form a compressed pipe address to save power within the memory controller. In one embodiment, a memory folding scheme may be employed to reduce the number of active memory devices and/or memory controllers in a system when the full complement of memory is not needed.

Description

Address hashing in multi-memory controller systems

Technology field

Embodiments described herein relate to memory addressing in computer systems, and more particularly to distributing memory address space across multiple memory devices.

background technology

Contains a large amount of system memory that is directly accessible to processors and other hardware agents within the system through the memory address space (e.g., compared to the I/O address space, which is mapped to specific I/O devices) There are various computer systems that do this. System memory is typically implemented as multiple dynamic random access memory (DRAM) devices. In other cases, other types of memory, such as static random access memory (SRAM) devices, various types of magnetic memory devices (e.g., MRAM), flash memory, or non-volatile memory devices such as read-only memory (ROM). , other types of random access memory devices may also be used. In some cases, in addition to the portions of the memory address space that are mapped to RAM devices, a portion of the memory address space may be mapped to such devices (and memory-mapped I/O devices may also be used).

The mapping of memory addresses to memory devices can have a significant impact on the performance of a memory system (eg, in terms of sustainable bandwidth and memory latency). For example, typical non-uniform memory architecture (NUMA) systems are comprised of computing nodes that include processors, peripheral devices, and memory. Computing nodes communicate, and one computing node can access data in another computing node, but at increased latency. The memory address space is mapped in large contiguous sections (e.g., one node contains addresses 0 through N-1, where N is the number of bytes of memory in the node, and another node contains addresses N through 2N-1). including etc.). This mapping optimizes accesses to local memory at the expense of accesses to non-local memory. However, this mapping also limits the operating system both in how it maps virtual pages to physical pages and in the choice of computing nodes on which a given process can run in the system to achieve higher performance. Additionally, the bandwidth and latency of accesses by a process to large amounts of data are limited by the performance of a given local memory system and are exacerbated when memory within other computing nodes is accessed.

The following detailed description refers to the accompanying drawings, which are now briefly described.
1 is a block diagram of one embodiment of a plurality of system on a chip (SOC), where a given SOC includes a plurality of memory controllers.
Figure 2 is a block diagram illustrating one embodiment of memory controllers and physical/logical arrangement on SOCs.
Figure 3 is a block diagram of one embodiment of a binary decision tree for determining which memory controller services a particular address.
Figure 4 is a block diagram illustrating one embodiment of a plurality of memory location configuration registers.
5 is a flow diagram illustrating the operation of one embodiment of SOCs during boot/power-up.
Figure 6 is a flow diagram illustrating the operation of one embodiment of SOCs for routing memory requests.
Figure 7 is a flow diagram illustrating the operation of one embodiment of a memory controller responding to a memory request.
Figure 8 is a flow chart illustrating the operation of one embodiment of monitoring system operation to determine memory folding.
Figure 9 is a flow diagram illustrating the operation of one embodiment of folding a memory slice.
Figure 10 is a flow diagram illustrating the operation of one embodiment of unfolding a memory slice.
11 is a flow chart illustrating one embodiment of a memory folding method.
Figure 12 is a flow chart illustrating one embodiment of a method for hashing a memory address.
Figure 13 is a flow diagram illustrating one embodiment of a method for forming a compressed pipe address.
Figure 14 is a block diagram of one embodiment of the system and various implementations of the system.
Figure 15 is a block diagram of a computer accessible storage medium.
Although the embodiments described in this disclosure are susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will be described in detail herein. However, the drawings and detailed description thereto are not intended to limit the embodiments to the specific form disclosed; on the contrary, the intent is to make all modifications, equivalents, and modifications within the spirit and scope of the appended claims. It should be understood that it is intended to cover alternatives. Headings used herein are for organizational purposes only and are not intended to be used to limit the scope of the description.

1 is a block diagram of one embodiment of a plurality of system-on-chip (SOC) 10 forming a system. The SOCs 10 may be instances of a common integrated circuit design, so one of the SOCs 10 is shown in greater detail. Other instances of SOC 10 may be similar. In the illustrated embodiment, SOC 10 includes a plurality of memory controllers 12A-12H, one or more processor clusters (P clusters) 14A, 14B, one or more graphics processing units (GPUs) 16A, 16B), one or more I/O clusters 18A, 18B, and a communication fabric comprising a western interconnect (IC) 20A and an eastern IC 20B. I/O clusters 18A, 18B, P clusters 14A, 14B, and GPUs 16A, 16B may be coupled to west IC 20A and east IC 20B. West IC 20A may be coupled to memory controllers 12A through 12D, and east IC 20B may be coupled to memory controllers 12E through 12H.

The system shown in FIG. 1 further includes a plurality of memory devices 28 coupled to memory controllers 12A through 12H. In the example of Figure 1, four memory devices 28 are coupled to each memory controller 12A through 12H. Other embodiments may have more or fewer memory devices 28 coupled to a given memory controller 12A-12H. Additionally, different memory controllers 12A-12H may have different numbers of memory devices 28. Memory devices 28 may vary in capacity and configuration, or may have consistent capacity and configuration (eg, banks, bank groups, row size, ranks, etc.). Each memory device 28 may be coupled to a respective memory controller 12A-12H through an independent channel in this implementation. Channels shared by two or more memory devices 28 may be supported in other embodiments. In one embodiment, memory devices 28 may be mounted on a corresponding SOC 10 in a chip-on-chip (CoC) or package-on-package (PoP) implementation. there is. In another embodiment, memory devices 28 may be packaged with SOC 10 in a multi-chip-module (MCM) implementation. In another embodiment, memory devices 28 may be mounted on one or more memory modules, such as single in-line memory modules (SIMMs), dual in-line memory modules (DIMMs), etc. In one embodiment, memory devices 28 may be dynamic random access memory (DRAM), such as synchronous DRAM (SDRAM) and, more specifically, double data rate (DDR) SDRAM. In one embodiment, memory devices 28 may be implemented with a low-power (LP) DDR SDRAM specification, also known as mobile DDR (mDDR) SDRAM.

In one embodiment, interconnects 20A, 20B are also coupled to an off-SOC interface to another instance of SOC 10, thereby connecting the system to more than one SOC (e.g., more than one semiconductor die, where SOC ( A given instance of 10) can be implemented on a single semiconductor die, but can be expanded to multiple instances (multiple instances can be combined to form a system). Accordingly, the system may be scalable to two or more semiconductor dies on which instances of SOC 10 are implemented. For example, two or more semiconductor dies can be configured as a single system where the presence of multiple semiconductor dies is transparent to software running on the single system. In one embodiment, delays in die-to-die communication can be minimized, so that die-to-die communication is typically an aspect of software transparency for multi-die systems. ) It does not cause significant additional latency compared to communication. In other embodiments, the communication fabric within SOC 10 may not have physically distinct interconnections 20A, 20B, but rather source hardware agents and memory controllers within the system (sending memory requests). There may be a full interconnect (eg, a full crossbar) between (12A to 12H). Such embodiments may still include the concept of logical interconnections 20A, 20B, in one embodiment, for hashing and routing purposes.

Memory controller 12A is shown in more detail in FIG. 1 and may include control circuitry 24 and various internal buffer(s) 26. Other memory controllers 12B through 12H may be similar. Control circuit 24 is coupled to internal buffers 26 and memory location configuration registers 22F (discussed below). In general, control circuitry 24 may be configured to control access to memory devices 28 to which memory controller 12A is coupled, such as controlling channels for memory devices 28, performing calibration, etc. This includes performing, ensuring accurate refresh, etc. Control circuitry 24 may also be configured to schedule memory requests to attempt to minimize latency, maximize memory bandwidth, etc. In one embodiment, memory controllers 12A through 12H may employ memory caches to reduce memory latency, and control circuitry 24 may access the memory cache for memory requests and detect hits in the memory cache. and misses, and evictions from the memory cache. In one embodiment, memory controllers 12A through 12H can manage coherency (e.g., a directory-based coherency scheme) for the memory attached thereto, and control circuitry 24 can manage coherency Can be configured to manage runtime. The channel to memory device 28 may include low-level communication circuitry (e.g., physical layer (PHY) circuitry) as well as physical connections to the device.

As illustrated in FIG. 1 , I/O clusters 18A, 18B, P clusters 14A, 14B, GPUs 16A, 16B, and memory controllers 12A through 12H have a memory location configuration ( MLC) registers (reference numbers 22A through 22H, 22J through 22N, and 22P). In some embodiments, west and east ICs 20A, 20B may also include memory location configuration registers. Because the system includes multiple memory controllers 12A through 12H (and possibly multiple sets of memory controllers in multiple instances of SOC 10), the address accessed by a memory request can be decoded ( For example, hashing) can be used to determine the memory controllers 12A through 12H, which in turn can determine the specific memory device 28 that is mapped to that address. That is, memory addresses can be defined within a memory address space that maps memory addresses to memory locations within memory devices. A given memory address within the memory address space uniquely identifies a memory location within one of the memory devices 28 coupled to one of the plurality of memory controllers 12A-12H. MLC registers 22A through 22H, 22J through 22N, and 22P can be programmed to describe a mapping of memory address bits as specified by MLC registers 22A through 22H, 22J through 22N, and 22P. Hashing may identify the memory controllers 12A through 12H, which in turn may identify the memory device 28 (and, in one embodiment, the bank group and/or bank within memory device 28) to which the memory request is directed. You can.

A given circuit may have more than one MLC register. For example, there may be an MLC register for each level of granularity in the hierarchy of levels of granularity to identify the memory controller 12A through 12H. The number of levels decoded by a given circuit may depend on the number of granularity levels the given circuit uses to determine how to route a memory request to the correct memory controller 12A through 12H, and in some cases the correct memory. One may rely on much lower granularity levels within controllers 12A through 12H. Memory controllers 12A through 12H may include MLC registers for each level of the hierarchy, at least up to the specific memory device 28. Generally, the granularity levels can be viewed as recursive powers of 2 of at least two of the plurality of memory controllers 12A through 12H. Therefore, although the MLC registers 22A through 22H, 22J through 22N, and 22P are assigned the same general reference number, the MLC registers 22A through 22H, 22J through 22N, and 22P may not all be the same set of registers. You can. However, instances of registers 22A through 22H, 22J through 22N, and 22P corresponding to the same granularity level may be identical and may be programmed consistently. Additional details are discussed further below.

Memory controllers 12A-12H may be physically distributed across the integrated circuit die on which SOC 10 is implemented. Accordingly, memory controllers within a system may be physically distributed across multiple integrated circuit dies, and physically distributed within an integrated circuit die. That is, memory controllers 12A through 12H may be distributed across the area of the semiconductor die on which SOC 10 is formed. For example, in Figure 1, the location of memory controllers 12A through 12H within SOC 10 may represent the physical locations of those memory controllers 12A through 12H within the SOC 10 die area. there is. Accordingly, determining which memory controller 12A through 12H a given memory request is mapped to (“targeted memory controller”) may be used to route the memory request through the communication fabric within SOC 10 to the targeted memory controller. . The communication fabric may include, for example, West IC 20A and East IC 20B, and may further include additional interconnections not shown in FIG. 1 . In other embodiments, memory controllers 12A-12H may not be physically distributed. Nonetheless, a hashing mechanism as described herein may be used to identify targeted memory controllers 12A through 12H.

I/O clusters 18A, 18B, P clusters 14A, 14B, and GPUs 16A, 16B use memory addresses within memory devices 28 via memory controllers 12A through 12H. These may be examples of hardware agents configured to access data. Other hardware agents may also be included. Generally, a hardware agent may be a hardware circuit that may be the source of a memory request (eg, a read or write request). The request is routed from the hardware agent to the targeted memory controller based on the contents of the MLC registers.

In one embodiment, memory addresses are used by memory controllers 12A through 12H (and corresponding memory controllers in other instances of SOC 10 included in the system) to distribute data within pages throughout the memory system. ) can be mapped across. Such an approach can improve bandwidth usage of the memory controller and communication fabric for applications that access most or all of the data within a page. That is, a given page within a memory address space may be partitioned into multiple blocks, and the multiple blocks of a given page may be distributed across multiple memory controllers within the system. A page may be an allocation unit of memory in a virtual memory system. That is, when memory is allocated to an application or other process/thread, the memory is allocated in pages. The virtual memory system generates translations from physical addresses within the memory address space and virtual addresses used by an application, which identifies locations within memory devices 28. Page sizes vary from embodiment to embodiment. For example, a 16 kilobyte (16 kB) page size may be used. Smaller or larger page sizes may be used (eg, 4 kB, 8 kB, 1 megabyte (MB), 4 MB, etc.). In some embodiments, multiple page sizes are supported simultaneously in the system. Typically, pages are aligned to page-size boundaries (e.g., a 16 kB page is allocated on 16 kB boundaries, so that the least significant 14 address bits form an offset within the page, and the remaining address bits identify the page. do).

The number of blocks into which a given page is divided may be related to the number of memory controllers and/or memory channels within the system. For example, the number of blocks may be equal to the number of memory controllers (or number of memory channels). In such an embodiment, when all data in a page is accessed, the same number of memory requests may be sent to each memory controller/memory channel. Other embodiments may have the number of blocks equal to a multiple of the number of memory controllers, or a fraction of the memory controllers (e.g., a fraction to a power of 2), allowing pages to be distributed across a subset of memory controllers.

In one embodiment, MLC registers may be programmed to map adjacent blocks of a page to memory controllers that are physically distant from each other within the SOC(s) 10 of the system. Accordingly, the access pattern in which successive blocks of a page are accessed can be distributed across the system, utilizing different parts of the communication fabric and interfering with each other in a minimal manner (or perhaps not at all). For example, memory requests for adjacent blocks may take different paths through the communication fabric and thus will not consume the same fabric resources (e.g., portions of interconnects 20A, 20B). That is, the paths may be at least partially non-overlapping. In some cases, the paths may be completely non-overlapping. Additional details regarding the distribution of memory accesses are provided below in relation to FIG. 2 . Maximizing the distribution of memory accesses can improve overall system performance by reducing overall latency and increasing bandwidth utilization. Additionally, flexibility in scheduling processes to processors can be achieved because similar performance can occur on any similar processor within any P cluster 14A, 14B.

MLC registers 22A through 22H, 22J through 22N, and 22P can independently specify the address bits that are hashed to select each level of granularity in the system for a given memory address. For example, the first level of granularity may select the semiconductor die to which memory requests are routed. A second level of granularity may select a slice, which may be a set of memory controllers (e.g., the top four memory controllers 12A, 12B and 12E, 12F may form a slice, and the bottom four memory controllers 12C, 12D and 12F, 12G) can form different slices). Other levels of granularity may include selecting a “side” (east or west in Figure 1), and a row within a slice. There may be additional levels of granularity within memory controllers 12A-12H, ultimately creating a selected memory device 28 (and possibly, in one embodiment, a bank group and bank within device 28). . Any number of granularity levels may be supported in various embodiments. For example, if more than two dies are involved, there may be one or more granularity levels coarser than the die level, from which groups of dies are selected.

Independent specification of address bits for each granularity level can provide significant flexibility in the system. Additionally, changes to the design of the SOC 10 itself can be managed by using different programming in the MLC registers, and thus hardware within the memory system and/or interconnect, resulting in different mappings of addresses to memory devices. There is no need to change to accept. Additionally, programmability in the MLC registers allows for the depopulation of memory devices 28 in a given product containing SOC(s) 10, thereby reducing the total number of memory devices 28 in that product. If full complement is not required, cost and power consumption can be reduced.

In one embodiment, each level of granularity is a binary decision: a binary 0 resulting from the hash selects one outcome at that level, and a binary 1 resulting from the hash selects the other outcome. Hashes can be any combinational logic operation on the input bits selected for levels by programming the MLC registers. In one embodiment, the hash may be an exclusive-OR reduction in which the address bits are exclusive-ORed with each other to produce a binary output. Other embodiments may generate a multi-bit output value to select between more than two results.

Internal buffers 26 within a given memory controller 12A-12H may be configured to store a significant number of memory requests. Internal buffers 26 include transaction tables that track the status of the various memory requests being processed at a given memory controller 12A through 12H, as well as the various pipeline stages through which requests may flow as they are processed. Can contain static buffers. The memory address accessed by a request may be a significant portion of the data describing the request and therefore a significant component of power consumption in storing the requests and moving them through various resources within a given memory controller 12A through 12H. It can be. In one embodiment, memory controllers 12A through 12H may be configured to drop an address bit from each set of address bits (corresponding to each granularity level) used to determine the targeted memory controller. You can. In one embodiment, the remaining address bits, along with the fact that the request is to a targeted memory controller, can be used to recover dropped address bits if necessary. In some embodiments, the dropped bit may be an address bit that is not included in any other hash corresponding to another granularity level. Exclusion of dropped bits from different levels can allow recovery of dropped bits in parallel since the operations are independent. If a given dropped bit is not excluded from other levels, it can be recovered first and then used to recover other dropped bits. Therefore, exclusion may be optimal for recovery. Other embodiments may not require recovery of the original address, so the dropped bits do not need to be unique in each hash, or may recover the bits in a serial manner if exclusion is not implemented. The remaining address bits (without dropped bits) can form a compressed pipe address that can be used internally to the memory controller for processing. Dropped address bits are not needed because the amount of memory in memory devices 28 coupled to a given memory controller 12A through 12H can be uniquely addressed using the compressed pipe address. . MLC registers 22A through 22H, 22J through 22N, and 22P, in one embodiment, may include programmable registers to identify drop bits.

SOC 10 of FIG. 1 includes a certain number of memory controllers 12A through 12H, P clusters 14A, 14B, GPUs 16A, 16B, and I/O clusters 18A, 18B. do. In general, various embodiments may include any number of memory controllers 12A through 12H, P clusters 14A, 14B, GPUs 16A, 16B, and I/O clusters 18A, as desired. 18B) may be included. As mentioned above, P clusters 14A, 14B, GPUs 16A, 16B, and I/O clusters 18A, 18B generally implement the operations described herein for each component. It includes hardware circuits configured to do so. Similarly, memory controllers 12A-12H generally include hardware circuits (memory controller circuits) to implement the operations described herein for each component. Interconnects 20A, 20B and other communication fabrics generally include circuits for transporting communications (eg, memory requests) between other components. Interconnects 20A, 20B may include point-to-point interfaces, shared bus interfaces, and/or hierarchies of one or both interfaces. The fabric may be circuit switched, packet switched, etc.

Figure 2 is a block diagram illustrating one embodiment of a plurality of memory controllers and physical/logical arrangement on SOC die(s), for one embodiment. Memory controllers 12A-12H are illustrated for two instances of SOC 10, illustrated as Die 0 and Die 1 in Figure 2 (e.g., separated by short dashed line 30). Die 0 may be the portion illustrated above dashed line 30 and die 1 may be the portion below dashed line 30. Memory controllers 12A through 12H on a given die may be divided into slices based on the physical location of memory controllers 12A through 12H. For example, in Figure 2, slice 0 may include memory controllers 12A, 12B and 12E, 12F, physically located on either half of die 0 or die 1. Slice 1 may include memory controllers 12C, 12D and 12G, 12H, physically located on die 0 or the other half of die 1. The slice on the die is delimited by dashed lines 32 in FIG. 2 . Within slices, memory controllers 12A-12H may be partitioned into rows based on physical location within the slice. For example, slice 0 of die 0 is shown in Figure 1 to include two rows, with memory controllers 12A, 12E above dashed line 34 in row 0 covering one half of the area occupied by slice 0. is physically located on the Memory controllers 12B, 12F of row 1 of slice 1 are physically located on the other half of the area occupied by slice 0, below dotted line 34. Other slices can be similarly divided into rows. Additionally, a given memory controller 12A-12H may be reachable via either the western interconnect 20A or the eastern interconnect 20B.

Accordingly, a memory address may be hashed at multiple levels of granularity to identify a given memory controller 12A-12H on a given die 0 or 1 to which the memory address is mapped. In this embodiment, the levels may include die level, slice level, row level, and side level (east or west). The die level may specify which of a plurality of integrated circuit dies contains a given memory controller. The slice level may specify which of a plurality of slices within a die contains a given memory controller, wherein the plurality of memory controllers on the die are logically partitioned into a plurality of slices based on their physical location on the given integrated circuit die. , a given slice includes at least two memory controllers among a plurality of memory controllers in the die. Within a given slice, memory controllers may be logically partitioned into multiple rows based on their physical location on the die, and more specifically, within a given slice. The row level may specify which of a plurality of rows contains a given memory controller. A row may be divided into multiple sides, again based on physical location within the die, and more specifically within a given row. The side level can specify which side of a given row contains a given memory controller.

Other embodiments may include more or fewer levels, based on the number of memory controllers 12A-12H, number of dies, etc. For example, an embodiment including more than two dies may include multiple granularity levels for selecting a die (e.g., die groups group pairs of SOCs 10 in a four die implementation can be used to do so, and the die level can be selected among the dies within the selected pair). Similarly, an implementation that includes four memory controllers per die instead of eight could eliminate either the slice or row levels. Implementations that include a single die rather than multiple dies can eliminate die levels.

At each of the granularity levels, a binary decision is made based on a hash of a subset of address bits to select one or the other level. Therefore, a hash can operate logically on the address bits to produce a binary output (one bit, 0 or 1). Any logical function can be used in the hash. In one embodiment, XOR reduction may be used, for example, where the hash produces a result by exclusive-oring (XORing) a subset of the address bits together. XOR reduction can also provide hashing reversibility. Reversibility can allow recovery of dropped bits, but XORing the binary result with non-dropped address bits (one dropped bit per level). In particular, in one embodiment, dropped address bits may be excluded from subsets of address bits used for other levels. Other bits in the hash may be shared between hashes, but the bit to be dropped cannot. Although XOR reduction is used in this embodiment, other embodiments may implement any logically reversible Boolean operation as a hash.

FIG. 3 is a block diagram of one embodiment of a binary decision tree for determining which memory controller 12A through 12H (and die) serves a particular memory address (i.e., the memory controller to which a particular memory address is mapped). The decision tree may include determining a die (see 40), a slice on the die (see 42), a row within a slice (see 44), and a side within a row (see 46). In one embodiment, there may be additional binary decisions within the memory controller to guide processing of a memory request. For example, the embodiment of FIG. 3 may include a planar level 48 and a pipe level 50 . Internal granularity levels may map a memory request to a specific memory device 28 that stores the data affected by the memory request. That is, the most precise level of granularity may be the level that maps to a specific memory device 28. Memory planes are independent, allowing multiple memory requests to proceed in parallel. Additionally, various structures included in the memory controller (e.g., a memory cache that caches previously accessed data in memory devices 28, coherency control hardware such as redundant tags or directories, various buffers) and queues, etc.) can be partitioned between planes, so memory structures can be smaller and easier to design, capable of meeting timing at a given operating frequency, etc. Accordingly, performance can be increased through both higher achievable clock frequencies and parallel processing for hardware structures of a given size. In other embodiments, there may also be additional internal levels of granularity within the memory controller.

The binary decision tree illustrated in Figure 3 shows that decisions at die level 40, slice level 42, row level 44, side level 46, planar level 48, and pipe 50 are made in series. I don't mean to imply it. Logic for performing decisions may operate in parallel, selecting sets of address bits and performing hashes to generate resulting binary decisions.

Returning to Figure 2, the programmability of the address mapping for memory controllers 12A through 12H and dies 0 and 1 can provide distribution of consecutive addresses between physically distant memory controllers 12A through 12H. there is. That is, if a source is accessing consecutive addresses of a page of memory, for example, memory requests may be distributed across different memory controllers (at some address granularity). For example, consecutive cache blocks (e.g., aligned 64-byte or 128-byte blocks) may be mapped to different memory controllers 12A through 12H. Lower granularity mappings may also be used (eg, 256-byte, 512-byte, or 1-kilobyte blocks may be mapped to different memory controllers). That is, multiple consecutive memory addresses accessing data in the same block may be routed to the same memory controller, and then the next multiple consecutive memory addresses may be routed to a different memory controller.

Mapping contiguous blocks to physically distributed memory controllers 12A through 12H may have performance benefits. For example, because memory controllers 12A through 12H are independent of each other, the bandwidth available across the set of memory controllers 12A through 12H can be more fully utilized when an entire page is accessed. Additionally, in some embodiments, the root of memory requests within the communication fabric may be partially non-overlapping or completely non-overlapping. That is, at least one segment of the communication fabric that is part of the route for one memory request may not be part of the route for another memory request, and vice versa for partially non-overlapping routes. Completely non-overlapping routes may use distinct and completely separate parts of the fabric (eg, no segments can be identical). Accordingly, traffic in the communication fabric can be spread out and not interfere with each other as much as the traffic might otherwise interfere with.

Accordingly, the MLC registers (22A to 22H, 22J to 22N, 22P), when the first address and the second address are adjacent addresses at the second granularity level, allow the circuitry to receive the first memory request with the first address in multiple numbers. It may be programmable with data that causes routing to a first memory controller among the memory controllers and routes a second memory request with a second address to a second memory controller that is physically distant from the first memory controller among the plurality of memory controllers. there is. In one embodiment, the first route of the first memory request through the communications fabric and the second route of the second memory request through the communications fabric are completely non-overlapping. In other cases, the first and second routes may be partially non-overlapping. One or more registers cause the communication fabric to route multiple memory requests to different memory controllers of the plurality of memory controllers in a pattern that distributes the multiple memory requests for consecutive addresses across physically distant memory controllers. It can be programmed with data.

For example, in Figure 2, memory controllers 12A through 12H on Die 0 and Die 1 are labeled MC 0 through MC 15. Starting with address 0 in the page, successive addresses at the level of granularity defined in the programming of the MLC registers (22A through 22H, 22J through 22N, 22P) are first MC0 (memory controller 12A in die 0), then MC1. (Memory Controller (12G) in Die 1), MC2 (Memory Controller (12D) in Die 1), MC3 (Memory Controller (12F) in Die 0), MC4 (Memory Controller (12A) in Die 1), MC5 (Memory Controller (12A) in Die 1) MC6 (memory controller (12D) in die 0), MC7 (memory controller (12F) in die 1), MC8 (memory controller (12C) in die 0), MC9 (memory controller (12D) in die 1) Memory Controller (12E)), MC10 (Memory Controller (12B) in Die 1), MC11 (Memory Controller (12H) in Die 0), MC12 (Memory Controller (12C) in Die 1), MC13 (Memory Controller (12B) in Die 0) (12E)), MC14 (memory controller 12B in die 0), and then MC15 (memory controller 12H in die 1). If the second granularity level is less than 1/N ^th of the page size, where N is the number of memory controllers in the system (e.g., 16 in this embodiment), the next successive access after MC15 may return to MC0. there is. More random access patterns may result in memory requests being routed to physically nearby memory controllers, while more common regular access patterns (which involve strides where one or more memory controllers are skipped in the above sequence) even if) can be well distributed in the system.

4 is a block diagram illustrating one embodiment of a plurality of memory location configuration registers 60 and 62. In general, registers 60 in a given hardware agent may be programmable with data identifying which address bits are included in the hash at one or more of a plurality of levels of granularity. In the illustrated embodiment, registers 60 include die registers, slice registers, row registers, side registers, plane registers, and pipe registers corresponding to the previously-described levels, as well as memory devices that store data ( 28) may include a bank group and a bank register defining a bank and a bank group (BankG) in (for embodiments where DRAM memory devices have both bank groups and banks). Note that although separate registers 60 are shown for each granularity level in Figure 4, other embodiments may combine two or more granularity levels as fields within a single register, as desired.

The die resistor is shown in an exploded view for one embodiment; other resistors 60 may be similar. In the illustrated embodiment, the die resistor may include an invert field 66 and a mask field 68. The inversion field 66 may be a bit with a set state indicating an inversion and a cleared state indicating no inversion (or vice versa, or multiple bit values may be used). The mask field 68 may be a field of bits corresponding to respective address bits. For that level of granularity, a set state in the mask bit may indicate that the respective address bit is included in the hash, and a cleared state may indicate that the respective address bit is excluded from the hash (or vice versa).

The invert field 66 may be used to specify that the result of the hash of the selected address bits should be inverted. Inversion may allow additional flexibility in the memory controller's decisions. For example, programming a mask of all zeros results in binary zeros at that granularity level for any address, forcing the decision in the same direction every time. If binary 1's are desired at a given granularity level for any address, the mask can be programmed to all zeros and the invert bit can be set.

MLC registers 22A through 22H, 22J through 22N, and 22P each use a subset or all of registers 60, depending on the hardware agent and the levels of granularity used to route memory requests by the hardware agent. It can be included. In general, a given hardware agent can employ all levels of granularity, if desired, up to the bank level (curly braces labeled “bank” in Figure 4). However, some hardware agents do not need to implement such many levels of granularity. For example, the hardware agent employs die, slice, row, and side granularity levels to route memory requests to targeted memory controllers 12A through 12H (braces labeled “MC” in Figure 4) on the targeted die. It can be delivered. Memory controllers 12A through 12H may process the remaining hashing levels. Another hardware agent may have two routes to a given memory controller 12A through 12H, one for each plane. Accordingly, such a hardware agent may employ die, slice, row, side, and plane registers (braces labeled “plane” in Figure 4). Another hardware agent may include die, slice, row, side, and plane granularity levels, as well as a pipe level (braces labeled “channel” in Figure 4) that identifies the desired channel. Accordingly, a first hardware agent may be programmable to a first number of a plurality of granularity levels, and a second hardware agent may be programmable to a second number of the plurality of granularity levels, where the second number is a second number of the plurality of granularity levels. 1 It is different from the number. In other embodiments, bank group, bank, and other intradevice granularity levels may be specified differently than other granularity levels, and thus may be separately-defined registers that are not included in registers 60. there is. In still other embodiments, bank group, bank, and other intra-device granularity levels may be fixed in hardware.

Another set of registers that may be included in some sets of MLC registers 22A through 22H, 22J through 22N, and 22P are drop registers 62 shown in FIG. 4. In particular, in one embodiment, drop registers 62 may be included in MLC registers 22F through 22H and 22J through 22N in memory controllers 12A through 12H. Drop registers 62 may include a register for each granularity level, and at least one address bit in the subset of address bits corresponding to that granularity level to be dropped by the targeted memory controller 12A through 12H. It may be programmable to identify. The specified bit is a bit included in the hash of that granularity level and is one of the bits specified in the corresponding register 60. In one embodiment, the dropped address bit may be included exclusively in the hash for that granularity level (eg, the dropped address bit is not specified at any other granularity level in registers 60). Other bits included in a given hash may be shared across different granularity levels, but a dropped bit may be unique to a given granularity level. Drop registers 62 can be programmed in any way to indicate the address bit to be dropped (eg, the bit number can be specified as a hexadecimal number, or a bit mask can be used as shown in Figure 4). The bit mask may include a bit for each address bit (or each selectable address bit if some address bits are not eligible for drop). The bit mask may be a “one hot” mask, with only one set bit, which may indicate the selected drop bit. In other embodiments, a single bit mask in single drop register 62 may specify a drop bit for each granularity level and thus may not be a single hot mask.

The memory controller can be programmed through drop registers 62 to specify drop bits. The memory controller (and more specifically, the control circuit 24) determines an internal address for each memory request for internal use by the memory controller in internal buffers 26 (the "compressed pipe address" mentioned above, or more may be configured to simply generate a "compressed address") and address the memory device 28. A compressed pipe address may be created by dropping some or all of the specified address bits and shifting the remaining address bits together. You can.

As previously mentioned, multiple internal buffers with copies of an address can save power by eliminating unnecessary address bits. Additionally, using a reversible hash function, dropped bits can be recovered to recover the entire address. The presence of a memory request within a given memory controller 12A through 12H provides the result of a hash at a given granularity level, and hashing the result with other address bits included in that granularity level produces a dropped address bit. Recovery of the entire address may be useful if it is needed for a response to a request, snoops for consistency reasons, etc.

Referring now to Figure 5, a flow diagram illustrating the operation of one embodiment of SOCs during boot/power-up is shown. For example, the operations illustrated in FIG. 5 may be performed by instructions executed by a processor (e.g., low level boot code executed to initialize the system for execution of an operating system). Alternatively, all or part of the operations shown in Figure 5 may be performed by hardware circuitry during boot. Although blocks are shown in a specific order for ease of understanding, other orders may be used. Blocks may be performed in parallel in combinational logic within SOCs 10. Blocks, combinations of blocks, and/or the entire flow diagram may be pipelined over multiple clock cycles.

The boot code may be configured to support a SOC configuration (e.g., one or more chips containing instances of SOC 10, a partial SOC containing fewer memory controllers 12A through 12H, or one of a plurality of SOC designs supported by the system). SOC design differences, memory devices 28 coupled to each memory controller 12A through 12H, etc.) may be identified (block 70). Identifying a configuration generally determines the number of destinations for memory requests (e.g., the number of memory controllers 12A through 12H in the system, the number of planes within each memory controller 12A through 12H, the number of planes to be enabled during use). number of memory controllers 12A to 12H, etc.). A given memory controller 12A through 12H may be unavailable during use, such as if memory devices 28 are not populated in a given memory controller 12A through 12H or if there is a hardware failure in memory devices 28. It may be impossible. In other cases, a given memory controller 12A-12H may be unavailable in certain test modes or diagnostic modes. Identifying the configuration may also include determining the total amount of available memory (e.g., the number of memory devices 28 coupled to each memory controller 12A through 12H and the capacity of the memory devices 28). You can.

These decisions may affect the size of contiguous blocks within a page to be mapped to each memory controller 12A through 12H, which may affect the memory controllers 12A through 12H (and, when more than one instance is provided, the SOC (10) represents a trade-off between spreading memory requests within a page among instances and the efficiencies that can be achieved from grouping requests to the same addresses. Accordingly, the boot code can determine the block size to be mapped to each memory controller 12A through 12H (block 72). In other modes, a linear mapping of addresses to memory controllers 12A through 12H may be used (e.g., mapping all of the memory devices 28 within memory controllers 12A through 12H to adjacent blocks of addresses in the memory address space). ), a hybrid of interleaving at one or more granularity levels and linear at other granularity levels can be used. Boot code may determine how to program MLC registers 22A through 22H, 22J through 22N, 22P to provide a desired mapping of addresses to memory controllers 12A through 12H (block 74). For example, mask registers 60 can be programmed to select address bits at each granularity level, and drop bit registers 62 can be programmed to select a drop bit for each granularity level.

6 is a flow diagram illustrating the operation of various SOC components to determine the route for a memory request from a source component to an identified memory controller 12A-12H for the memory request. Although blocks are shown in a specific order for ease of understanding, other orders may be used. Blocks may be performed in parallel in combinational logic within SOCs 10. Blocks, combinations of blocks, and/or the entire flow diagram may be pipelined over multiple clock cycles.

The component may apply registers 60 to the address of the memory request to determine various levels of granularity, such as die, slice, row, side, etc. (block 76). Based on the results at the granularity levels, the component may route the memory request through the fabric to the identified memory controllers 12A through 12H (block 78).

7 is a flow diagram illustrating the operation of one embodiment of memory controllers 12A-12H in response to memory requests. Although blocks are shown in a specific order for ease of understanding, other orders may be used. Blocks may be performed in parallel in combinational logic within SOCs 10. Blocks, combinations of blocks, and/or the entire flow diagram may be pipelined over multiple clock cycles.

Memory controllers 12A-12H may use plane, pipe, bank group, and bank mask registers 60 to identify the plane, pipe, bank group, and bank for the memory request (block 80). . For example, memory controllers 12A through 12H may logically AND the mask from the corresponding register 60 with the address, logically combine the bits (e.g., XOR decrement), and invert when indicated. Memory controllers 12A through 12H use drop masks from drop registers 62 to select the memory specified by each granularity level (e.g., die, slice, row, side, plane, pipe, bank group, and bank). The address bits can be dropped and the remaining address bits can be shifted together to form a compressed pipe address (block 82). For example, memory controllers 12A through 12H can mask an address with the inverse logical AND of the drop masks and shift the remaining bits together. Alternatively, memory controllers 12A through 12H may simply shift the address bits together, naturally dropping the identified bits. Memory controllers 12A-12H may perform (block 84) a specified memory request (e.g., read or write) (e.g., with read data, or with write completion if the record is not a posted record). ) can respond to the source. If the full address is needed for response or other reasons during processing, the full address is stored in the compressed pipe address, the contents of registers 60 for each level, and the memory controller 12A through 12H that received the memory request. A recovery can be made from the known results for each corresponding level (block 86).

The number of memory controllers 12A through 12H in the system, and the number of memory devices 28 coupled to memory controllers 12A through 12H can be a significant source of power consumption in the system. At certain points during operation, a relatively small amount of memory may be in active use, and power may be conserved by disabling one or more slices of the memory controllers/memory devices when accesses to these slices are infrequent. You can. Disabling a slice may include any mechanism to reduce power consumption in the slice and make the slice unavailable until the slice is re-enabled. In one embodiment, data may be retained by memory devices 28 while the slice is disabled. Accordingly, the power supply to memory devices 28 may remain active, but memory devices 28 may be placed in a lower power mode (e.g., DRAM devices may be placed in a lower power mode to allow the devices to retain data). may be placed in self-refresh mode, generating refresh operations internally for the device but not accessible from the SOC 10 until self-refresh mode is terminated). Memory controller(s) 12A-12H within a slice may also be in a low-power mode (eg, clock gated). Memory controller(s) 12A through 12H within a slice may be power gated, so they can be powered on and reconfigured when enabling the slice and after disabling it.

In one embodiment, software (e.g., part of an operating system) can monitor activity within the system to determine whether a slice or slices are disabled. The software can also monitor attempts to access data within the slice during the disabled time and re-enable the slice as desired. Additionally, in one embodiment, the monitor software can detect pages of data within a slice that are accessed at a rate greater than a specified rate before disabling the slice, and copy those pages to another slice that will not be disabled. (Remapping virtual-to-physical address translations for those pages). Accordingly, some pages within a slice may remain available and can be accessed while the slice is disabled. The process of reallocating the pages being accessed and disabling the slice is referred to herein as “folding” the slice. Re-enabling a folded slice may be referred to as “unfolding” the slice, and the process of re-enabling involves spreading the pages across the available slices by remapping previously reallocated pages. (And, if the data in the reallocated pages is modified during the time the slice was folded, copying the data to the reallocated physical page).

Figure 8 is a flow diagram illustrating the operation of one embodiment of monitoring system operation to determine whether to fold or unfold memory. Although blocks are shown in a specific order for ease of understanding, other orders may be used. One or more code sequences (“code”) comprising a plurality of instructions executed by one or more processors on SOC(s) 10 may result in operations including the operations as indicated below. For example, memory monitor and fold/unfold code, when executed by processors on SOC(s) 10, may cause a system comprising SOCs to perform operations including those shown in FIG. 8. It can contain commands.

The memory monitor and fold/unfold code may monitor conditions within the system to identify opportunities to fold a slice or activity that indicates a folded slice will be unfolded (block 90). Activity that can be monitored may include, for example, access rates for various pages included in a given slice. If the pages within a given slice are not accessed at a rate above the threshold rate (or a significant number of pages are not accessed at a rate above the threshold rate), then the given slice may be a candidate for folding because the slice is often idle. Power states within processors within SOCs may be another factor monitored by the memory monitor and fold/unfold code, since processors in lower power states may access memory less frequently. In particular, processors that are in sleep states may not access pages of memory. Consumed bandwidth in the communication fabric within the SOC(s) 10 may be monitored. Other system factors may also be monitored. For example, the memory may fold due to the system detecting that the battery providing power is reaching a low state of charge. Another factor may be a change in power source, such as when the system was connected to a continuous, virtually unlimited power source (e.g., a wall outlet) and then unplugged and is now relying on battery power. Other factors may be system temperature overload, power supply overload, etc. Folding memory can reduce thermal or electrical load. Any set of factors indicative of the level of activity in the system may be monitored in various embodiments.

If activity indicates that one or more memory slices can be folded without significant performance impact (decision block 92, “yes” leg), the memory monitor and fold/unfold code will cause the folding of at least one slice. May commence (block 94). If activity indicates that demand for memory may be (or may soon be) increasing (decision block 96, “yes” leg), the memory monitor and fold/unfold code will May commence (block 98).

In one embodiment, folding of slices may be gradual and occur in stages. Figure 9 is a flow diagram illustrating one embodiment of progressive folding of a slice. Although blocks are shown in a specific order for ease of understanding, other orders may be used. Code executed by one or more processors on SOC(s) 10 may cause operations including the operations shown below.

The folding process may begin with determining which slices to fold (block 100). A slice may be selected by determining that the slice is the least frequently accessed among the slices, or among the least frequently accessed. A slice may be selected randomly (in one embodiment, not including slices that may be designated to remain active). A slice may be selected based on the lack of wired and/or copy-on-write pages (discussed below) in the slice, or a slice may have fewer wired and/or Or you can have copies of pages on write. A slice may be selected based on its relative independence from other folded slices (eg, physical distance, lack of shared segments in the communication fabric with other folded slices, etc.). Any factor or factors may be used to determine the slice. Slices can be marked by folding. In one embodiment, the folding process may disable slices with powers of 2, matching a binary decision tree for hashing. At least one slice may be designated as non-foldable and may be kept active to ensure that data is accessible in the memory system.

Initiating a fold may include suppressing new memory allocations for physical pages within the folding slice. Therefore, the memory monitor and fold/unfold code communicate with the virtual memory page allocator code, which allocates physical pages for virtual pages not yet mapped to memory, causing the virtual memory page allocator to allocate physical pages within the slice. can be stopped (block 102). Deactivation/disable can also potentially wait for wired pages within a slice to become unwired. Wired pages may be pages that are not allowed to be paged out by the virtual memory system. For example, pages of kernel code and pages of related data structures may be wired. When a copy-on-write page is allocated, it may be allocated to a slice that should remain active and therefore not allocated to a folding slice. Copy-on-write pages allow independent code sequences (e.g., processes, or threads within a process or processes) to share pages as long as none of the independent code sequences write the pages. can be used When an independent code sequence creates a write, the write may cause the virtual memory page allocator to allocate a new page and copy data to the newly allocated page.

Accordingly, the virtual memory page allocator can recognize which physical pages are mapped to which slices. In one embodiment, when folding is used, a linear mapping of addresses to memory may be employed instead of spreading blocks of each page across different memory controllers/memory. Alternatively, the mapping of addresses may be contiguous in a given slice, but pages may be spread among memory controllers/memory channels within a slice. In one particular embodiment, the address space may be mapped to each slice as a single contiguous block (e.g., one slice may be mapped to addresses 0 through slice_size-1 and the other slice may be mapped to addresses slice_size through 2). may be mapped to *slice_size-1, etc.). Other mechanisms may use interleaving between page boundaries, mapping pages to a limited number of slices that can be folded/unfolded as a unit, etc.

During the transition period when a slice is being folded, the pages within the selected (folding) slice may be tracked over a period of time to determine which pages are actively accessed (block 104). For example, access bits in page table translations can be used to track which pages are being accessed (checking the access bits periodically and clearing them when checked so new accesses can be detected). Pages that are active and found to be dirty (data has been modified since being loaded into memory) can be moved to a slice where they will remain active. That is, pages may be remapped to a different slice by the virtual memory page allocator (block 106). Pages that are found to be active but clean (not modified after initial load into memory) may optionally be remapped to a different slice (block 108). If active but clean pages are not remapped, access to pages after the slice has been folded may cause the slice to be re-enabled/activated, thus limiting the power savings that can be achieved. Therefore, the general intent may be for no actively-accessed pages to remain in a disabled/folded slice.

Once the foregoing is complete, the memory devices 28 (e.g., DRAMs) within the slice can be actively placed into self-refresh (block 110). Alternatively, since accesses are not occurring over time, the memory devices 28 naturally fall into self-refresh, relying on power management mechanisms built into the memory controllers 12A through 12H hardware to refresh themselves. -Can cause transition to refresh. Other types of memory devices may be actively placed in a low-power mode (or allowed to fall naturally) depending on the definition of those devices. Optionally, memory controllers 12A through 12H within the slice may be reduced to a lower power state due to lack of traffic, but may continue to listen for and respond to memory requests as they occur (block 112) ).

In one embodiment, a hard fold may be applied as a more aggressive mode in addition to the current fold if there is sufficiently high confidence that data within the folded slice is not required. That is, the memory devices 28 may actually be powered off if there is no access to the folded slice for an extended period of time.

Unfolding (re-enabling or activating) a slice can be gradual or rapid. Progressive unfolding can occur when the amount of active memory or bandwidth required by running applications is increasing and is approaching a threshold where currently active slices will not be able to meet the demand, limiting performance. Rapid unfolding can occur when a large memory allocation or a significant increase in bandwidth demand occurs (e.g., when a display turns on, a new application starts, or a user unlocks the system or otherwise interacts with the system by pressing a button or other input device). participating in a system, acting, etc.).

Figure 10 is a flow diagram illustrating one embodiment of unfolding a memory slice. Although blocks are shown in a specific order for ease of understanding, other orders may be used. Code executed by one or more processors on SOC(s) 10 may cause operations including the operations shown below.

A slice may be selected to unfold (block 120), or a number of slices may be selected, such as a power of 2 number of slices, as discussed above. Any mechanism for selecting the slice/slices may be used. For example, if a memory access to a folded slice occurs, that slice may be selected. Slices may be selected randomly. A slice may be selected based on its relative independence from other non-folded slices (eg, physical distance, lack of shared segments in the communication fabric with other non-folded slices, etc.). Any argument or combination of factors can be used to select a slice to unfold.

The power state of the memory controller(s) 12A through 12H within the unfolding slice can optionally be increased and/or the DRAMs can be self-refreshed (or, for other types of memory devices 28, in another low-power mode). ) may be activated to terminate (block 122). Alternatively, memory controllers 12A through 12H and memory devices 28 may naturally transition to higher performance/power states in response to the arrival of memory requests when physical pages within an unfolding memory slice arrive. there is. The memory monitor and fold/unfold code may notify the virtual memory page allocator that physical page allocations within the selected memory slice are available for allocation (block 124). Over time, the virtual memory page allocator may allocate pages within the selected memory slice to newly requested pages (block 126). Alternatively or in addition to allocating newly requested pages, the virtual memory page allocator may reallocate previously allocated pages from the selected memory slice to the selected memory slice. In another embodiment, a virtual memory page allocator can quickly reallocate pages to selected slices.

Slices may be defined as previously described with respect to Figure 2 (eg, slices may be coarser grained than rows). In other embodiments, for purposes of memory folding, a slice may be of any size up to a single memory channel (e.g., single memory device 28). Other embodiments may define a slice as one or more memory controllers 12A through 12H. Generally, a slice is a physical memory resource into which a plurality of pages are mapped. The mapping may be determined according to programming of MLC registers 22A through 22H, 22J through 22N, and 22P, in one embodiment. In other embodiments, the mapping may be fixed in hardware or may be otherwise programmable.

In one embodiment, the choice of slice size may be based, in part, on the data capacity and bandwidth used by low-power use cases of interest in the system. For example, the slice size can be selected so that a single slice can sustain the system's primary display and have enough memory capacity to sustain the operating system and a small number of background applications. Use cases may include, for example, watching a movie, playing music, fetching email while the screen saver is on, or downloading updates in the background.

Figure 11 is a flow diagram illustrating one embodiment of a method for folding a memory slice (e.g., to disable or deactivate a slice). Although blocks are shown in a specific order for ease of understanding, other orders may be used. Code executed by one or more processors on SOC(s) 10 may cause operations including the operations shown below.

The method may include detecting whether a first memory slice of a plurality of memory slices within the memory system should be disabled (decision block 130). If the detection indicates that the first memory slice should not be disabled (decision block 130, “no” leg), the method may be completed. If the detection indicates that the first memory slice should be disabled, the method may continue (decision block 130, “Yes” leg). Based on detecting that the first memory slice should be disabled, the method may include copying a subset of physical pages within the first memory slice to another of the plurality of memory slices. Data in a subset of physical pages may be accessed above a threshold rate (block 132). The method may include remapping virtual addresses corresponding to a subset of physical pages to another memory slice based on detecting that the first memory slice should be disabled (block 134). The method may also include disabling the first memory slice based on detecting that the first memory slice should be disabled (block 136). In one embodiment, disabling the first memory slice may include actively placing one or more dynamic access memories (DRAMs) within the first memory slice in a self-refresh mode. In another embodiment, disabling the first memory slice may include allowing one or more dynamic access memories (DRAMs) within the first memory slice to transition to a self-refresh mode due to lack of access. . In one embodiment, the memory system includes a plurality of memory controllers, and the physical memory resource includes at least one of the plurality of memory controllers. In another embodiment, a memory system includes a plurality of memory channels, and a given dynamic random access memory (DRAM) is coupled to one of the plurality of memory channels. A given memory slice includes at least one of a plurality of memory channels. For example, in one embodiment, a given memory slice is one memory channel of a plurality of memory channels.

In one embodiment, determining that the first memory slice should be disabled includes detecting that the access rate for the first memory slice is below a first threshold; and identifying a subset of physical pages that are accessed more frequently than a second threshold. In one embodiment, the method disables assigning a plurality of physical pages corresponding to a first memory slice to virtual addresses in a memory allocator based on detecting that the access rate is below a first threshold. may additionally be included. The method may further include disabling allocation of the plurality of physical pages followed by performing identification. In one embodiment, copying includes copying data from one or more physical pages of the subset containing the modified data to another memory slice in the memory system. In some embodiments, copying further includes copying data from one or more physical pages followed by copying data from remaining physical pages in the subset.

In accordance with the foregoing, a system may include one or more memory controllers coupled to one or more memory devices forming a memory system, wherein the memory system includes a plurality of memory slices, one of the plurality of memory slices having a given memory A slice is a physical memory resource to which a plurality of physical pages are mapped. The system includes one or more processors; and a non-transitory computer-readable storage medium storing a plurality of instructions that, when executed by one or more processors, cause the system to perform operations including methods as highlighted above. A non-transitory computer-readable storage medium is also an embodiment.

12 illustrates a method for hashing an address to route a memory request for an address to a targeted memory controller and, in some cases, to a targeted memory device and/or a bank group and/or bank within a memory device. This is a flowchart illustrating one embodiment of. Although blocks are shown in a specific order for ease of understanding, other orders may be used. Various components of SOC 10, such as source hardware agents, communication fabric components, and/or memory controller components, may be configured to perform part or all of the method.

The method may include generating a memory request having a first address in a memory address space that is mapped to memory devices in a system having a plurality of memory controllers physically distributed across one or more integrated circuit die (block ( 140)). In one embodiment, a given memory address within the memory address space uniquely identifies a memory location within one of memory devices coupled to one of a plurality of memory controllers, and a given page within the memory address space is partitioned into a plurality of blocks. , and multiple blocks of a given page are distributed across multiple memory controllers. The method may further include directing a memory request to a first memory controller of the plurality of memory controllers by hashing independently-specified sets of address bits from the first address, wherein the independently-specified sets of address bits are The specified sets locate the first memory controller at a plurality of granularity levels (block 142). The method may further include routing the memory request to the first memory controller based on hashing (block 144).

In one embodiment, the one or more integrated circuit dies are a plurality of integrated circuit dies; The plurality of granularity levels include a die level; The die level specifies which of the plurality of integrated circuit dies contains the first memory controller. In one embodiment, the plurality of memory controllers on a given integrated circuit die are logically partitioned into a plurality of slices based on their physical locations on the given integrated circuit die; At least two of the plurality of memory controllers are included in a given slice of the plurality of slices; The plurality of granularity levels include a slice level; The slice level specifies which of the plurality of slices contains the first memory controller. In one embodiment, the at least two memory controllers within a given slice are logically partitioned into a plurality of rows based on their physical location on a given integrated circuit die; The plurality of granularity levels include a row level; The row level specifies which of the plurality of rows contains the first memory controller. In one embodiment, the plurality of rows includes a plurality of sides based on physical location on a given integrated circuit die; The plurality of granularity levels include a side level; The side level specifies which side of a given row of the plurality of rows contains the first memory controller. In one embodiment, a given hardware agent of a plurality of hardware agents generating memory requests includes one or more registers, and the method identifies which address bits are included in the hash at one or more of the plurality of granularity levels. It further includes programming one or more registers with data. In one embodiment, a first hardware agent of the plurality of hardware agents is programmable for a first number of the plurality of granularity levels, and a second hardware agent of the plurality of hardware agents is programmable to a first number of the plurality of granularity levels. Programmable for numbers, where the second number is different from the first number. In one embodiment, a given memory controller of a plurality of memory controllers has one or more registers programmable with data identifying which address bits are included at a plurality of granularity levels and one or more different granularity levels within the given memory controller. includes them.

13 is a flow diagram illustrating one embodiment of a method for dropping address bits to form a compressed pipe address in a memory controller. Although blocks are shown in a specific order for ease of understanding, other orders may be used. The memory controller may be configured to perform portions or all of the method.

The method may include receiving an address including a plurality of address bits from a first memory controller of a plurality of memory controllers in the system. The address is routed to the first memory controller, and a first memory device of the plurality of memory devices controlled by the first memory controller is selected based on the plurality of hashes of the plurality of sets of address bits (block 150) ). The method may further include dropping a plurality of the plurality of address bits (block 152). A plurality of given bits among the plurality of address bits are included in one of the plurality of hashes and excluded from the remaining ones of the plurality of hashes. The method may include shifting remaining address bits of the plurality of address bits to form a compressed address for use within the first memory controller (block 154).

In one embodiment, the method may further include recovering a plurality of the plurality of address bits based on the identification of the first memory controller and the plurality of sets of address bits used in the plurality of hashes. In one embodiment, the method may further include accessing a memory device controlled by a memory controller based on the compressed address. In one embodiment, the method may further include programming a plurality of configuration registers to identify a plurality of sets of address bits included in respective ones of the plurality of hashes. In one embodiment, programming may include programming a plurality of configuration registers with bit masks that identify a plurality of sets of address bits. In one embodiment, the method further includes programming a plurality of configuration registers to identify a plurality of the plurality of address bits that are dropped. In one embodiment, programming includes programming a plurality of configuration registers with one-hot bit masks.

computer system

Referring next to Figure 14, a block diagram of one embodiment of system 700 is shown. In the illustrated embodiment, system 700 includes at least one instance of a system-on-chip (SOC) 10 coupled to one or more peripherals 704 and external memory 702. A power supply unit (PMU) 708 is provided that supplies supply voltages to the SOC 10, as well as one or more supply voltages to the memory 702 and/or peripherals 154. In some embodiments, as previously noted, more than one instance of SOC 10 may be included (and more than one memory 702 may also be included). In one embodiment, memory 702 may include memory devices 28 as illustrated in FIG. 1 .

Peripherals 704 may include any desired circuitry depending on the type of system 700. For example, in one embodiment, system 704 may be a mobile device (e.g., a personal digital assistant (PDA), smart phone, etc.) and peripherals 704 may be Wi-Fi , may include various types of wireless communication devices such as Bluetooth, cellular, global positioning system, etc. Peripherals 704 may also include additional storage, including RAM storage, solid state storage, or disk storage. Peripherals 704 may include user interface devices, such as a display screen, including a touch display screen or a multitouch display screen, a keyboard or other input device, a microphone, speakers, etc. In other embodiments, system 700 may be any type of computing system (eg, desktop personal computer, laptop, workstation, nettop, etc.).

External memory 702 may include any type of memory. For example, the external memory 702 may include SRAM, dynamic RAM (DRAM), such as synchronous DRAM (SDRAM), double data rate (DDR, DDR2, DDR3, etc.) SDRAM, RAMBUS DRAM, It may be low-power versions of DDR DRAM (eg, LPDDR, mDDR, etc.). External memory 702 may include one or more memory modules on which memory devices are mounted, such as single inline memory modules (SIMMs), dual inline memory modules (DIMMs), etc. You can. Alternatively, external memory 702 may be one or more memory devices mounted on SOC 20 in a chip-on-chip or package-on-package implementation. may include.

As illustrated, system 700 is shown to have a wide range of areas of application. For example, system 700 may operate on a desktop computer 710, a laptop computer 720, a tablet computer 730, a cellular or mobile phone 740, or a television 750 (or a set-top box coupled to a television). It can be used as part of chips, circuitry, components, etc. Smartwatch and health monitoring device 760 are also illustrated. In some embodiments, a smartwatch may include various general-purpose computing-related functions. For example, a smartwatch can provide access to email, cell phone service, the user's calendar, etc. In various embodiments, a health monitoring device may be a dedicated medical device or may otherwise include dedicated health-related functionality. For example, health monitoring devices can monitor a user's vital signs, track the user's proximity to other users for epidemiological social distancing purposes, contact tracing, and identify health risks. This may include providing communications to emergency services. In various embodiments, the above-mentioned smartwatch may or may not include some or any health monitoring-related features. Other wearable devices, such as devices worn around the neck, implantable devices in the body, glasses designed to provide augmented and/or virtual reality experiences, etc. are also contemplated.

System 700 may further be used as part of cloud-based service(s) 770. For example, the previously mentioned devices, and/or other devices, may access computing resources (i.e., remotely located hardware and/or software resources) within the cloud. Still further, system 700 may be used on one or more devices in the home other than those previously mentioned. For example, devices in the home can monitor and detect conditions that require attention. For example, various devices within the home (e.g., refrigerators, cooling systems, etc.) may monitor the status of the devices and provide alerts to the homeowner (or, for example, a repair facility) when certain events are detected. . Alternatively, a thermostat can monitor the temperature within the home and automate adjustments to the heating/cooling system based on a history of responses to various conditions by the homeowner. Additionally, application of system 700 to various transportation modes is illustrated in FIG. 14. For example, system 700 can be used on aircraft, trains, buses, rental cars, personal automobiles, watercraft from personal boats to cruise liners, scooters (for rental or ownership), etc. It can be used in control and/or entertainment systems. In various instances, system 700 may be used to provide automated guidance (e.g., self-driving vehicles), general system control, etc. Many other embodiments of any of these are possible and contemplated. Note that the devices and applications illustrated in FIG. 14 are illustrative only and are not intended to be limiting. Other devices are possible and contemplated.

computer-readable storage media

Referring now to Figure 15, a block diagram of one embodiment of a computer accessible storage medium 800 is shown. Generally speaking, a computer accessible storage medium may include any storage medium that is accessible by a computer during use to provide instructions and/or data to the computer. For example, computer accessible storage media includes magnetic or optical media, such as disks (fixed or removable), tape, CD-ROM, DVD-ROM, CD-R, CD-RW, DVD-R, DVD. -May include storage media such as RW or Blu-Ray. Storage media may further include volatile or non-volatile memory media such as RAM (e.g., synchronous dynamic RAM (SDRAM), Rambus DRAM (RDRAM), static RAM (SRAM), etc.), ROM, or flash memory. there is. Storage media may be physically included within a computer on which the storage media provides instructions/data. Alternatively, storage media may be connected to a computer. For example, storage media may be connected to a computer through a network or wireless link, such as network attached storage. Storage media may be connected through a peripheral interface such as USB (Universal Serial Bus). In general, computer accessible storage medium 800 may store data in a non-transitory manner, and non-transitory in this context may refer to not transmitting instructions/data on a signal. For example, non-transitory storage can be volatile (and may lose stored instructions/data in response to a power failure) or non-volatile.

Computer-accessible storage medium 800 of FIG. 15 may store database 804 representing SOC 10. Generally, database 804 may be a database that can be read by a program and used directly or indirectly to fabricate hardware that includes SOC 10. For example, a database can be a behavioral-level description or register-transfer level description of hardware functions in a high level design language (HDL) such as Verilog or VHDL. , RTL) description. The description can be read by a synthesis tool that can synthesize the description to generate a netlist containing a list of gates from the synthesis library. The netlist contains a set of gates that also represent the functionality of the hardware comprising the SOC 10. The netlist can then be placed and routed to create a data set that describes the geometries to be applied to the masks. The masks may then be used in various semiconductor manufacturing steps to create a semiconductor circuit or circuits corresponding to SOC 10. Alternatively, database 804 on computer accessible storage medium 800 may be a netlist (with or without a synthesis library) or a data set, as desired.

Although computer accessible storage medium 800 stores a representation of SOC 10, other embodiments may include any subset of the components shown in FIG. 1, as desired, of any portion of SOC 10. Expressions can be returned. Database 804 may represent any portion of the above.

As illustrated in FIG. 13 , computer accessible storage medium 800 may further store one or more of a virtual memory page allocator 806 and a memory monitor and fold/unfold code 808 . Virtual memory page allocator 806, when running on a computer, such as the various computer systems described herein that include one or more SOCs 10 (and more specifically one of P clusters 14A, 14B) and instructions, when executed on a processor within the foregoing, that cause the computer to perform operations including those described above with respect to a virtual memory page allocator (e.g., with respect to FIGS. 8-11). Similarly, the memory monitor and fold/unfold code 808, when executed on a computer, such as the various computer systems described herein that include one or more SOCs 10 (and more specifically P clusters ( instructions, when executed on a processor within one or more of 14A, 14B), that cause the computer to perform operations including those described above for the memory monitor and fold/unfold code (e.g., with respect to FIGS. 8-11) may include.

***

This disclosure includes references to “an embodiment” or “groups of embodiments” (e.g., “some embodiments” or “various embodiments”). Embodiments are different implementations or instances of the disclosed concepts. References to “one embodiment,” “one embodiment,” “particular embodiment,” etc. do not necessarily refer to the same embodiment. Many possible embodiments are contemplated, including those specifically disclosed, as well as modifications or alternatives that fall within the spirit or scope of the disclosure.

This disclosure may discuss potential advantages that may arise from the disclosed embodiments. All implementations of these embodiments will not necessarily exhibit any or all of the potential advantages. Whether a benefit is realized for a particular implementation depends on many factors, some of which are beyond the scope of this disclosure. In fact, there are many reasons why implementations that fall within the scope of the claims may not exhibit some or all of the disclosed advantages. For example, a particular implementation may include other circuitry outside the scope of the disclosure, along with one of the disclosed embodiments, that negates or diminishes one or more of the disclosed advantages. Moreover, suboptimal design implementation (e.g., implementation techniques or tools) of a particular implementation may also negate or diminish the disclosed advantages. Even assuming a skillful implementation, realization of the benefits may still depend on other factors such as the environmental circumstances in which the implementation unfolds. For example, inputs supplied to a particular implementation may prevent one or more problems addressed in the disclosure from occurring at a particular opportunity, and as a result, the benefits of the solution may not be realized. Given the existence of possible factors outside the present disclosure, it is expressly intended that any potential advantages described herein should not be construed as claim limitations that must be met to establish infringement. Rather, the identification of such potential advantages is intended to illustrate the type(s) of improvement available to designers having the benefit of the present disclosure. That such benefits are acceptably described (e.g., by stating that a particular benefit "could occur") is not intended to convey doubt as to whether such benefits can in fact be realized, but rather that the realization of those benefits is It is intended to recognize the technical reality that this often depends on additional factors.

Unless otherwise stated, the examples are non-limiting. That is, the disclosed embodiments are not intended to limit the scope of claims made based on this disclosure, even if only a single example is described for a particular feature. The disclosed embodiments are intended to be illustrative rather than restrictive, without any statement of the disclosure to the contrary. Accordingly, this application is intended to admit not only the claims that cover the disclosed embodiments, but also such alternatives, modifications and equivalents that will be apparent to those skilled in the art having the benefit of this disclosure.

For example, features in this application may be combined in any suitable way. Accordingly, new claims may be made during examination of this application (or an application claiming priority thereto) for any such combination of features. In particular, with reference to the appended claims, features from dependent claims may be combined with features of other dependent claims, where appropriate, including claims that rely on other independent claims. Similarly, features from individual independent claims may be combined where appropriate.

Accordingly, although attached dependent claims may be written so that each depends on a single other claim, additional dependencies are also contemplated. Any combination of features consistent with this disclosure is contemplated and may be claimed in this or other application. In short, the combinations are not limited to those specifically recited in the appended claims.

Where appropriate, claims written in one format or statutory type (e.g., device) are also considered to be intended to support corresponding claims in another format or statutory type (e.g., method).

***

Because this disclosure is a legal document, various terms and phrases may be subject to administrative and judicial interpretation. Notice is given herein that the following paragraphs, as well as the definitions provided throughout this disclosure, will be used in determining how to interpret claims made based on this disclosure.

References to the singular form of an article (i.e., a noun or noun phrase preceded by “a”, “an” or “the”) are intended to mean “one or more” unless the context clearly dictates otherwise. do. Accordingly, reference to “an item” in a claim does not exclude additional instances of the item without accompanying context. “Plural” items refer to two or more sets of items.

The word “may” is used in this specification not in a mandatory sense (i.e., must) but in a permissive sense (i.e., having the possibility of doing, being able to do).

The terms “comprising” and “including” and their forms are open-ended and mean “including but not limited to.”

When the term “or” is used in this disclosure in relation to a list of options, it will generally be understood to be used in an inclusive sense, unless the context provides otherwise. Accordingly, reference to “x or y” is equivalent to “x or y, or both” and thus covers 1) x but not y, 2) y but not x, and 3) both x and y. On the other hand, phrases like “either x or y, but not both” make it clear that “or” is being used in an exclusive sense.

Reference to "w, x, y, or z, or any combination thereof" or "... at least one of w, It is intended to cover possibilities. For example, given a set [w, x, y, z], these phrases refer to any single element of the set (e.g., w but not for example w and x but not y or z), any three elements (e.g. w, x and y but not z) and all four elements. Thus, the phrase "...at least one of w, x, y, and z" refers to at least one element of the set [w, x, y, z], thereby exhausting all possible combinations within this list of elements. Cover. These phrases should not be interpreted to require that there is at least one instance of w, at least one instance of x, at least one instance of y, and at least one instance of z.

Various “labels” may precede nouns or noun phrases in this disclosure. Unless the context provides otherwise, different labels used for a feature (e.g., “first circuit,” “second circuit,” “particular circuit,” “given circuit,” etc.) refer to different instances of the feature. refers to Additionally, when applied to features, the “first,” “second,” and “third” labels do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless otherwise noted. .

The phrase “based on” is used to describe one or more factors that influence a decision. These terms do not exclude the possibility that additional factors may influence the decision. That is, the decision may be based solely on the specified factors or may be based on the specified factors as well as other, unspecified factors. Consider the statement “Decide A based on B.” These phrases specify that B is used to determine A or is a factor influencing A's decision. This phrase does not exclude that A's decision may also be based on some other factors, such as C. Additionally, this phrase is intended to cover embodiments in which A is determined based solely on B. As used herein, the phrase “based on” is synonymous with the phrase “based at least in part.”

The phrases “in response to” and “in response to” describe one or more factors that trigger an effect. This phrase does not exclude the possibility that additional factors may influence or otherwise trigger the effect jointly with or independently of the specified factors. That is, the effect may respond only to these factors, or it may respond to the specified factors as well as other unspecified factors. Consider the statement “Do A in response to B.” These phrases specify that B is an argument that triggers the execution of A or triggers a specific result for A. This phrase does not exclude that performing A may also respond to some other argument, such as C. This phrase also does not exclude that performing A may be performed jointly in response to B and C. This phrase is also intended to cover embodiments in which A is performed in response only to B. As used herein, the phrase “in response” is synonymous with the phrase “at least in part in response.” Similarly, the phrase “in response to” is synonymous with the phrase “at least in part in response to.”

***

Within this disclosure, different entities (which may be variously referred to as “units,” “circuits,” other components, etc.) are described as being “configured” to perform one or more tasks or operations. may be or may be charged. This expression - [entity] configured to [perform one or more tasks] - is used herein to refer to a structure (i.e., a physical thing). More specifically, this expression is used to indicate that this structure is arranged to perform one or more tasks during operation. A structure can be said to be "configured" to perform some task even if the structure is not currently performing it. Accordingly, an entity described or referred to as being “configured” to perform some task refers to something physical, such as a device, circuit, system having a processor unit and memory storing executable program instructions to implement the task, etc. This phrase is not used herein to refer to something intangible.

In some cases, various units/circuits/components may be described herein as performing a task or set of operations. It is understood that these entities, even if not specifically mentioned, are “configured” to perform such tasks/actions.

The term “configured to” is not intended to mean “configurable to”. For example, an unprogrammed FPGA would not be considered “configured” to perform a specific function. However, these unprogrammed FPGAs can be “configurable” to perform their functions. After appropriate programming, the FPGA can then be said to be “configured” to perform a specific function.

For purposes of U.S. patent applications based on this disclosure, a statement in a claim that a structure is “configured” to perform one or more tasks expressly means that a claim element is within the meaning of 35 U.S.C. §112(f) is not intended to apply. If an applicant seeks to invoke section 112(f) during prosecution of a U.S. patent application based on this disclosure, the applicant will recite the claim elements using a “means for performing a function” structure.

Different “circuits” may be described in this disclosure. These circuits or “circuitry” may include combinational logic, clocked storage devices (e.g., flip-flops, registers, latches, etc.), finite state machines, memory (e.g., random access memory, embedded It constitutes hardware that includes various types of circuit elements such as dynamic random access memory, programmable logic arrays, etc. The circuitry can be custom designed or taken from standard libraries. In various implementations, the circuitry may include digital components, analog components, or a combination of both, as appropriate. Certain types of circuits may be generally referred to as “units” (eg, decode unit, arithmetic logic unit (ALU), functional unit, memory management unit (MMU), etc.). Such units also refer to circuits or circuit sections.

Accordingly, the disclosed circuits/units/components and other elements illustrated in the drawings and described herein include hardware elements such as those described in the previous paragraph. In many cases, the internal arrangement of hardware elements within a particular circuit can be specified by describing the functionality of that circuit. For example, a particular “decode unit” may be described as performing the function of “processing the opcode of an instruction and routing the instruction to one or more of a plurality of functional units,” which means that the decode unit means “configured” to perform these functions. This functional specification is sufficient to suggest to those skilled in the computer arts the set of possible structures for the circuit.

In various embodiments, as discussed in the previous paragraph, other elements, arrangements, and other elements, arrangements, and such circuits/units/components are defined by circuits, units, and the functions or operations they are configured to implement. With respect to each other and the way they interact, they form the microarchitectural definition of the hardware that is ultimately fabricated in an integrated circuit or programmed into an FPGA to form a physical implementation of the microarchitectural definition. Accordingly, the microarchitecture definition is recognized by those skilled in the art as a structure from which many physical implementations can be derived, all of which fall within the broader structure described by the microarchitecture definition. That is, a person skilled in the art, given the microarchitecture definition provided in accordance with the present disclosure, will be able, without undue experimentation and by the application of ordinary skill, to describe the circuits/units/components in a hardware description language (HDL) such as Verilog or VHDL. ) You can implement the structure by coding. HDL descriptions are often expressed in a way that may appear functional. However, for those skilled in the art, this HDL description is the method used to translate the structure of a circuit, unit or component to the next level of implementation details. Such HDL descriptions can be either behavior code (which is typically not synthesizable), register transfer language (RTL) code (which, as opposed to behavior code, is typically synthesizable), or structural code (e.g., logic gates and their connections). It can take the form of a netlist specifying . The HDL description can be sequentially synthesized for a library of cells designed for a given integrated circuit manufacturing technology and modified for timing, power and other reasons to create the final design database, which can be sent to the foundry. can be used to create masks and ultimately create integrated circuits. Some hardware circuits or portions thereof can also be custom designed with a schematic editor and captured together with the synthesized circuitry into an integrated circuit design. Integrated circuits may include transistors and other circuit elements (eg, passive elements such as capacitors, resistors, inductors, etc.) and interconnections between the transistors and circuit elements. Some embodiments may implement multiple integrated circuits coupled together to implement a hardware circuit and/or in some embodiments discrete elements may be used. Alternatively, the HDL design can be synthesized into a programmable logic array, such as a Field Programmable Gate Array (FPGA), and implemented on the FPGA. This decoupling between the design of a group of circuits and the subsequent low-level implementation of those circuits generally allows for scenarios in which a circuit or logic designer does not specify a particular set of structures for the low-level implementation beyond a description of what the circuit is configured to do. This is because these processes are performed at different stages of the circuit implementation process.

The fact that many different low-level combinations of circuit elements can be used to implement the same specification of a circuit results in a large number of equivalent structures for that circuit. As mentioned, these low-level circuit implementations may vary depending on changes in manufacturing technology, the foundry selected to fabricate the integrated circuit, the library of cells provided for a particular project, etc. In many cases, the choices made by different design tools or methodologies to create these different implementations may be arbitrary.

Moreover, it is common for a single implementation of a particular functional specification of a circuit to include a large number of devices (eg, millions of transistors) for a given embodiment. Accordingly, the sheer volume of this information makes it impractical to provide a complete description of the low-level structures used to implement a single embodiment, let alone the vast array of equivalent possible implementations. For this reason, this disclosure describes the structure of circuits using functional shorthand commonly used in the industry.

Various embodiments are described herein, including those described in the following numbered examples:

1. As a method,

detecting that at least one first memory slice among the plurality of memory slices in the memory system should be disabled, wherein a given memory slice among the plurality of memory slices is a physical memory resource to which a plurality of physical pages are mapped; and

Based on detecting that the first memory slice should be disabled:

copying a subset of physical pages in the first memory slice to another of the plurality of memory slices, wherein data in the subset of physical pages is being accessed above a threshold rate;

remapping virtual addresses corresponding to a subset of physical pages to another memory slice; and

A method comprising disabling the first memory slice.

2. The method of Example 1, wherein the at least one first memory slice is a plurality of the plurality of memory slices and the plurality of number of the plurality of memory slices is a power of 2.

3. The method of Example 1 or Example 2, wherein disabling the first memory slice includes actively placing one or more dynamic access memories (DRAMs) within the first memory slice in a self-refresh mode. method.

4. The method of Example 1 or Example 2, wherein disabling the first memory slice allows one or more dynamic access memories (DRAMs) within the first memory slice to transition to self-refresh mode due to lack of access. A method comprising the steps of:

5. The method of any of examples 1-4, wherein the memory system includes a plurality of memory controllers and the physical memory resource includes at least one of the plurality of memory controllers.

6. The example of any of Examples 1-4, wherein the memory system includes a plurality of memory channels, a given dynamic random access memory (DRAM) is coupled to one of the plurality of memory channels, and the given memory slice has a plurality of memory slices. A method comprising at least one of the memory channels.

7. The method of Example 6, wherein the given memory slice is one memory channel of a plurality of memory channels.

8. The method of any of examples 1-7, wherein determining that the first memory slice should be disabled includes:

detecting that the access rate for the first memory slice is lower than a first threshold; and

A method comprising: identifying a subset of physical pages that are accessed more frequently than a second threshold.

9. For Example 8:

The method further comprising disabling assignment of the plurality of physical pages corresponding to the first memory slice to virtual addresses in the memory allocator based on detecting that the access rate is below the first threshold.

10. In Example 9,

The method further comprising disabling allocation of the plurality of physical pages followed by performing identification.

11. The method of any of examples 1 through 10, wherein the copying step includes:

A method comprising copying data from one or more physical pages of a subset containing modified data to another memory slice in a memory system.

12. In Example 11, the copying step is:

The method further comprising copying data from the remaining physical pages of the subset subsequent to copying the data from the one or more physical pages.

13. The method of any of examples 1-12, wherein determining that the first memory slice should be disabled is based on detecting one or more factors in the system that indicate a decrease in power consumption.

14. As a system,

One or more memory controllers coupled to one or more memory devices forming a memory system - the memory system includes a plurality of memory slices, and a given memory slice of the plurality of memory slices is a physical memory resource to which a plurality of physical pages are mapped. lim -;

one or more processors; and

A non-transitory computer-readable storage medium storing a plurality of instructions that, when executed by one or more processors, cause a system to perform operations, the operations comprising:

detecting that a first memory slice of the plurality of memory slices in the memory system should be disabled; and

Based on detecting that the first memory slice should be disabled:

copying a subset of physical pages in a first memory slice to another of the plurality of memory slices, where data in the subset of physical pages is being accessed above a threshold rate;

remapping virtual addresses corresponding to a subset of physical pages to a different memory slice; and

A system comprising disabling a first memory slice.

15. The system of Example 14, where disabling the first memory slice includes actively placing one or more dynamic access memories (DRAMs) within the first memory slice in a self-refresh mode.

16. The method of Example 14, where disabling the first memory slice includes allowing one or more dynamic access memories (DRAMs) within the first memory slice to transition to a self-refresh mode due to lack of access. , system.

17. The system of any of examples 14-16, wherein the memory system includes a plurality of memory controllers and the physical memory resource includes at least one of the plurality of memory controllers.

18. The example of any of Examples 14-17, wherein the memory system includes a plurality of memory channels, a given dynamic random access memory (DRAM) is coupled to one of the plurality of memory channels, and the given memory slice is coupled to one of the plurality of memory channels. A system comprising at least one of the memory channels.

19. The method of any of examples 14-18, wherein determining that the first memory slice should be disabled includes:

detecting that the access rate for the first memory slice is below a first threshold; and

A system comprising: identifying a subset of physical pages that are accessed more frequently than a second threshold.

20. In Example 19,

The system further comprising disabling assignment of the plurality of physical pages corresponding to the first memory slice to virtual addresses in the memory allocator based on detecting that the access rate is below the first threshold.

21. In Example 20,

The system further comprising performing identification followed by disabling allocation of the plurality of physical pages.

22. A non-transitory computer-readable storage medium storing a plurality of instructions that, when executed by one or more processors in a system including a memory system, cause a system to perform operations, the operations comprising:

detecting that a first memory slice of the plurality of memory slices in the memory system should be disabled, where a given memory slice of the plurality of memory slices is a physical memory resource to which a plurality of physical pages are mapped; and

Based on detecting that the first memory slice should be disabled:

copying a subset of physical pages in a first memory slice to another of the plurality of memory slices, where data in the subset of physical pages is being accessed above a threshold rate;

remapping virtual addresses corresponding to a subset of physical pages to a different memory slice; and

A non-transitory computer-readable storage medium comprising disabling a first memory slice.

23. As a system,

a plurality of memory controllers configured to control access to memory devices;

a plurality of hardware agents configured to access data in memory devices using memory addresses; and

comprising a communication fabric coupled to a plurality of memory controllers and a plurality of hardware agents, wherein

the communication fabric is configured to route a memory request having a first memory address to a first memory controller of the plurality of memory controllers based on the first memory address;

A plurality of subsets of address bits of the first memory address are hashed to direct the memory request to the first memory controller at a plurality of levels of granularity,

At least one address bit in a given subset of the plurality of subsets is not included in the remaining subsets of the plurality of subsets;

The first memory controller is configured to drop a plurality of address bits of the first memory address to form a second address used within the first memory controller;

A system, wherein each bit of the plurality of address bits is at least one address bit in a given subset of the plurality of subsets.

24. The system of Example 23 where hash is a logically reversible Boolean operation.

25. The system of Example 24, wherein the hash is an exclusive-or (XOR) reduction of the address bits.

26. The system of any of examples 23-25, wherein the first memory controller is configured to recover dropped bits from other address bits and an identification of the first memory controller to which the memory request is directed.

27. The system of any of examples 23-26, further comprising a plurality of configuration registers programmable to identify a plurality of subsets of address bits to be hashed at respective levels of the plurality of granularity levels.

28. The system of Example 27, wherein the plurality of configuration registers are programmable with bit masks that identify address bits.

29. The system of any of examples 23-28, further comprising a plurality of configuration registers programmable to identify the plurality of address bits that are dropped.

30. The system of Example 28 or Example 29, wherein the plurality of configuration registers are programmable with one-hot bit masks.

31. Example The example of any of claims 23-30, wherein the plurality of memory controllers are physically distributed across one or more integrated circuit die within the system, and wherein the subset of the plurality of granularity levels corresponds to the physical location of the first memory controller and associated system.

32. The system of example 31, wherein the subset includes at least one die level that identifies the integrated circuit die on which the first memory controller is located.

33. The system of example 31 or 32, wherein the subset includes a plurality of levels that identify a physical location within the integrated circuit die.

34. The method of any of clauses 31-33, wherein the subset identifies which of the plurality of memory devices controlled by the first memory controller is storing data associated with the first memory address. A system comprising a plurality of levels.

35. As a method,

Receiving an address including a plurality of address bits from a first memory controller among a plurality of memory controllers in the system - the address is routed to the first memory controller, and the address is routed to one of the plurality of memory devices controlled by the first memory controller. a first memory device is selected based on a plurality of hashes of a plurality of sets of address bits;

dropping a plurality of the plurality of address bits, wherein a given plurality of bits of the plurality of address bits are included in one of the plurality of hashes and excluded from the remainder of the plurality of hashes; and

A method comprising shifting remaining address bits of the plurality of address bits to form a compressed address for use within a first memory controller.

36. In Example 35:

The method further comprising recovering a plurality of the plurality of address bits based on the identification of the first memory controller and the sets of the plurality of address bits used in the plurality of hashes.

37. The method of examples 35 or 36, further comprising accessing a memory device controlled by the memory controller based on the compressed address.

38. The method of any of Examples 35 through 27,

The method further comprising programming a plurality of configuration registers to identify a plurality of sets of address bits included in respective ones of the plurality of hashes.

39. The method of example 38, wherein programming includes programming a plurality of configuration registers with bit masks that identify a plurality of sets of address bits.

40. The method of any of Examples 35 through 39,

The method further comprising programming a plurality of configuration registers to identify a plurality of the plurality of address bits that are dropped.

41. The method of examples 39 or 40, wherein programming includes programming the plurality of configuration registers with one-hot bit masks.

42. As a memory controller,

a plurality of configuration registers programmable to identify a respective plurality of address bits that are hashed to select a memory controller as a destination for a memory request containing an address containing the address bits; and

a control circuit coupled to a plurality of configuration registers, wherein the control circuit is configured to generate a compressed address to access one or more memory devices controlled by the memory controller, wherein the control circuit is configured to generate a compressed address to access the plurality of address bits. A memory controller configured to generate a compressed address by dropping and shifting remaining bits of the address.

43. The memory controller of example 42, further comprising a second plurality of configuration registers coupled to the control circuit, wherein the second plurality of configuration registers are programmable to identify the plurality of address bits that are dropped.

Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully understood. It is intended that the following claims be construed to encompass all such variations and modifications.

Claims (27)

As a system,
a plurality of memory controllers configured to control access to memory devices;
A plurality of hardware agents configured to access data in the memory devices using memory addresses.
- the memory addresses are defined within a memory address space that maps memory addresses to memory locations within the memory devices,
A given memory address in the memory address space uniquely identifies a memory location within one of the memory devices coupled to one of the plurality of memory controllers,
A given page in the memory address space is divided into a plurality of blocks,
the plurality of blocks of the given page are distributed across two or more of the plurality of memory controllers; and
A communication fabric coupled to the plurality of memory controllers and the plurality of hardware agents,
the communication fabric is configured to route a memory request having a first memory address to a first memory controller of the plurality of memory controllers based on the first memory address,
wherein circuitry within the system is configured to hash independently-specified address bits of the first memory address to direct the memory request to the first memory controller at a plurality of levels of granularity. According to paragraph 1,
and wherein a given level of the plurality of granularity levels selects between at least two memory controllers of the plurality of memory controllers. According to claim 1 or 2,
the plurality of memory controllers are physically distributed across one or more integrated circuit die within the system;
the one or more integrated circuit dies are a plurality of integrated circuit dies;
the plurality of granularity levels include a die level;
The system of claim 1, wherein the die level specifies which of the plurality of integrated circuit dies includes the first memory controller. According to any one of claims 1 to 3,
the plurality of memory controllers are physically distributed across one or more integrated circuit die within the system;
the plurality of memory controllers on a given integrated circuit die are logically divided into a plurality of slices based on physical locations on the given integrated circuit die;
at least two of the plurality of memory controllers are included in a given one of the plurality of slices;
the plurality of granularity levels include a slice level;
The system of claim 1, wherein the slice level specifies which of the plurality of slices includes the first memory controller. According to paragraph 4,
the at least two memory controllers within the given slice are logically partitioned into a plurality of rows based on their physical location on the given integrated circuit die;
the plurality of granularity levels include a row level;
wherein the row level specifies which of the plurality of rows contains the first memory controller. According to clause 5,
the plurality of rows include a plurality of sides based on physical location on the given integrated circuit die;
the plurality of granularity levels include a side level;
The system of claim 1, wherein the side level specifies which side of a given row of the plurality of rows contains the first memory controller. 7. The method of any one of claims 1 to 6, wherein a given hardware agent of the plurality of hardware agents is programmed with data identifying which address bits are included in the hash at one or more of the plurality of granularity levels. A system, possibly including one or more registers. The method of claim 7, wherein a first hardware agent of the plurality of hardware agents is programmable for a first number of the plurality of granularity levels, and a second hardware agent of the plurality of hardware agents is programmable for a first number of the plurality of granularity levels. A system, wherein the system is programmable for a second number of levels, the second number being different from the first number. 9. The method of claim 7 or 8, wherein a given memory controller of the plurality of memory controllers identifies which address bits are included in the plurality of granularity levels and one or more other granularity levels within the given memory controller. A system comprising one or more registers programmable with data. 10. The system of claim 9, wherein the finest level of granularity identifies a memory device coupled to the given memory controller. 11. The method of any one of claims 7 to 10, wherein the one or more registers cause the communication fabric to route a first memory request having a first address to a first memory controller of the plurality of memory controllers. It is programmable with data that routes a second memory request having a second address to a second memory controller that is physically distant from the first memory controller among the plurality of memory controllers, and the first address and the second address are Adjacent addresses at a second granularity level, the system. 12. The system of claim 11, wherein the first route of the first memory request through the communications fabric and the second route of the second memory request through the communications fabric are completely non-overlapping. 11. The method of any one of claims 7-10, wherein the one or more registers allow the communication fabric to store the plurality of memory requests for consecutive addresses in a pattern that distributes the plurality of memory requests across physically distant memory controllers. A system, programmable with data to route memory requests to different memory controllers of the plurality of memory controllers. According to any one of claims 1 to 13,
the independently-specified address bits include a plurality of subsets of address bits corresponding to respective levels of the plurality of granularity levels,
At least one address bit in a given subset of the plurality of subsets is not included in the remaining subsets of the plurality of subsets;
the first memory controller is configured to drop a plurality of address bits of the first memory address to form a second address used within the first memory controller;
and wherein respective bits of the plurality of address bits are the at least one address bit within the given subset of the plurality of subsets. 15. The system of claim 14, wherein the hash is a logically reversible Boolean operation. 16. The system of claim 15, wherein the hash is an exclusive-or (XOR) reduction of the address bits. 17. The system of claim 15 or 16, wherein the first memory controller is configured to recover the dropped bits from the other address bits and an identification of the first memory controller to which the memory request is directed. As a method,
generating a memory request having a first address in a memory address space that is mapped to memory devices in a system having a plurality of memory controllers.
- a given memory address within the memory address space uniquely identifies a memory location within one of the memory devices coupled to one of the plurality of memory controllers,
A given page in the memory address space is divided into a plurality of blocks,
the plurality of blocks of the given page are distributed across two or more of the plurality of memory controllers;
Directing the memory request to a first of the plurality of memory controllers by hashing independently-specified sets of address bits from the first address, wherein the independently-specified sets of address bits are positioning the first memory controller at granularity levels; and
and routing the memory request to the first memory controller based on the hashing. According to clause 18,
and wherein a given level of the plurality of granularity levels selects between at least two memory controllers of the plurality of memory controllers. According to claim 18 or 19,
the plurality of memory controllers are physically distributed across one or more integrated circuit die;
the one or more integrated circuit dies are a plurality of integrated circuit dies;
the plurality of granularity levels include a die level;
The method of claim 1, wherein the die level specifies which of the plurality of integrated circuit dies includes the first memory controller. According to any one of claims 18 to 20,
the plurality of memory controllers are physically distributed across one or more integrated circuit die;
the plurality of memory controllers on a given integrated circuit die are logically divided into a plurality of slices based on physical locations on the given integrated circuit die;
at least two of the plurality of memory controllers are included in a given one of the plurality of slices;
the plurality of granularity levels include a slice level;
The method of claim 1, wherein the slice level specifies which of the plurality of slices includes the first memory controller. According to clause 21,
the at least two memory controllers within the given slice are logically partitioned into a plurality of rows based on their physical location on the given integrated circuit die;
the plurality of granularity levels include a row level;
wherein the row level specifies which of the plurality of rows contains the first memory controller. According to clause 22,
the plurality of rows include a plurality of sides based on physical location on the given integrated circuit die;
the plurality of granularity levels include a side level;
wherein the side level specifies which side of a given row of the plurality of rows contains the first memory controller. 24. The method of any one of claims 18 to 23, wherein a given hardware agent of the plurality of hardware agents generating memory requests comprises one or more registers, the method comprising:
The method further comprising programming the one or more registers with data identifying which address bits are included in the hash at one or more of the plurality of granularity levels. 25. The method of claim 24, wherein a first hardware agent of the plurality of hardware agents is programmable for a first number of the plurality of granularity levels, and a second hardware agent of the plurality of hardware agents is programmable for a first number of the plurality of granularity levels. A method, wherein the method is programmable for a second number of levels, wherein the second number is different from the first number. 26. The method of claim 24 or 25, wherein a given memory controller of the plurality of memory controllers identifies which address bits are included in the plurality of granularity levels and one or more other granularity levels within the given memory controller. A method comprising one or more registers programmable with data. According to clause 18,
Detecting that at least one first memory slice among a plurality of memory slices in the system should be disabled, wherein a given memory slice among the plurality of memory slices is a physical memory resource to which a plurality of physical pages are mapped, At least one memory slice includes two or more memory controllers of the plurality of memory controllers to which a given page of the plurality of pages is mapped;
Based on detecting that the first memory slice should be disabled:
copying a subset of physical pages in the first memory slice to another of the plurality of memory slices, where data in the subset of physical pages is being accessed above a threshold rate;
remapping virtual addresses corresponding to the subset of physical pages to the different memory slice; and
The method further comprising disabling the first memory slice.

Images