WO2011094436A2 - Procédés et appareil faisant intervenir une interface pour des dispositifs mémoire - Google Patents

Procédés et appareil faisant intervenir une interface pour des dispositifs mémoire Download PDF

Info

Publication number
WO2011094436A2
WO2011094436A2 PCT/US2011/022762 US2011022762W WO2011094436A2 WO 2011094436 A2 WO2011094436 A2 WO 2011094436A2 US 2011022762 W US2011022762 W US 2011022762W WO 2011094436 A2 WO2011094436 A2 WO 2011094436A2
Authority
WO
WIPO (PCT)
Prior art keywords
memory
bus
hint
request
controller
Prior art date
Application number
PCT/US2011/022762
Other languages
English (en)
Other versions
WO2011094436A3 (fr
Inventor
Naveen Muralimanohar
Norman Paul Jouppi
Original Assignee
Hewlett-Packard Development Company, L.P.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hewlett-Packard Development Company, L.P. filed Critical Hewlett-Packard Development Company, L.P.
Priority to CN201180003837.6A priority Critical patent/CN102576338B/zh
Priority to US13/386,396 priority patent/US8938589B2/en
Priority to EP11737662.4A priority patent/EP2529312A4/fr
Publication of WO2011094436A2 publication Critical patent/WO2011094436A2/fr
Publication of WO2011094436A3 publication Critical patent/WO2011094436A3/fr
Priority to US14/564,215 priority patent/US20160246711A9/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/14—Handling requests for interconnection or transfer
    • G06F13/16—Handling requests for interconnection or transfer for access to memory bus
    • G06F13/1605—Handling requests for interconnection or transfer for access to memory bus based on arbitration
    • G06F13/161—Handling requests for interconnection or transfer for access to memory bus based on arbitration with latency improvement
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
    • G06F13/14—Handling requests for interconnection or transfer
    • G06F13/16—Handling requests for interconnection or transfer for access to memory bus
    • G06F13/1605—Handling requests for interconnection or transfer for access to memory bus based on arbitration
    • G06F13/1642—Handling requests for interconnection or transfer for access to memory bus based on arbitration with request queuing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
    • G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
    • G06F12/0804—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches with main memory updating

Definitions

  • DRAMs dynamic random access memories
  • SDRAM synchronous DRAM
  • DRAM physical interfaces include separate address and data lines and separate command lines to accommodate communications between a memory controller and memory devices.
  • a memory controller To perform read or write operations, a memory controller first sends part of an address called a row address, which a DRAM uses to identify a bank and read a corresponding row. The memory controller then sends a column address to identify a particular cache line in an open row. In addition, the memory controller sends separate control signals to differentiate between row addresses and column addresses.
  • a DRAM relies on numerous signals and communications from a memory controller for a significant amount of its operations.
  • FIG. 1 depicts a block diagram of an interface configuration between a memory controller and a memory.
  • FIG. 2 depicts an example read/write bus packet that can be used to communicate read and/or write requests from memory controllers to memory devices.
  • FIG. 3 depicts an example hint bus packet that can be used to communicate hint information from memory controllers to memory devices.
  • FIG. 4 depicts an example response bus packet that can be used to communicate responsive communications from memory devices to memory controllers.
  • FIG. 5 depicts an isometric view of an example printed circuit board (PCB) configuration of a memory bus interconnecting a memory controller with a plurality of memory devices.
  • PCB printed circuit board
  • FIG. 6 depicts an example PCB in-line memory module having a plurality of memory chips and a memory bus interface to communicate with the memory chips.
  • FIG. 7 is a block diagram of a memory module controller that can be used to communicate via a memory bus interface and process hint information.
  • FIG. 8 is a diagram of a memory controller configured to communicate with the DRAM of FIGS. 1 , 5, and 7 via the memory bus interface of FIGS. 1 , 5, and 7.
  • FIGS. 9A and 9B depict a flow diagram of an example process that can be executed by memory modules to process memory access requests and hint information.
  • FIG. 10 depicts a flow diagram of an example process that can be implemented in memory modules to process received hint information.
  • FIG. 1 1 depicts a flow diagram of an example process that can be implemented in a memory controller to generate hint information.
  • FIG. 12 depicts an example 3D chip stack memory module.
  • Example methods, apparatus, and articles of manufacture disclosed herein can be used to facilitate greater scalability than can be achieved using traditional DRAM interface designs without increasing (or without significantly increasing) processor overhead.
  • example methods, apparatus, and articles of manufacture disclosed herein can be used to interface with memory devices to achieve higher bandwidth memory accesses and more power-efficient and time-efficient memory accesses.
  • Example methods, apparatus, and articles of manufacture are disclosed in connection with dynamic random access memory (DRAM) architectures, but may be implemented in connection with other types of memories including memristor memory devices, phase-change RAM (PCRAM) devices, static RAM (SRAM) devices, Ferroelectric RAM (FRAM) devices, etc.
  • DRAM dynamic random access memory
  • PCRAM phase-change RAM
  • SRAM static RAM
  • FRAM Ferroelectric RAM
  • DRAM devices have been designed to operate strictly in accordance with commands from memory controllers such that known DRAM devices execute received commands in a passive manner without deviation.
  • DRAM devices have traditionally had little to no independent logic and, thus, have exhibited a low-degree of autonomy.
  • SDRAM synchronous DRAM
  • DRAM devices operate in accordance with a clock signal such that communications (e.g., read, write, data communications) must be received, processed, and output in accordance with strict timing guidelines associated with the clock signal.
  • physical interfaces of traditional DRAM devices pose limitations for expanding capacity and achieving higher data rate communications. That is, traditional DRAM physical interfaces include separate address and data lines and separate command lines that are not easily expandable.
  • traditional DRAM physical interfaces accommodate communications between a memory controller and memory devices but not memory device-to-memory device communications.
  • PCB printed circuit board
  • DRAM interface designs include how memory interfaces are used to communicate memory access requests and other commands. For example, to perform read or write operations, a memory controller first sends out part of an address called a row address, which a DRAM uses to identify a bank and read a corresponding row. The memory controller then sends out the rest of the address called a column address to identify a particular cache line in an open row. The memory controller sends separate control signals to differentiate between row addresses and column addresses. In addition, the memory controller must model the state of open rows in the DRAM which limits the use of DRAM to the specific type modeled by the controller. Thus, a DRAM relies on numerous communications from a memory controller for a significant amount of its operations.
  • a drawback of such a traditional master-slave design is that as the number of memory chips or ranks inside a memory module increases, memory controller complexity also increases. As processors are provided with more memory controllers, increased complexity in memory controller design will require additional processor area and power budget. Another drawback is that DRAM
  • Drawbacks of traditional DRAM interface designs are further seen in connection with optical interconnects. For example, due to the significant bandwidth increase provided by optics, the number of memory modules that can be connected to a memory controller is relatively high. A memory controller to keep track of all open banks in all DRAM modules would be significantly more complex. Thus, scalability using traditional DRAM interface designs may not be cost-effective or feasible.
  • Example methods, apparatus, and articles of manufacture disclosed herein involve providing memories with bus interfaces configured to exchange information with memory controllers and with other memories (e.g., memory-to- memory transfers) using bus packet formats.
  • the bus packets can be used to communicate address, data, and/or control information between one or more memory devices and one or more memory controllers on a single memory bus using point-to-point or broadcast communications.
  • the example memory bus interfaces described herein can be used in a multi-controller mode to enable connecting multiple memory controllers to one or more memory devices.
  • An example memory bus interface described herein includes a command/request bus and a separate response bus.
  • the command/request bus is used to communicate command information and memory access request information from one or more memory controllers to one or more memory devices.
  • the response bus is used to communicate response information including acknowledgements and requested data between memory devices and memory controllers.
  • the bus packet communications used in connection with the memory bus interface described herein enable more efficient communications between memory controllers and memory devices and better use of memory access bandwidth internal to a memory device.
  • traditional DRAM memory interfaces are configured to serve only one memory controller and only one memory access request at a time. Although this enables a traditional memory controller to be in control of all memory access requests and most or all internal memory operations (e.g., pre-charge, self-refresh, low-power mode transitions, etc.), traditional memory interface architectures place a significant burden on the memory controllers and force all data transfers through the memory controllers.
  • the memory bus, bus packet communications, and internal structures disclosed herein enable a DRAM memory (or other types of memories) to concurrently serve memory controllers and other memory devices on the same memory bus and arbitrate between multiple pending memory access requests from one or more memory controllers or memory devices. That is, the memory device structures described herein can receive memory access requests via bus packets from memory controllers or memory devices, while internally arbitrating pending memory access requests and returning data to the same or other memory controller(s) or memory devices on the memory bus. In this manner, example methods, apparatus, and articles of manufacture disclosed herein enable memory interfaces having significantly less timing constraints than traditional memory interfaces and enable memory devices that can more efficiently access memory storage locations while concurrently receiving additional memory access requests or other bus packet communications from memory controllers or memory devices.
  • any memory can initiate a communication to another memory connected to the same bus without requiring intermediate communication of the copied data to the memory controller or a processor's cache.
  • Checkpointing is an example use of such memory-to- memory transfers.
  • data must be copied to a processor's cache when copying that data from one memory module to another memory module, which causes delays and pollutes the processor's cache leading to unnecessary cache misses.
  • a memory controller can initiate transfers directly between memory modules.
  • example methods, apparatus, and articles of manufacture disclosed herein enable memory devices to perform operations with relatively more autonomy than known memory devices. In this manner, memory bus utilization and memory access operations can be relatively more efficient than in known memory device implementations.
  • example methods, apparatus, and articles of manufacture disclosed herein enable memory controllers to communicate hint information to memory devices that trigger autonomous action by the memory devices. Such hint information can be communicated via, for example, a command/request bus of a memory bus interface.
  • Example hint information may be indicative of a memory controller not needing to access a memory device for some known or expected amount of time such that the memory device can enter a self-refresh state or low power mode (e.g., a standby or sleep mode) without delaying.
  • autonomous decision logic of the memory devices described herein can determine whether to act on or ignore the hint information without needing to receive further direction from the memory controller which communicated the hint information.
  • a memory device implemented in accordance with example methods, apparatus, and articles of manufacture disclosed herein can arbitrate hints received from a memory controller in view of any internally executing or queued memory access operations yet to be fulfilled by the memory device. Through such arbitration, the memory device can autonomously determine whether it can or should act on any particular hint. For example, if a first memory controller determines that it will not be performing any memory accesses for at least the next 100
  • the first memory controller generates a hint to indicate that the memory device is permitted to enter a self-refresh or lower power mode. If concurrently, the memory device receives a subsequent memory access request from the same or another memory controller or another memory device, the memory device can autonomously determine to ignore or defer the hint from the first memory controller because the memory device should remain active to service the subsequent memory access request.
  • ms milliseconds
  • the hint information described herein enables off-loading, sharing, or shifting of significant portions of the memory operations management to memory devices.
  • memory device structures described herein enable a memory device to receive multiple memory access requests from multiple memory controllers and internally arbitrate how to handle such memory access requests for the memory controllers.
  • example memory controllers disclosed herein are burdened with relatively less bus timing constraints because they can send memory access requests and hint information to memory devices and let the memory devices determine the timing for performing the requested or hinted operations.
  • an example memory interface configuration 100 shows a memory controller 102 operatively coupled to a memory 104.
  • the memory controller 102 can be a standalone memory controller integrated circuit (IC) or an embedded memory controller implemented in a processor chip (e.g., fabricated on the same die or located in the same chip package as a processor core).
  • the memory 104 is a DRAM memory.
  • the DRAM memory 104 can be a single DRAM memory IC or a memory module including multiple DRAM memory ICs.
  • the DRAM memory 104 can be an embedded memory implemented in a processor chip.
  • the memory interface configuration 100 includes a memory bus 106.
  • the memory bus 106 may be any number of bits wide and is used to communicate address, data, commands, and/or hint information between the memory controller 102 and the DRAM memory 104. When implemented as a memory module, each line of the memory bus 106 can be connected to multiple memory chips (or memory devices) of the memory module. The memory controller 102 can selectively communicate with separate ones of the memory chips based on chip identification information and/or address ranges as discussed in detail below. [0029] In the illustrated example, the memory controller 102 and the DRAM memory 104 communicate via bus packets transmitted through the memory bus 106. A bus packet can contain one or more of hints 108, data 1 10, addresses 1 12, or operation codes 1 14. In some example implementations, packets may be used to communicate hint information alone, and separate read/write packets may be used to communicate memory access requests. Example bus packets are shown in FIGS. 2, 3, and 4.
  • an example read/write bus packet 200 of FIG. 2 can be used to communicate memory access requests from memory controllers to memory devices.
  • the bus packet 200 can also be used to communicate memory access requests between memory devices.
  • the read/write bus packet 200 includes a header field 202, a destination select field 204, an operation code field 206, and an address field 208.
  • the read/write bus packet 200 can also include a data field 210, a checksum field 212, a parity field 214, and an error correction code (ECC) field 216.
  • ECC error correction code
  • the data field 210, the checksum field 212, the parity field 214, and the ECC field 216 may be present when the read/write bus packet 200 is used to request a write operation.
  • the data field 210 stores write data.
  • the checksum field 212 stores a checksum value
  • the parity field 214 stores a parity value
  • the ECC field 216 stores an ECC value, all of which can be used to detect any errors in the write data communicated in the data field 210 and/or other information in the read/write bus packet 200.
  • a read request message can include address bits and a burst length to reduce the number of read request messages that need to be communicated on the memory bus 106.
  • read request messages can also include stride and request interval values indicating a stride and interval with which burst data should be communicated to a requesting memory controller.
  • stride and request interval values can be used in connection with streaming applications that typically need data at particular intervals or times. The use of stride and request interval values reduces the quantity of read request messages needing to be communicated by memory controllers.
  • a memory controller can communicate a subsequent packet having an interrupt message. Such a packet can be in the form of a hint bus packet 300 described below in connection with FIG. 3 and carrying a non-ignorable interrupt instructing a memory to stop streaming data.
  • the header field 202 includes an
  • the header field 202 can additionally or alternatively include any other information that would be suitable for communicating as header information. Such other information can be, for example, an indication of the type of information or operation being communicated (e.g., read/write/hint information).
  • the destination select field 204 can be used to communicate information indicative of a particular memory device to which a requesting memory controller (or other memory device) intends to send the bus packet 200.
  • the destination select field 204 may be, for example, a memory device identifier that uniquely identifies a specific memory device on a bus (e.g., the memory bus 106).
  • the destination select field 204 is omitted and the address field 208 is used to indicate a target memory device. For example, if a memory device receives the read/write bus packet 200 and determines that the included address pertains to its memory range portion of a physical memory map (e.g., the physical memory map 816 of FIG. 8), the memory device tags or identifies the read/write bus packet 200 as relevant and further processes the bus packet 200.
  • a physical memory map e.g., the physical memory map 816 of FIG. 8
  • the operation code field 206 is used to communicate codes indicating a requested operation.
  • Example codes can be indicative of read operations, write operations, burst reads, etc.
  • the address field 208 can include one or more addresses and/or address offset information indicative of storage locations from which data is to be read or to which data is to be written.
  • the data field 210 can include data communicated by the memory controller 102 when the operation code field 206 indicates a write operation. This data may be, for example, data to be stored in the addressed memory.
  • the checksum field 212 includes a checksum for data in the data field 210.
  • an example hint bus packet 300 can be used to communicate hint information from memory controllers to memory devices or between memory devices.
  • the hint bus packet 300 is shown as having a header field 302, a destination select field 304, an operation code field 306, a hint field 308, an optional parity field 310, and an optional ECC field 312.
  • the parity field 310 and the ECC field 312 can be used to detect errors in the transmission of the hint bus packet 300.
  • the header field 302 can be substantially similar or identical to the header field 202 of FIG. 2.
  • the destination select field 304 can be substantially similar or identical to the destination select field 204 of FIG. 2.
  • the operation code field 306 can be substantially similar or identical to the operation code field 206 of FIG. 2.
  • the operation code field 306 may include a no operation performed (NOP) code or some other code indicating that the bus packet 300 conveys hint information.
  • the hint field 308 is used to convey hint information, which may be used by a memory controller to inform one or more memory devices of internal memory device operations that are permissible based on the memory controller's memory access needs (or lack thereof) for some subsequent amount of time.
  • the amount of time may be specified in the hint or may be pre-known by the memory device based on, for example, the type of hint received.
  • the memory device may not be made aware of an amount of time, but instead may perform the hinted operation (e.g., self-refresh, standby, low-power mode transition, etc.) until it is subsequently activated (e.g., through one or more control lines or a wake operation code in the operation code field 306 of a subsequent hint bus packet) or receives a subsequent memory access request.
  • the hinted operation e.g., self-refresh, standby, low-power mode transition, etc.
  • the hinted operation e.g., self-refresh, standby, low-power mode transition, etc.
  • hints can be used to control hybrid memory modules containing different types of memory technologies (e.g., a DRAM/memristor memory module or a DRAM/PCRAM memory module).
  • a DRAM/memristor memory module e.g., flash memories, memristor memories, and PCRAM memories
  • a DRAM (or SRAM) used as a local cache or write buffer in a memory module can store frequently written or changing data that can be periodically written through to the non-volatile memories.
  • Hint bus packets e.g., the hint bus packet 300
  • the hint bus packet 300 can be
  • a destination memory module before or following a write request to indicate that the write request contains either frequently written data (e.g., a frequently written data hint) or read-only data (e.g., a read-only data hint).
  • the destination memory module can elect to cache the data or write-through the data to the non-volatile memory.
  • hints can be used to inform memory modules of memory access idle times during which write-through operations can be performed.
  • the example response bus packet 400 of FIG. 4 can be used to send a responsive communication from a memory device to a memory controller or to another memory device.
  • the response bus packet 400 includes a header field 402, a destination select field 404, an optional data field 406, an optional checksum field 408, an optional parity field 410, and an optional ECC field 412.
  • the header field 402 can be used to convey
  • the destination select field 404 can be used to communicate information indicative of a particular destination device (e.g., the memory controller 102) to which the response bus packet 400 is intended.
  • the destination select field 404 may store, for example, a device identifier that uniquely identifies a specific device on a bus (e.g., the memory bus 106). In some instances when only two devices (e.g., the memory controller 102 and the memory 104 of FIG. 1 ) are present on a memory bus (e.g., the memory bus 106), the destination select field 404 may be omitted, ignored, or may always communicate the same information.
  • the response bus packet 400 can include data retrieved from memory in the data field 406.
  • the checksum field 408 is used to store a checksum
  • the parity field 410 is used to store a parity value
  • the ECC field 412 is used to store an ECC value, all of which can be used to detect any errors in the data communicated in the data field 406 and/or other information in the response bus packet 400.
  • a processor 1 16 is shown in communication with the memory controller 102.
  • the processor 1 16 includes a cache memory 1 18 that functions as a temporary quick access storage area for a processor core of the processor 1 16.
  • the cache memory 1 18 is formed of a plurality of cache lines, one of which is denoted as cache line 120.
  • a size of a cache line indicates the number of bytes that may be read from an external memory (e.g., a DRAM) to fill a width of a cache.
  • FIG. 5 depicts an isometric view of an example printed circuit board (PCB) configuration of the memory bus 106 (FIG. 1 ) interconnecting the memory controller 102 (FIG. 1 ) with a plurality of memory devices 104, 502, 504, and 506.
  • the memory controller 102 can operate as a source or destination device and each of the memory devices 104, 502, 504, and 506 can also operate as source or destination devices.
  • the memory devices 104, 502, 504, and 506 can communicate with one another to, for example, transfer memory contents therebetween. For example, during checkpointing processes, contents from one memory device can be transferred or copied to another memory device using memory-to-memory transfer operations.
  • additional memory controllers may also be placed in communication with the memory bus 106, and the memory bus 106 can be operated as a multi-source and multi-destination bus.
  • the memory controllers and the memory devices can communicate bus packets on the memory bus 106 in point-to-point fashion when targeting specific memory controllers or memory devices or in broadcast fashion when targeting a plurality of devices.
  • the memory bus 106 includes a command/request/data (CMD/RQST/DATA) bus 508 and a response/data bus 510.
  • CMD/RQST/DATA command/request/data
  • the CMD/RQST/DATA bus 508 forms an egress communication path used to communicate commands, memory access requests, write data, and/or hints from the memory controller 102 (and/or any other memory controllers on the memory bus 106) to one or more of the memory devices 104, 502, 504, and 506.
  • the response/data bus 510 forms an ingress communication path used to communicate response information including acknowledgements and data from the memory devices 104, 502, 504, and 506 to the memory controller 102 (and/or any other memory controllers on the memory bus 106).
  • the CMD/RQST/DATA bus 508 and the response/data bus 510 enable communications between the memory devices 104, 502, 504, and 506 to perform, for example, memory-to-memory transfers.
  • the memory device 104 may request access to the CMD/RQST/DATA bus 508 to transfer data to the memory device 502 or any of the other memory devices 504, 506.
  • any of the memory devices 104, 502, 504, and 506 may request access to the response/data bus 510 to send responses or data to another one of the memory devices 104, 502, 504, and 506.
  • Memory-to-memory transfers between memory devices may be used to implement direct memory access (DMA) transfers between homogeneous memory technologies (i.e., memory devices of the same type of memory technology) or between heterogeneous memory technologies (i.e., memory devices of different types of memory technologies).
  • DMA direct memory access
  • memory-to-memory transfers may be used to perform data write-through operations from memory technologies having high write endurance (e.g., DRAM memories) used to store frequently changing data to memory technologies having lower write endurance (e.g., flash memories, memristor memories, and/or PCRAM memories) used for storing longer-term persistent data.
  • the memory bus 106 can be implemented using electrical
  • interconnects or optical interconnects can be formed on a PCB using known techniques.
  • Optical interconnects can be formed, for example, as described in U.S. Patent Application No. 1 1/873,325, filed on October 16, 2007, assigned to Hewlett-Packard Development Company, L.P., and titled Optical Interconnect System Providing Communication Between Computer System Components," which is hereby incorporated herein by reference in its entirety.
  • a DRAM PCB in-line memory module 600 (e.g., a dual in-line memory module (DIMM)) is implemented as a multi-chip memory module including four memory chips 602a-d mounted on a PCB 604.
  • the DRAM PCB in-line memory module 600 may be advantageously used in optical interface systems in which the memory module 600 is connected to other subsystems (e.g., other memory modules and/or memory controllers) via optical lines.
  • the memory module 600 may also be used in electrical interface systems.
  • the memory module 600 is provided with memory bus interface pads 606.
  • Each line of the memory bus interface pads 606 may be in communication with one or more of the memory chips 602a-d. In this manner, the memory bus interface pads 606 form a local bus on the memory module 600 to exchange communications between a main memory bus (e.g., the memory bus 106) and the memory chips 602a-d using
  • the memory module 600 can operate using source synchronous clocking or external clocking.
  • the memory bus interface pads 606 are divided into CMD/RQST/DATA bus interface pads 608 to communicate with the CMD/RQST/DATA bus 508 of FIG. 5 and response/data bus interface pads 610 to communicate with the response/data bus 510 of FIG. 5.
  • the memory module 600 is also provided with power and ground pads 612 for interconnecting power and ground to the memory chips 602a-d.
  • the memory module 600 is shown as having four memory chips, example methods, apparatus, and articles of manufacture disclosed herein may be implemented in memory modules having fewer or more chips.
  • the memory bus interface pads 606 can alternatively be implemented using optical interconnect interfaces and the memory module 600 may be provided with local waveguides for routing optical signals between the optical interconnect interfaces and on-board photo- detectors or directly between the optical interconnect interfaces and the memory chips 602a-d.
  • the memory module 600 also includes a module controller 614 in communication between the memory bus 106 and the memory chips 602a-d to filter and arbitrate messages from memory controllers and other memory modules or devices and exchange information between the memory bus 106 and the memory chips 602a-d.
  • a module controller 614 in communication between the memory bus 106 and the memory chips 602a-d to filter and arbitrate messages from memory controllers and other memory modules or devices and exchange information between the memory bus 106 and the memory chips 602a-d.
  • An example implementation of the module controller 614 is described below in connection with FIG. 7.
  • the memory module 600 may be implemented as a multiple memory-type module on which memories of different technology types can be provided.
  • the memory chip 602a may be a volatile SDRAM-type memory and the memory chips 602b-d may be nonvolatile memristor-type memories.
  • the SDRAM-type memory may be used as a local cache for frequently written data because of its low data access times and high write endurance ratings (e.g., write cycle rating). Data from the SDRAM- type memory can be periodically written through to the non-volatile memristor- type memories, which can typically have higher data access times and lower write endurance ratings.
  • the memory chips 602b-d could alternatively be implemented using PCRAM, flash memory, or any other type of memory.
  • each of the memory chips 602a-d could be implemented using a 3D chip stack in which two or more memory dies (e.g., of homogeneous or heterogeneous memory technology types) are stacked (e.g., similar to the 3D stack structure shown in FIG. 12).
  • memory dies e.g., of homogeneous or heterogeneous memory technology types
  • only select ones of the memory chips 602a-d may be implemented using 3D chip stacks.
  • FIG. 12 An example 3D chip stack memory module 1200 is shown in FIG. 12.
  • the 3D chip stack memory module 1200 may be advantageously used in electrical interface systems in which the memory module 1200 is connected to other subsystems (e.g., other memory modules and/or memory controllers) via electrical lines. Alternatively or additionally, the memory module 1200 may be used in optical interface systems.
  • the example 3D chip stack memory module 1200 of FIG. 12 includes a first IC die 1202 stacked on a second IC die 1204, which is stacked on a third IC die 1205.
  • the IC dies 1202, 1204, and 1205 are carried on a ball grid array (BGA) chip package 1206.
  • BGA ball grid array
  • the chip package 1206 is shown as a BGA chip package in the illustrated example, any other suitable type of chip package may be used to implement the example 3D chip stack memory module 1200.
  • the first IC die 1202 is a SDRAM memory core and the second IC die 1204 can be another SDRAM memory core or any other type of memory (e.g., memristor memory, SRAM, flash memory, PCRAM, etc.) or IC (e.g., a processor, a controller, etc.).
  • IC e.g., a processor, a controller, etc.
  • address, control, and data lines of the SDRAM die can be routed directly to the processor or controller die internal to the chip stack package.
  • memory access external from the chip stack package might not occur.
  • address, control, and data lines of the memory IC dies can be routed to external chip interfaces (e.g., BGA pads, surface mount pads, chip leads, etc.).
  • a memory module controller may be stacked with multiple memory die.
  • the IC die 1205 may be a module controller (e.g., similar or identical to the module controller 614).
  • the 3D chip stack memory module 1200 is shown as a BGA package, other types of packages may be used.
  • FIG. 7 is a block diagram of the module controller 614 of FIG. 6.
  • the different portions of the module controller 614 described below can be implemented as integrated circuits within a chip or IC die.
  • the chip or IC die containing the module controller 614 can then be mounted electrically or optically on a PCB (e.g., the PCB 604 of FIG. 6) and/or a 3D chip stack structure (e.g., the 3D chip stack memory module 1200 of FIG. 12) having one or more memory devices to exchange information between the memory bus 106 (FIGS. 1 and 5) and corresponding memory chips (e.g., the memory chips 602a-d of FIG. 6).
  • a PCB e.g., the PCB 604 of FIG. 6
  • 3D chip stack structure e.g., the 3D chip stack memory module 1200 of FIG. 12
  • the module controller 614 is described below in connection with the diagram of FIG. 7, other example implementations are likewise appropriate. For example, additional structures may be added, and
  • the module controller 614 includes a bus data interface 702 to communicatively couple the module controller 614 to the memory bus 106 (FIGS. 1 and 5).
  • the bus data interface 702 may include instate bi-directional buffers to enable receiving/sending information on the memory bus 106 while the module controller 614 is actively communicating on the memory bus 106, and to enable placing bus pins or pads at a high- impedance tri-state level when the module controller 614 is not actively communicating on the memory bus 106.
  • the bus data interface 702 may also include a clock interface in example implementations in which the module controller 614 operates using external clocking. Otherwise, if the module controller 614 operates using source synchronous clocking, the module controller 614 can include a clock source (not shown).
  • the module controller 614 is provided with a message input subsystem 704.
  • the message input subsystem 704 includes a message decoder 706, a message filter 708, an operation decoder 710, and an address decoder 712.
  • the message decoder 706 parses bus packets (e.g., the bus packets 200 and 300 of FIGS. 2 and 3) to identify information in different fields (e.g., the fields 202, 204, 206, 208, 210, and 212 of FIG. 2 and/or the fields 302, 306, and 308 of FIG. 3) of the bus packets.
  • the message filter 708 filters received bus packets by identifying which bus packets are relevant to the memory module 600 (e.g., relevant to the memory chips 602a-d in communication with the module controller 614) and which bus packets can be ignored (e.g., they are relevant to other memory devices on the memory bus 106). This can be done by snooping the headers of packets. For example, the message filter 708 can retrieve memory device identification information (e.g., from the destination select field 204 of FIG. 2 or the destination select field 304 of FIG. 3) from header information (e.g., the headers 202, 302, 402 of FIGS.
  • memory device identification information e.g., from the destination select field 204 of FIG. 2 or the destination select field 304 of FIG. 3
  • header information e.g., the headers 202, 302, 402 of FIGS.
  • a destination select field may be blank or contain a general code indicating that the bus packet is a broadcast packet intended for all memory devices on the same memory bus. If a bus packet is relevant to the memory module 600, the message filter 708 can generate an indication that the received bus packet should be processed (i.e., should not be ignored) by the module controller 614. Otherwise, if the message filter 708 determines that a bus packet is not relevant based on snooping header information, the remainder of the packet can be filtered out at the bus interface 702.
  • the operation decoder 710 retrieves and identifies operation codes from received bus packets (e.g., from the operation code fields 206 and 306 of FIGS. 2 and 3).
  • the address decoder 712 retrieves and decodes addresses from received bus packets (e.g., from the address field 208 of FIG. 2). For example, if the memory chips 602a-d operate internally using row and column addresses, the address decoder 712 can separate address information into row and column addresses.
  • the module controller 614 is provided with a hint logic subsystem 716 that includes a hint decoder 718 and a hint controller 720.
  • the hint decoder 718 receives hint information extracted from bus packets by the message decoder 706 and decodes the hint information to identify different types of hints.
  • the hint controller 720 analyzes the identified hints to determine whether the hints should be acted on or ignored.
  • the hint controller 720 bases such decisions on different factors (or criteria) including whether the memory module 600 is executing a pending memory access request, whether memory access requests are queued up, and/or whether the memory module 600 is or will be processing some other memory operation that may prevent it from acting on a hint.
  • the hint controller 720 can also drop irrelevant hints (e.g., a sleep hint received when memory chips are already in a sleep mode).
  • the module controller 614 is provided with a maintenance controller 722.
  • the maintenance controller 722 determines when to implement pre-charging of bit cells (to enable memory reads), self-refresh operations, low-power mode transitions, wake transitions, etc.
  • the maintenance controller 722 can work in cooperation with the hint logic subsystem 716 to implement certain internal maintenance operations during opportunities identified by the hint logic subsystem 716 based on received hint information.
  • the hint controller 720 can identify an opportunity to the maintenance controller 722 to perform a self- refresh operation, to enter a low-power mode, and/or to perform other maintenance operations.
  • the module controller 614 is provided with a memory device interface 726, which is in communication with the memory chips 602a-d in the illustrated example.
  • the memory device interface 726 can include bi-directional buffers for reading data from and writing data to the memory chips 602a-d and for providing address information to the memory chips 602a-d.
  • the module controller 614 is provided with a data store access arbiter 728.
  • the data store access arbiter 728 can store and/or access a memory operation queue 729 of memory access requests communicated to the memory module 600 and allow servicing of those requests in an orderly manner such as on a first in, first out priority basis.
  • the data store access arbiter 728 also manages and monitors the memory operation queue 729 to determine when the quantity of queued requests exceeds a threshold indicative of when no further memory access requests can be added to the queue 729. For instance, this can happen when the queue is full and can no longer buffer incoming requests.
  • the data store access arbiter 728 of the illustrated example causes a message output subsystem 736 to respond to subsequent memory access requests received via the bus data interface 702 with negative acknowledgements until the quantity of requests in the queue 729 falls below the threshold.
  • the negative acknowledgements can indicate to one or more requesting device(s) (e.g., the memory controller 102 of FIGS. 1 , 5, and 8 or any of the memory devices 104, 502, 504, and 506 of FIGS. 1 and 5) that the memory access requests cannot be granted.
  • the negative acknowledgements also indicate that the requesting device(s) should re-send the memory access request to the memory module 600 at some later time (e.g., a time that may or may not be specified by the data store access arbiter 728 or the module controller 614). In some examples, the negative acknowledgements cause the requesting device(s) to resend the memory access request to the memory module 600 at some later time (e.g., a time that may or may not be specified by the data store access arbiter 728 or the module controller 614).
  • the module controller 614 is provided with a data store status monitor 730.
  • the data store status monitor 730 tracks when operations are being performed on the memory chips 602a-d including read/write operations, self-refresh operations, pre-charge operations, power down operations, etc.
  • the data store status monitor 730 tracks when the memory chips 602a-d are in a sleep mode, standby, or other low-power mode.
  • the data store status monitor 730 can determine whether the memory chips 602a-d are busy or idle and whether queued or recently received operations can be performed immediately or need to be delayed until other operations are completed.
  • the data store status monitor 730 can exchange information with the data store access arbiter 728, the maintenance controller 722, and the hint logic subsystem 716.
  • the module controller 614 is provided with a message output subsystem 732.
  • the message output subsystem 732 may generate the response bus packet 400 of FIG. 4 for communicating data to a memory controller (e.g., the memory controller 102 of FIGS. 1 and 5) in response to a read request from that memory controller.
  • the message output subsystem 732 includes a destination selector 734 and a message generator 736.
  • the destination selector 734 selects unique identifications of destination devices (e.g., the memory controller 102 of FIGS. 1 and 5) to which bus packets are to be communicated and stores unique identifications in destination select fields (e.g., the destination select field 404 of FIG.
  • the destination selector 734 can store a data structure of device identifier codes, each of which corresponds to a respective device connected to the module controller 614 via the memory bus 106.
  • the destination selector 734 can receive device identifier codes from the data store access arbiter 728 in connection with the message output subsystem 732 receiving data from the data store interface 726. In this manner, a device identifier code received from the data store access arbiter 728 could be used to indicate the device that requested data received from the data store interface 726.
  • the message generator 736 forms the response bus packet 400 by, for example, concatenating information from the header field 402, the destination select field 404, the data field 406, and the checksum field 408.
  • the message generator 736 may be configured to generate checksums, parity values, and ECC values for read data, while in other example implementations, the data store interface 726 may be configured to generate such information.
  • the module controller 614 of the illustrated example is provided with a bus request line 738.
  • the bus request line 738 is external to the module controller 614 and is configured to be connected to the memory controller 102 to request that the memory controller 102 grant the memory module 600 access to the memory bus 106.
  • the memory module 600 can send data and/or messages to other devices (e.g., to the memory controller 102 and/or to other memory devices such as the memory devices 104, 502, 504, and/or 506 of FIGS. 1 and 5) via the memory bus 106 without causing collisions or bus contention on the memory bus 106.
  • the bus request line 738 is internally connected to the message output subsystem 732. In this manner, when the message output subsystem 732 is ready to
  • the message output subsystem 732 of the illustrated example causes a signal assertion on the bus request line 738 to request access to the memory bus 106.
  • the bus request line 738 is a bi-directional line via which the memory controller 102 can respond to grant memory bus access.
  • the bus request line 738 is a unidirectional output line from the module controller 614 and the memory controller 102 sends bus access grant responses to the module controller 614 via the memory bus 106.
  • the other memory modules are also provided with respective bus request lines similar or identical to the bus request line 738 of FIG. 7 to allow the other memory modules to request access to the memory bus 106.
  • FIG. 8 is a block diagram of the example memory controller 102 (FIGS. 1 and 5).
  • the different portions of the example memory controller 102 described below can be implemented as integrated circuits within a single IC die as a stand-alone memory controller or as a memory controller embedded in a processor IC die. Alternatively, some portions of the memory controller 102 can be implemented as integrated circuits on one or more separate integrated circuit dies. Additionally or alternatively, the memory controller 102 may be
  • the memory controller 102 may be implemented using any combination(s) of application specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)), field programmable logic device(s) (FPLD(s)), discrete logic, hardware, firmware, etc.
  • ASIC application specific integrated circuit
  • PLD programmable logic device
  • FPLD field programmable logic device
  • the memory controller 102 may be implemented as any combination(s) of any of the foregoing techniques, for example, any combination of firmware, software, discrete logic and/or hardware.
  • the memory controller 102 is described below in connection with the diagram of FIG. 8, other example implementations may include additional and/or alternative structures. For example, some of the portions of the memory controller 102 depicted in FIG. 8 may be eliminated or combined with other portions.
  • the memory controller 102 includes a bus data interface 802 to communicatively couple the memory controller 102 to the memory bus 106 (FIGS. 1 , 5, and 7).
  • the bus data interface 802 may include tri-state bi-directional buffers to enable receiving/sending information on the memory bus 106 while the memory controller 102 is actively communicating on the memory bus 106, and to enable placing bus pins or pads at a high- impedance tri-state level when the memory controller 102 is not actively communicating on the memory bus 106.
  • a processor e.g., the processor 1 16 of FIG. 1
  • the memory controller 102 is provided with a processor bus interface 804.
  • the memory controller 102 is provided with a hint generator 806.
  • the hint generator 806 can generate hints based on the receipt of memory access requests from a processor (e.g., the processor 1 16 of FIG. 1 ) or based on status information received from the processor.
  • the memory controller 102 can generate a hint informing one or more memory devices that the memory controller 102 will not access the memory device(s) for a particular time period, length of time, or duration (e.g., thereby permitting the memory device(s) to perform one or more internal processes such as self-refreshing, entering a low- power mode, etc.).
  • the hint generator 806 can receive different control or status information from a connected processor indicating the operating mode of the processor such as active, standby, sleep, deep-sleep, powered-off.
  • the hint generator 806 can generate hint information based on the operating modes of the processor. For instance, if the processor is in a deep-sleep mode, the hint generator 806 can generate a hint informing one or more memory devices that they can transition into a very low-power mode (e.g., a deep-sleep mode).
  • a very low-power mode e.g., a deep-sleep mode
  • the memory controller 102 is provided with a bus arbiter 810.
  • the bus arbiter 810 can be used to control access to the memory bus 106 and identify when the memory bus 106 is available to transmit
  • bus arbiter 810 can send out a negative acknowledgement message to the source device asking for redelivery.
  • the bus arbiter 810 can keep track of requests pending at various memory modules on the memory bus 106 and ensure the availability of input buffers of those memory modules before allowing the memory controller 102 to issue a request.
  • the bus arbiter 810 may be implemented for use on an electrical- based memory bus or an optical memory bus.
  • An example optical memory bus for which the bus arbiter 810 can be used involves use of an optical token channel in which a token is circulated on a bus for use in claiming exclusive use of the bus at different times by different devices.
  • the bus arbiter 810 can track when a token is circulating on the memory bus 106 to identify when the bus is available and when it is in use by another device.
  • the memory controller 102 is provided with a message output subsystem 808.
  • the message output subsystem 808 may generate the read/write bus packet 200 of FIG. 2 for communicating data access requests to one or more memory devices (e.g., the DRAM memory 104 of FIGS. 1 , 5, and 7) and the hint bus packet 300 of FIG. 3 for communicating hints to one or more memory devices.
  • the message output subsystem 808 includes an address generator (not shown), a message generator 812, and a device selector 814.
  • the message generator 812 forms bus packets (e.g., the read/write bus packet 200 and/or the hint bus packet 300). In some example, the read/write bus packet 200 and/or the hint bus packet 300.
  • the message generator 812 may be configured to generate checksums, parity values, and error correction codes for write data (e.g., data in the data field 210 of FIG. 2), while in other example implementations, checksums, parity values, and error correction codes may be provided by a processor connected to the memory controller 102.
  • the message generator 812 can split an address into two or more separately transferrable portions so that a lengthy address can be transferred to a memory device using two or more separate bus packets.
  • the device selector 814 selects a unique identification of a memory device (e.g., one of the memory devices 104, 502, 504, and 506 of FIG. 5) to which a bus packet is to be communicated and stores the unique identification in a chip/device select field (e.g., the destination select field 204 of FIG. 4) of the bus packet.
  • the device selector 814 can store a data structure of memory device identifier codes, each of which corresponds to a respective memory device connected to the memory controller 102 via the memory bus 106.
  • the device selector 814 can receive memory device identifier codes based on different physical memory slots occupied by different memory devices.
  • the device selector 814 may be configured to assign unique identifier codes to each memory device detected on the memory bus 106. In this manner, the device selector 814 can
  • the device selector 814 can identify a destination device based on a physical address of a memory request.
  • the memory controller 102 can include a device ID-to-address data structure 816 storing cross-references between device ID's and respective memory address ranges.
  • the device selector 814 can use the device ID-to- address data structure 816 to identify a destination device based on an address in a memory access request.
  • the memory controller 102 To decode and filter messages or bus packets received from the memory bus 106, the memory controller 102 is provided with a message input subsystem 818.
  • the message input subsystem 818 includes a message decoder 820 and a message filter 822.
  • the message decoder 820 parses bus packets (e.g., the response bus packet 400 of FIG. 4) to identify information in different fields (e.g., the fields 402, 404, 406, and 408 of FIG. 4) of the bus packets.
  • the message filter 822 filters received bus packets by identifying which bus packets are relevant to the memory controller 102 and which bus packets can be ignored (e.g., they may be relevant to other devices on the memory bus 106 but are not relevant to the memory controller associated with the message filter 822). For example, the message filter 822 can retrieve device identification information (e.g., from the destination select field 404 of FIG. 4) from bus packets and compare the retrieved device identification information to a unique identification value of the memory controller 102 to determine whether bus packets are relevant to the memory controller 102. If a bus packet is relevant to the memory controller 102, the message filter 822 can generate an indication that the received bus packet should be processed (i.e., should not be ignored) by the memory controller 102.
  • device identification information e.g., from the destination select field 404 of FIG. 4
  • the message filter 822 can generate an indication that the received bus packet should be processed (i.e., should not be ignored) by the memory controller 102.
  • the memory controller 102 To buffer data from a processor or from memory devices, the memory controller 102 is provided with a data buffer 824.
  • the data buffer 824 stores data received from a processor to be written to memory in response to a write request and stores data from memory to be
  • the message generator 812 can retrieve
  • the message decoder 820 can parse data from a data field (e.g., the data field 406 of FIG. 4) of the response bus packet and store the data in the data buffer 824 for subsequent communication to the requesting processor.
  • the memory controller 102 of the illustrated example is provided with bus request lines 826.
  • the bus request lines 826 are external to the memory controller 102 and are configured to be connected to memory devices or memory modules (e.g., the memory devices 104, 502, 504, and 506 of FIGS. 1 and 5) or other memory controllers or processors that request access to the memory bus 106 to exchange communications with the memory controller 102 and/or with one another via the memory bus 106.
  • the memory devices or memory modules and/or the memory controller 102 can send data and/or messages to one another via the memory bus 106 without causing collisions or bus contention on the memory bus 106.
  • the memory controller having a bus arbiter (e.g., the bus arbiter 810), is the master memory controller.
  • all of the memory controllers may have a bus arbiter, but only the memory controller designated as the master memory controller will enable its bus arbiter and the slave memory controllers will disable or not use their bus arbiters so that multiple memory controllers will not contend with managing use of the memory bus 106.
  • the bus request lines 826 are internally connected to the bus arbiter 810.
  • the bus arbiter 810 of the illustrated example determines whether to and/or when to grant access to the memory bus 106. For example, the bus arbiter 810 may grant access to the memory bus 106 based on statuses of the memory bus 106 such as statuses of when the memory bus 106 is busy (e.g., the memory bus 106 is being used by another memory controller, a memory module or memory device, a processor, etc.). In some examples, the bus arbiter 810 may include or be in communication with a bus access queue 828 to store bus access requests and grant memory bus access based on the queued requests.
  • the bus request lines 826 are bi-directional lines via which the memory controller 102 can respond to grant memory bus access. In other examples, the bus request lines 826 are unidirectional input lines into the memory controller 102 and the memory controller 102 sends bus access grant responses to requesting devices via the memory bus 106.
  • FIGS. 9A and 9B depict a flow diagram of an example process that can be executed by memory modules to process memory access requests and hint information.
  • the example process is described in connection with the memory module 600 of FIGS. 6 and 7, but may alternatively be implemented using other memories (e.g., the memory devices 502, 504, and 506 of FIG. 5 and/or the memory module 1200 of FIG. 12) and/or other types of memories.
  • FIGS. 9A and 9B are described below in connection with the flow diagram as depicted, some examples employ different operations in addition to or instead of the operations of FIGS. 9A and 9B. For instance, some operations of FIGS. 9A and 9B may be omitted or combined with other operations or performed in a different order or in parallel with other operations.
  • the message input subsystem 704 receives a bus packet from a source device such as, for example, the memory controller 102 (FIGS. 1 , 5, and 8) (block 902) (FIG. 9A).
  • the source device may be implemented using any memory device (e.g., any of the memory devices 104, 502, 504, and 506 of FIGS. 1 and 5) on the same memory bus (e.g., the memory bus 106) as the example memory module 600 and request to perform an inter-memory-module memory-to-memory data transfer to, for example, write data to the memory module 600.
  • the bus packet may be received via the bus data interface 702.
  • the source device may be a memory chip (e.g., one of the memory chips 602a-d of FIG. 6) or IC die (e.g., one of the IC dies 1202, 1204, 1205 of FIG. 12) located on the same memory module as the message input subsystem 704, and the bus packet may contain a request to perform an intra-memory-module memory- to-memory data transfer to, for example, write data between memories within the same memory module 600 of FIG. 6 or within the same memory module 1200 of FIG. 12.
  • the bus packet may be received via the data store interface 726.
  • block 902 is described as involving receipt of a bus packet, a communication received at block 902 may be received using a communication other than a bus packet communication.
  • the message decoder 706 parses the memory device identification information from the received bus packet (block 904).
  • the message decoder 706 can retrieve a memory device identifier from a device select field (e.g., the destination select field 204 of FIG. 2 or the destination select field 304 of FIG. 3).
  • the message filter 708 determines whether the received bus packet is relevant to the memory module 600 (block 906) by, for example, comparing the memory device identifier retrieved from the bus packet with a memory device identifier of the memory module 600.
  • a destination select field may be blank or contain a general code indicating that the bus packet is a broadcast packet intended for all memory devices on the memory bus 106.
  • the message filter 708 determines whether the bus packet is relevant based on whether an address communicated in the bus packet (and decoded by the address decoder 712 (FIG. 7)) falls within a memory address range assigned to the memory module 600.
  • the module controller 614 ignores the bus packet (block 910). However, if the received bus packet is relevant (block 906), the message decoder 706 continues decoding the bus packet (block 912).
  • the operation decoder 710 determines whether the bus packet contains hint information (block 914). For example, an operation code in an operation code field (e.g., the operation code field 306 of FIG. 3) or a code in a header field (e.g., the header field 302 of FIG. 3) can be used to identify the bus packet as a hint bus packet (e.g., the hint bus packet 300 of FIG. 3).
  • the module controller 614 processes the hint information (block 916).
  • An example process that can be used to implement block 916 is described below in connection with FIG. 10.
  • the operation decoder 710 determines that the bus packet does not contain hint information (block 914)
  • the operation decoder 710 decodes the requested operation from the bus packet (block 918), for example, based on an operation code in an operation code field (e.g., the operation code field 202 of FIG. 2).
  • the requested operation can be a read operation, a write operation, or some variation thereof (e.g., a burst read, a page-mode read, etc.).
  • the address decoder 712 decodes an address from the bus packet (block 920) (e.g., an address stored in the address field 208 of FIG. 2). In example implementations in which an address of a target storage location is transferred using two bus packets, the address decoder 712 can decode the address information from two corresponding bus packets to form an address useable for accessing the target storage location of the memory chips 602a-d (FIGS. 6 and 7).
  • the data store status monitor 730 determines whether the memory chips 602a-d are currently performing a read operation, a write operation, a self- refresh operation, a pre-charge operation, a power down operation or are currently in a sleep mode, a standby mode, or other low-power mode. If the memory chips 602a-d are performing any of these operations (or any other operations), the memory chips 602a-d are busy and cannot immediately perform the requested operation decoded at block 918. Otherwise, if the memory chips 602a-d are idle (i.e., not performing any operations that indicate they are busy), the memory chips 602a-d are not busy and can immediately perform the requested operation decoded at block 918.
  • the data store access arbiter 728 determines whether the memory operation queue 729 of the memory module 600 is too full to add another queued memory access request (block 926). For example, the data store access arbiter 728 may determine that the memory operation queue 729 is too full if the quantity of queued requests exceeds a threshold indicative of when no further memory access requests can be added to the queue 729.
  • the data store access arbiter 728 can determine whether it can service the requested memory access operation based on the number of access requests in the queue 729 without relying on a busy status of the memory chips 602a-d determined at block 924 based on the status of the memory chips 602a-d determined at block 922.
  • the operations of blocks 922 and 924 may be omitted.
  • the data store access arbiter 728 of the illustrated example prompts or causes the message output subsystem 732 to respond to the bus packet received at block 902 with a negative acknowledgement (block 928) via, for example, a bus packet communication including an identification of the requesting device.
  • the negative acknowledgement indicates that the memory access request cannot be granted (or cannot be serviced).
  • the negative acknowledgement causes the requesting device to re- send the memory access request to the memory module 600 at some later time (e.g., a time that may or may not be indicated by the data store access arbiter 728 or the module controller 614).
  • the negative acknowledgement causes the requesting device to re- send the memory access request to the memory module 600 at some later time (e.g., a time that may or may not be indicated by the data store access arbiter 728 or the module controller 614).
  • the negative acknowledgement causes the requesting device to re- send the memory access request to the memory module 600 at
  • acknowledgement can explicitly indicate that the requesting device should re- send the memory access request to the memory module 600 at some later time.
  • the data store access arbiter 728 determines whether the requested operation has been reached in the memory operation queue 729 for servicing (block 932). If the requested operation has not been reached in the memory operation queue 729, the data store access arbiter 728 continues to monitor the queue 729 at block 932 to determine when the requested operation is reached in the memory operation queue 729 for servicing.
  • the data store access arbiter 728 of the illustrated example causes the requested read or write operation(s) to be performed (block 934).
  • the data store access arbiter 728 can queue the memory access operation if other memory access requests are still being performed or if other maintenance operations are being performed on the memory chips 602a-d such that the memory chips 602a-d cannot immediately be accessed to perform another memory access operation.
  • the operation of block 934 can be an immediate or a delayed performance of the read or write operation(s) as determined by the data store access arbiter 728.
  • the destination selector 734 selects a device identification (block 936) to identify the requesting memory controller (e.g., the memory controller 102) for which a response bus packet (e.g., the response bus packet 400 of FIG. 4) is to be communicated.
  • the message generator 736 (FIG. 7) generates the response bus packet (block 938) including the device identification. If the requested memory access operation were a read request, the response bus packet can include the data retrieved from the memory chips 602a-d in a data field (e.g., the data field 406 of FIG.
  • the message generator 736 can also store a corresponding checksum in a checksum field (e.g., the checksum field 408 of FIG.4). If the requested memory access operation were a write request, the response bus packet can include an acknowledgement message acknowledging a successful write. The acknowledgement message could be included in a header field (e.g., the header field 402 of FIG. 4).
  • the message output subsystem 732
  • FIGS. 9A and 9B After communicating the response bus packet (block 940) or after ignoring the received bus packet (block 910) (FIG. 9A) or after processing the hint information (block 916) (FIG. 9A), the example process of FIGS. 9A and 9B is ended.
  • FIG. 10 depicts a flow diagram of an example process that can be implemented in memory modules to process received hint information.
  • the example process is described in connection with the memory module 600 of FIGS. 6 and 7, but may alternatively be implemented using other memories (e.g., the memory devices 502, 504, and 506 of FIG. 5 and/or the memory module 1200 of FIG. 12) and/or other types of memories.
  • FIG. 10 is described below in connection with the flow diagram as depicted, some examples may employ different operations in addition to or instead of the operations of FIG. 10. For instance, some operations of FIG. 10 may be omitted or combined with other operations or performed in a different order or in parallel with other operations.
  • the hint logic subsystem 716 (FIG.
  • the hint decoder 718 decodes the hint information (block 1004).
  • the hint controller 720 obtains the data store status (block 1006) of the memory chips 602a-d (FIGS. 6 and 7) from the data store status monitor 730 (FIG. 7). In this manner, the hint logic subsystem 716 can determine whether the memory module 600 can act on the hint. For example, if the memory chips 602a-d are busy with memory access requests (or
  • the hint controller 720 can determine that the memory module 600 cannot perform the hinted operation. In some instances, the hint controller 720 may determine that it can queue a hinted operation if the status of the memory chips 602a-d indicates that they are currently busy (e.g., with a memory access or maintenance operation) but that the operation will be finished soon and there are no other memory access requests queued by the data store access arbiter 728.
  • the hint controller 720 determines that the memory module 600 cannot act on the hint (block 1008), the hint logic subsystem 716 ignores the hint (block 1010). Otherwise, if the hint controller 720 determines that the memory module 600 can act on the hint (block 1008), the hint controller 720 determines whether the memory module 600 can act immediately (block 1012). If the memory module 600 can act immediately on the hint (block 1012), the memory module 600 immediately performs the hinted operation (block 1014). For example, the hint controller 720 can instruct the maintenance controller 722 to perform the hinted operation. If the memory module 600 cannot act immediately on the hint (block 1012), the hint controller 720 queues the hinted operation (e.g., in the memory operation queue 729 of FIG.
  • FIG. 1 1 depicts a flow diagram of an example process that can be implemented in a memory controller to generate hint information.
  • the example process of FIG. 1 1 is described in connection with the memory controller 102 of FIGS. 1 , 5, and 8, but may alternatively be implemented using other memory controllers or devices that access memory devices on memory buses such as the memory bus 106 of FIGS. 1 , 5, 7, and 8.
  • FIG. 1 1 is described below in connection with the flow diagram as depicted, some examples employ different operations in addition to or instead of the operations of FIG. 1 1. For instance, some operations of FIG. 1 1 may be omitted or combined with other operations or performed in a different order or in parallel with other operations without departing from the scope and spirit of this application.
  • the hint generator 806 determines the status of a connected processor (e.g., the processor 1 16 of FIG. 1 ) (block 1 102) to identify whether the processor is in an active state, an idle state, or in a low-power mode (e.g., sleep, deep-sleep, powered down, etc.).
  • the hint generator 806 also determines the status of memory access requests (block 1 104) received from the connected processor to identify whether any memory access requests are currently being processed or still need to be processed.
  • the hint generator 806 determines whether to generate a hint (block 1 106). For example, the hint generator 806 can determine that a hint can be generated if the processor is in an idle or low-power mode or if there are no pending memory access requests from the processor. If the hint generator 806 determines that it can generate a hint (block 1 106), the hint generator 806 determines hinted operation to generate (block 1 108). For example, if the processor is in an idle or low-power mode, the hinted operation may inform one or more memory devices that they can enter a low-power mode. Or, if there are no pending requests but the processor is active, the hinted operation can be a self-refresh operation.
  • the hint may indicate a duration for an idle time of the memory controller 102 (e.g., a duration for which no memory access requests will be made by the memory controller 102), and the one or more memory devices can use the indicated idle time duration to determine which internal operation(s) it can perform during the idle time duration.
  • the message generator 812 (FIG. 8) generates a hint bus packet (block 1 1 10) (e.g., the hint bus packet 300 of FIG. 3). If the hint is to be communicated to a single memory device (e.g., the DRAM memory 104), the device selector 814 (FIG. 8) can select the memory device identifier of the single memory device, and the message generator 812 can store the memory device identifier and the hinted operation in the hint bus packet. If the hint is a general hint applicable to all memory devices on the memory bus 106, a memory device identifier can be omitted from the hint bus packet and the hint bus packet can be communicated as a general broadcast bus packet.
  • a hint bus packet (block 1 1 10) (e.g., the hint bus packet 300 of FIG. 3). If the hint is to be communicated to a single memory device (e.g., the DRAM memory 104), the device selector 814 (FIG. 8) can select the memory device identifier of the single memory device,
  • the message output subsystem 808 communicates the hint bus packet on the memory bus 106 (block 1 1 12). After the hint bus packet is communicated (block 1 1 12) or if no hint is to be generated (block 1 106), the example process of FIG. 1 1 is ended.
  • one or more of the example processes of FIGS. 9A, 9B, 10, and/or 1 1 may be implemented using machine readable instructions that, when executed, cause a device (e.g., a
  • FIGS. 9A, 9B, 10, and/or 1 1 may be performed using a processor, a controller, and/or any other suitable processing device.
  • the example processes of FIGS. 9A, 9B, 10, and/or 1 1 may be implemented in coded instructions stored on a tangible machine readable medium such as a flash memory, a read-only memory (ROM), and/or a random- access memory (RAM) associated with a processor or controller.
  • a tangible machine readable medium such as a flash memory, a read-only memory (ROM), and/or a random- access memory (RAM) associated with a processor or controller.
  • the term tangible computer readable medium is expressly defined to include any type of computer readable storage and to exclude propagating signals.
  • FIGS. 9A, 9B, 10, and/or 1 1 may be implemented using coded instructions (e.g., computer readable instructions) stored on a non-transitory computer readable medium such as a flash memory, a read-only memory (ROM), a random-access memory (RAM), a cache, or any other storage media in which information is stored for any duration (e.g., for extended time periods, permanently, brief instances, for temporarily buffering, and/or for caching of the information).
  • a non-transitory computer readable medium such as a flash memory, a read-only memory (ROM), a random-access memory (RAM), a cache, or any other storage media in which information is stored for any duration (e.g., for extended time periods, permanently, brief instances, for temporarily buffering, and/or for caching of the information).
  • a non-transitory computer readable medium such as a flash memory, a read-only memory (ROM), a random-access memory (RAM), a cache, or any other storage media in which
  • FIGS. 9A, 9B, 10, and/or 1 1 may be implemented using any combination(s) of application specific integrated circuit(s) (ASIC(s)), programmable logic device(s) (PLD(s)), field programmable logic device(s) (FPLD(s)), discrete logic, hardware, firmware, etc. Also, the example processes of FIGS. 9A, 9B, 10, and/or 1 1 may be implemented as any combination(s) of any of the foregoing techniques, for example, any
  • FIGS. 9A, 9B, 10, and/or 1 1 are described with reference to the flow diagrams of FIGS. 9A, 9B, 10, and/or 1 1 , other methods of implementing the processes of FIGS. 9A, 9B, 10, and/or 1 1 may be employed.
  • the order of execution of the blocks may be changed, and/or some of the blocks described may be changed, eliminated, sub-divided, or combined.
  • any or all of the example processes of FIGS. 9A, 9B, 10, and/or 1 1 may be performed sequentially and/or in parallel by, for example, separate processing threads, processors, devices, discrete logic, circuits, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Techniques For Improving Reliability Of Storages (AREA)
  • Memory System (AREA)
  • Dram (AREA)

Abstract

Selon l'invention, un appareil utilisé à titre d'exemple comprend une interface (702, 726) conçue pour recevoir une demande d'accès à une mémoire (602a) d'un module mémoire (600) et un moniteur d'état de magasin de données (730) pour déterminer un état de la mémoire. L'appareil cité à titre d'exemple comprend également un sous-système d'émission de message (732) qui permet, lorsque la mémoire est occupée, de répondre à la demande par un accusé de réception négatif indiquant que la demande d'accès à la mémoire ne peut être accordée.
PCT/US2011/022762 2010-01-28 2011-01-27 Procédés et appareil faisant intervenir une interface pour des dispositifs mémoire WO2011094436A2 (fr)

Priority Applications (4)

Application Number Priority Date Filing Date Title
CN201180003837.6A CN102576338B (zh) 2010-01-28 2011-01-27 用于存储器设备的接口方法和装置
US13/386,396 US8938589B2 (en) 2010-01-28 2011-01-27 Interface methods and apparatus for memory devices using arbitration
EP11737662.4A EP2529312A4 (fr) 2010-01-28 2011-01-27 Procédés et appareil faisant intervenir une interface pour des dispositifs mémoire
US14/564,215 US20160246711A9 (en) 2010-01-28 2014-12-09 Interface methods and apparatus for memory devices

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US29915810P 2010-01-28 2010-01-28
US61/299,158 2010-01-28

Related Child Applications (2)

Application Number Title Priority Date Filing Date
US13/386,396 A-371-Of-International US8938589B2 (en) 2010-01-28 2011-01-27 Interface methods and apparatus for memory devices using arbitration
US14/564,215 Division US20160246711A9 (en) 2010-01-28 2014-12-09 Interface methods and apparatus for memory devices

Publications (2)

Publication Number Publication Date
WO2011094436A2 true WO2011094436A2 (fr) 2011-08-04
WO2011094436A3 WO2011094436A3 (fr) 2011-11-24

Family

ID=44320123

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2011/022762 WO2011094436A2 (fr) 2010-01-28 2011-01-27 Procédés et appareil faisant intervenir une interface pour des dispositifs mémoire

Country Status (3)

Country Link
EP (1) EP2529312A4 (fr)
CN (1) CN102576338B (fr)
WO (1) WO2011094436A2 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103488574A (zh) * 2012-06-12 2014-01-01 Ls产电株式会社 用于存储器共享的电路
US8938589B2 (en) 2010-01-28 2015-01-20 Hewlett-Packard Development Company, L. P. Interface methods and apparatus for memory devices using arbitration
US9361955B2 (en) 2010-01-28 2016-06-07 Hewlett Packard Enterprise Development Lp Memory access methods and apparatus

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9116856B2 (en) * 2012-11-08 2015-08-25 Qualcomm Incorporated Intelligent dual data rate (DDR) memory controller
US9411774B2 (en) * 2013-04-23 2016-08-09 Arm Limited Memory access control
US9324397B1 (en) * 2015-01-16 2016-04-26 Qualcomm Incorporated Common die for supporting different external memory types with minimal packaging complexity
US10402349B2 (en) * 2017-02-08 2019-09-03 Arm Limited Memory controller having data access hint message for specifying the given range of one or more memory addresses

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0718770A1 (fr) 1994-12-20 1996-06-26 Motorola, Inc. Procédé et appareil de communication entre des dispositifs électroniques maître et esclave dans lesquels l'accès à l'esclave peut être destructeur
US6052772A (en) 1998-04-13 2000-04-18 International Business Machines Corporation Memory request protocol method
US6523098B1 (en) 1999-12-22 2003-02-18 Intel Corporation Mechanism for efficient low priority write draining
US20080005479A1 (en) 2006-05-22 2008-01-03 International Business Machines Corporation Systems and methods for providing remote pre-fetch buffers

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6360285B1 (en) * 1994-06-30 2002-03-19 Compaq Computer Corporation Apparatus for determining memory bank availability in a computer system
US7020741B1 (en) * 2003-04-29 2006-03-28 Advanced Micro Devices, Inc. Apparatus and method for isochronous arbitration to schedule memory refresh requests
US7213082B2 (en) * 2004-03-29 2007-05-01 Micron Technology, Inc. Memory hub and method for providing memory sequencing hints
US7426613B2 (en) * 2005-06-16 2008-09-16 Lexmark International, Inc. Addressing, command protocol, and electrical interface for non-volatile memories utilized in recording usage counts

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0718770A1 (fr) 1994-12-20 1996-06-26 Motorola, Inc. Procédé et appareil de communication entre des dispositifs électroniques maître et esclave dans lesquels l'accès à l'esclave peut être destructeur
US6052772A (en) 1998-04-13 2000-04-18 International Business Machines Corporation Memory request protocol method
US6523098B1 (en) 1999-12-22 2003-02-18 Intel Corporation Mechanism for efficient low priority write draining
US20080005479A1 (en) 2006-05-22 2008-01-03 International Business Machines Corporation Systems and methods for providing remote pre-fetch buffers

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP2529312A4

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8938589B2 (en) 2010-01-28 2015-01-20 Hewlett-Packard Development Company, L. P. Interface methods and apparatus for memory devices using arbitration
US9361955B2 (en) 2010-01-28 2016-06-07 Hewlett Packard Enterprise Development Lp Memory access methods and apparatus
CN103488574A (zh) * 2012-06-12 2014-01-01 Ls产电株式会社 用于存储器共享的电路
US9218306B2 (en) 2012-06-12 2015-12-22 Lsis Co., Ltd. Memory sharing circuit employing a buffered address and data bus and preventing bus collision

Also Published As

Publication number Publication date
WO2011094436A3 (fr) 2011-11-24
CN102576338B (zh) 2015-06-10
EP2529312A4 (fr) 2013-07-03
EP2529312A2 (fr) 2012-12-05
CN102576338A (zh) 2012-07-11

Similar Documents

Publication Publication Date Title
US8938589B2 (en) Interface methods and apparatus for memory devices using arbitration
US20150095601A1 (en) Interface methods and apparatus for memory devices
JP6139010B2 (ja) デバイス
EP2529312A2 (fr) Procédés et appareil faisant intervenir une interface pour des dispositifs mémoire
TW200527427A (en) Memory buffer device integrating refresh
US11237903B2 (en) Technologies for providing ECC pre-provisioning and handling for cross-point memory and compute operations
JP2014508361A (ja) メモリ・インターフェース
CN115083451A (zh) 多通道的数据处理方法、装置、设备及存储介质
EP3716076A1 (fr) Technologies pour fournir une architecture informatique haute efficacité sur une mémoire de point de croisement pour des opérations d'intelligence artificielle
US20240071461A1 (en) Adaptive Memory Registers
JP6370958B2 (ja) デバイス
US20230195368A1 (en) Write Request Buffer
JP5932261B2 (ja) メモリ制御装置、メモリ制御方法
US8473682B2 (en) Cache unit and processing system
US20240070096A1 (en) Erroneous Select Die Access (SDA) Detection
US11836096B2 (en) Memory-flow control register
US20230343381A1 (en) Bank-Level Self-Refresh
US20240170038A1 (en) Adaptive Refresh Staggering
US20230343380A1 (en) Bank-Level Self-Refresh
US20240070093A1 (en) Asymmetric Read-Write Sequence for Interconnected Dies

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 201180003837.6

Country of ref document: CN

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 11737662

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2011737662

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 13386396

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE