Classifications

• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11C—STATIC STORES
  • G11C16/00—Erasable programmable read-only memories
  • G11C16/02—Erasable programmable read-only memories electrically programmable
  • G11C16/06—Auxiliary circuits, e.g. for writing into memory
  • G11C16/34—Determination of programming status, e.g. threshold voltage, overprogramming or underprogramming, retention
  • G11C16/349—Arrangements for evaluating degradation, retention or wearout, e.g. by counting erase cycles
  • G11C16/3495—Circuits or methods to detect or delay wearout of nonvolatile EPROM or EEPROM memory devices, e.g. by counting numbers of erase or reprogram cycles, by using multiple memory areas serially or cyclically
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11C—STATIC STORES
  • G11C16/00—Erasable programmable read-only memories
  • G11C16/02—Erasable programmable read-only memories electrically programmable
  • G11C16/06—Auxiliary circuits, e.g. for writing into memory
  • G11C16/34—Determination of programming status, e.g. threshold voltage, overprogramming or underprogramming, retention
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
  • G06F12/02—Addressing or allocation; Relocation
  • G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
  • G06F12/10—Address translation
  • G06F12/1009—Address translation using page tables, e.g. page table structures
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11C—STATIC STORES
  • G11C16/00—Erasable programmable read-only memories
  • G11C16/02—Erasable programmable read-only memories electrically programmable
  • G11C16/06—Auxiliary circuits, e.g. for writing into memory
  • G11C16/10—Programming or data input circuits
  • G11C16/14—Circuits for erasing electrically, e.g. erase voltage switching circuits
  • G11C16/16—Circuits for erasing electrically, e.g. erase voltage switching circuits for erasing blocks, e.g. arrays, words, groups
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11C—STATIC STORES
  • G11C7/00—Arrangements for writing information into, or reading information out from, a digital store
  • G11C7/04—Arrangements for writing information into, or reading information out from, a digital store with means for avoiding disturbances due to temperature effects
• • H—ELECTRICITY
  • H10—SEMICONDUCTOR DEVICES; ELECTRIC SOLID-STATE DEVICES NOT OTHERWISE PROVIDED FOR
  • H10B—ELECTRONIC MEMORY DEVICES
  • H10B41/00—Electrically erasable-and-programmable ROM [EEPROM] devices comprising floating gates
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
  • G06F2212/10—Providing a specific technical effect
  • G06F2212/1016—Performance improvement
  • G06F2212/1024—Latency reduction
• • H—ELECTRICITY
  • H10—SEMICONDUCTOR DEVICES; ELECTRIC SOLID-STATE DEVICES NOT OTHERWISE PROVIDED FOR
  • H10B—ELECTRONIC MEMORY DEVICES
  • H10B43/00—EEPROM devices comprising charge-trapping gate insulators

Definitions

• the disclosure herein relates to data storage technology.
• FIG. i illustrates an embodiment of an in-situ anncalablc system that includes a host processor or controller and an annealable memory system;
• Figure 2A illustrates the background anneal sequence of Figure 1 in greater detail
• Figure 2B illustrates a specific instance of the generalized anneal sequence of Figure 2A
• Figure 3 illustrates an exemplar ⁇ ' background anneal cycle in which selection, passive evacuation, active evacuation and anneal phases are carried out in sequence for each memory device within a memory subsystem having five independently annealable memory devices;
• Figure 4 illustrates an exemplary progression of evacuation states within an anneal controller corresponding to escalating evacuation urgency
• Figure 5 illustrates an exemplary operation of the anneal controller of Figure 1 during a memory access interval and in accordance with the evacuation states of Figure 4;
• Figure 6A illustrates a generalized embodiment of an anneal-enabling memory controller and the various annealable memory topologies it may support;
• Figure 6B illustrates a generalized data processing system in which the annealable memory system of Figure 6A may be deployed
• Figure 7A illustrates an example of a run-time annealable memory system formed by a flash coupled via respective point to point data links to a set of flash memory devices;
• Figure 7B illustrates the pairing of an anneal target and an alternate storage to form a virtual device within the context of the flash controller and flash memory subsystem of
• Figure 8 illustrates exemplary changes in state within the memory subsystem of
• Figures 9A and 9B illustrate exemplary data transfer operations that may be executed within the memory system of Figure 7 A to actively evacuate data from the anneal target to an alternate storage;
• Figure iOA illustrates an anneal progression and slice assignment according to a round-robin-progression, fixed-slice-assignm ⁇ nt embodiment
• Figure 1 OB illustrates an anneal progression and slice assignment according to an oscillaling-progression, fixed-slice-assignment embodiment
• Figure 1 OC illustrates an anneal progression and slice assignment according to a round-robin-progression, variable-slice-assignment embodiment
• Figure 1 I A illustrates a more detailed embodiment of a flash controller capable of managing the run- time- varying operational sets and corresponding slice assignments described in reference to Figures 1 OA-I OC;
• Figure 11 B illustrates a consequence of the shared-command/address architecture of Figure 1 1 A;
• Figure 11C illustrates an alternative rank-based memory architecture in which a flash controller is coupled to each flash memory device of a memory rank by a dedicated command/address path;
• Figure 1 ID illustrates an embodiment of a page table that may be used to support the device-specific command/address generation and sliced command/address architecture shown in the flash controller of Figure 11C;
• Figure 12 is a flow diagram illustrating an exemplary sequence of operations that may be carried out by the flash controller of Figures 7A-7B (and Figure 11) in response to host-requested or self-generated memory access commands;
• Figure 13 illustrates an embodiment of an annealable multi-drop memory system
• Figure 14 illustrates an exemplary operation of the annealable multi-drop memory system of Figure 13 in which composite storage is used to store data evacuated from the device to be annealed;
• Figure 15 illustrates an embodiment of an annealable multi-channel memory system and an exemplary pairing of devices therein;
• Figure 16 illustrates an alternate approach to pairing devices within an annealable multi-channel memory system
• Figure 17 illustrates an exemplary operation of the annealable multi-channel memory system of Figure 15 in which composite storage is used to store data evacuated from the device to be annealed;
• FIG. 18 illustrates an embodiment of a multi-channel memory controller having switched memory I/O circuitry to facilitate channel-to-channcl data transfer and thus speed data relocation operations;
• Figure 19 illustrates an alternative channel-to-channcl data relocation operation within the multi-channel memory controller of Figure 18 in which data read from an anneal target on one memory channel is stored temporarily within the controller core and then later written to an alternate storage on another memory channel:
• Figures 20A-20E illustrate anneal operations within a memory architecture that permits one spare device to be annealed simultaneously with data evacuation into an other spare device;
• Figures 21 A-2 IE illustrate anneal operations within a memory architecture that permits two spare devices to be annealed concurrently with data evacuation into a third spare device, or two spare devices to be evacuated concurrently with an anneal operation within a third spare device:
• Figure 22 illustrates a generalized sequence of operations that may be carried out in a foreground anneal 130;
• Figures 23A-23C illustrate a burst anneal sequence within a annealable memory- subsystem or host appliance
• Figure 24 illustrates an illustrates an exemplary compressed anneal operation that may be carried out to provide a spare memory device within a memory subsystem otherwise having no spare device, or to provide additional spare memory devices for purposes of leveraging the multiple-spare anneal methodology;
• Figure 25A exemplary user interface that may be presented to a human operator of a run-time annealable system in connection with foreground and/or background anneal operations;
• Figure 25B illustrates a generalized flow diagram of a software utility program that may be executed to render the user-interface of Figure 25 A and to prompt an operator to select or specify options in connection with data evacuation and/or maintenance operations;
• Figure 26 illustrates an embodiment of a flash memory system having a combination of ill-system and in-package anneal circuitry
• Figure 27 illustrates an embodiment of a multi-die flash memory package having in-package heating structures
• Flash memory meaning any non-volatile memory that operates on a charge-trapping principle, including floating-gate cells and as well as more modern structures such as SONOS (silicon-oxide-nitrogen-oxide-silicon) and the like
• flash memory meaning any non-volatile memory that operates on a charge-trapping principle, including floating-gate cells and as well as more modern structures such as SONOS (silicon-oxide-nitrogen-oxide-silicon) and the like
• SONOS silicon-oxide-nitrogen-oxide-silicon
• the data migration management techniques described herein are not limited to annealing applications and may be more generally applied to evacuate data from one or more memory devices within a memory system, thereby enabling '"hot" removal of a memory device, quarantining of a defective (or suspect) memory device, execution of an operation (maintenance or otherwise) that may result in data loss within a memory device, execution of an operation that may render a memory device temporarily unavailable for read and/or write access, and so forth.
• Figure 1 illustrates an embodiment of a run-time annealable system 100 that includes a host processor or controller 101 and a run-time annealable memory system 103.
• the run-time annealable memory system 103 itself includes one or more memory controllers 105 and an anneal-capable memory subsystem 107.
• the anneal capability may be implemented by anneal circuitry integrated within an integral ⁇ d-circuit memory die (on-di ⁇ anneal circuitry), inside an integrated circuit package along with one or more integrated-circuit memory dice (in-package anneal circuitry), and/or formed separately from but disposed in proximity to an integrated- circuit memory die or package (in-syst ⁇ m anneal circuitry).
• the anneal circuitry includes three basic components any or all of which may be on-die, in-package or in-syst ⁇ m: a heating element (or elements) that may be powered to heat an integrated circuit die to a desired annealing temperature or sequence of temperatures; power-control circuitry to deliver power to the heating elements in a manner that produces a desired annealing temperature and for selectively powering the heating elements on or off; and interface circuitry to receive commands that may be applied to the power-control circuitry to execute an anneal operation (power the heating elements on and to deliver power as necessary to reach a desired annealing temperature or sequence of annealing temperatures) and, if desirable, to terminate or interrupt an anneal operation.
• a heating element or elements
• power-control circuitry to deliver power to the heating elements in a manner that produces a desired annealing temperature and for selectively powering the heating elements on or off
• interface circuitry to receive commands that may be applied to the power-control circuitry to execute an anneal operation (power
• the duration of an anneal operation may be fixed in length (e.g., annealing temperature applied for a predetermined time in response to a command to execute an anneal operation), controlled by explicit start-anneal and stop-anneal commands, or programmed into a control register within the annealing circuitry.
• register programming any aspect of an annealing operation susceptible of external control or selection (e.g., anneal temperature, duration, frequency, specification of triggering event, etc.) may be controlled by explicit commands or by programming settings within a register that forms part of the interface or power-control circuitry.
• dedicated on-die anneal circuitry may be provided and/or additional existing circuit components may be reapplied to support annealing operation.
• wordlines or bitlines may double as heating elements as described in PCT Publication WO2008/067494 (Publication of PCT Application No.
• one or more separately-controlled heating elements may be disposed adjacent each integrated-circuit memory die (of which there may be one or many) to enable sequ ⁇ nced (i.e., die-by-die) annealing within the package.
• an entire set or stack of in-package memory dice may be annealed simultaneously by a single heating element or collectively-powered group of heatin Hg> elements.
• power-control and/or interface circuitry may be integrated with the heating elements or provided separately (e.g., disposed outside the package or provided by another integrated circuit and/or discrete components within the package).
• Annealing circuitry implemented partly or entirely separately from the memory device may be, for example, disposed on a printed circuit board (e.g., a mother board, daughter board, line card, memory module, etc.) beneath the one or more memory devices that form the memory subsystem or in a structure disposed over or enveloping individual memory devices or groups of memory devices.
• the power-control and/or interface circuitry may be integrated with the annealing circuitry or provided separately.
• the power-control circuitry may be provided by the memory controller itself.
• the anneal capability of the memory subsystem may be implemented by any practicable run-time controllable circuitry for heating an integrated -circuit memory die to a desired annealing temperature.
• memory controller 105 issues memory access commands, addresses and control signals to the memory subsystem 107 via command path 110, and corresponding read or write data is transferred between the memory controller and memory subsystem 107 via data path 108.
• Control signals such as device identifier (ID) signals or chip-select signals may be output by the memory controller to select all or a subset of the memory devices within the memory subsystem 107, and the controller may additionally output various maintenance commands (e.g., refresh, erase, etc.) and configuration commands to the memory subsystem to manage the devices therein.
• axmeal-eapable memory sub-system 107 may be implemented by any memory technology that exhibits anneal-reversible wear, embodiments described herein are generally assumed to contain flash memory devices (NOR or NAND) in which data is written, read and erased through program, read and erase operations, respectively. In contrast to other types of memory, data is not updated in place in such devices, but rather is rewritten to an alternate location while the originally programmed (and now stale) data is marked as invalid and ready for erasure.
• flash memory devices NOR or NAND
• a page tabic 1 11 may be provided within the memory controller 105 (which may be a flash controller, host controller or other type of memory controller) to track the physical location of valid, empty and invalid "pages" of data and thus to hide the underlying complexity of data access operations from the host processor 101 or other access requester.
• the memory controller 105 which may be a flash controller, host controller or other type of memory controller
• RAM volatile or non-volatile random access memory
• DRAM dynamic RAM
• SRAM static RAM
• Z-RAM zero- capacitor RAM
• TTRAM twin-transistor RAM
• Flash memory including nonvolatile memory technologies that implement a charge storage layer using polysilicon or other materials as in SONOS (silicon-oxide-nitride-oxide-silicon), SHINOS (silicon Hi-k nitride oxide silicon), etc. ), ferroelectric RAM (FeRAM or FRAM), magnetoresistive RAM
• MRAM programmable metallization cells
• PCM phase-change memory
• RRAM resistive RAM
• NRAM Racetrack memory
• annealing operations described herein may be applied to any type of memory technology that exhibits degraded performance over time (e.g., in response to program/erase operations or other operational events performed therein) and in which performance may be improved through execution of an annealing operation.
• implementations of the various embodiments presented herein are occasionally described in terms of instructional logic (i.e., processor or other execution engine executing instructions provided in a machine-readable media (i.e., software, firmware, etc.)) or dedicated hardware circuitry (e.g., logic circuits). In all cases, such embodiments may be implemented exclusively by instructional logic, exclusively by dedicated hardware, or by any combination of instructional logic and dedicated hardware.
• instructional logic i.e., processor or other execution engine executing instructions provided in a machine-readable media (i.e., software, firmware, etc.)
• dedicated hardware circuitry e.g., logic circuits
• memory controller 105 includes an anneal controller 109 that interacts with the page table 111 and selectively engages the anneal function of the memory subsystem 107.
• the anneal controller 109 issues anneal commands or control signals 112 (e.g., start anneal, terminate anneal and any other commands that may be used to control the anneal operation (e.g., temperature specification, register programming, etc. ⁇ via the same command path 110 used to convey memory commands (e.g., program commands, erase commands and read commands in a flash memory system).
• anneal commands or control signals 112 e.g., start anneal, terminate anneal and any other commands that may be used to control the anneal operation (e.g., temperature specification, register programming, etc. ⁇ via the same command path 110 used to convey memory commands (e.g., program commands, erase commands and read commands in a flash memory system).
• a separate anneal-command path formed by one or more additional signal lines may be provided to convey anneal-related commands.
• the host processor or controller 101 is responsible for the core (or primary) function of the run-time annealable system 100 and may be implemented by one or more special-purpose or general-purpose processors and/or application-specific integrated circuits (ASICs).
• the run-time annealable system 100 may form the processing foundation of a general-purpose computer system (e.g., a desktop workstation, or mobile computer), or a special purpose electronics device, such as a cell phone, global-positioning- system device (or similar handheld device), music or video player, set-top box, video display, network appliance or any other device that involves data processing.
• the run-time ann ⁇ alabl ⁇ system 100 may itself be a component to be employed within a larger system (e.g., a solid-state disk (SSD), a graphics engine (as in a graphics card), a blade (e.g., line card, switching fabric or processor) within a back-plane-interconnected network switching or routing device.
• a larger system e.g., a solid-state disk (SSD), a graphics engine (as in a graphics card), a blade (e.g., line card, switching fabric or processor) within a back-plane-interconnected network switching or routing device.
• the host processor or controller 101 may be considered part of the larger system and thus distinct from the run-tirn ⁇ annealable system itself.
• the larger system may include any number of additional (and unshown) components for interacting with an operator (e.g., display, audio-related components, keyboard, keypad, pointer- control, etc.), enabling access to mass-storage and/or removable- storage, enabling network access, and so forth.
• an operator e.g., display, audio-related components, keyboard, keypad, pointer- control, etc.
• the sequence of anneal operations directed by the anneal controller 109 may be broadly classified as either background anneal or foreground anneal, depending on whether core system operation continues or is suspended while the anneal operations are carried out.
• a generalized background anneal sequence within the memory system of Figure 1, individual memory ICs (i.e., integrated-eircuit memory devices) or groups of memory ICs are selected (121), evacuated (123) and annealed (125), all without substantial interruption of the core system operation (i.e., the user-demanded operation of the system continues to execute or be available for execution throughout the background anneal sequence).
• a triggering event i.e., triggering an anneal operation
• memory IC(s) are evacuated (133), annealed (135) and reloaded
• background anneal operations are carried out sequentially, with one memory TC (or one group of memory TCs) after another being selected, evacuated and annealed, while
• I-I I- foreground anneal operations may be carried out in parallel, evacuating and annealing the entire memory subsystem as a unit.
• core system operation may continue without the annealable memory subsystem (i.e., all data from memory subsystem is available from an alternative or redundant source such as a cache memory or backup memory or otherwise not needed)
• the entire memory subsystem may be evacuated and annealed at once (in the background), an operation referred to herein as a parallel anneal.
• memory ICs or groups of memory ICs within the memory subsystem may be annealed in sequence in a foreground anneal, particularly where the available backup storage for the annealable memory subsystem is insufficient to back-up the entire contents of the memory subsystem.
• memory evacuation, transferring data from the anneal target (the memory device to be annealed) to an alternate storage may be omitted in either the background or foreground anneal if the content of the anneal target is naturally backed up as part of system operation (e.g., content of anneal target copied to cache memory or recoverable/producible from other storage devices as in a RAID (redundant array of inexpensive disks) system).
• Figure 2A illustrates the background anneal sequence 120 of Figure 1 in greater detail.
• individual memory devices or groups of memory devices are sequentially annealed starting with selection of the anneal target (the memory IC to be annealed) in block 121, optional evacuation of the selected memory 1C in block 123, and finally the anneal operation itself in block 125.
• Anneal target selection may proceed in a fixed order, for example, stepping through each of the memory devices in the memory subsystem in a predetermined sequence, or may be programmed or even determined dynamically.
• a programmable selection sequence may be helpful, for example, where the memory subsystem is logically partitioned into storage regions having different classes of usage (e.g., infrequently-written long-term data storage vs. frequently- written short-term storage). Tn that case, memory devices allocated to usage classes that involve relatively infrequent writes (e.g., long-term non-voiatile data storage) may be annealed less frequently or omitted from the anneal sequence altogether, thereby enabling a higher anneal frequency for the remaining devices in higher usage classes. Similarly, devices allocated to use classes that involve relatively infrequent writes (e.g., long-term non-voiatile data storage) may be annealed less frequently or omitted from the anneal sequence altogether, thereby enabling a higher anneal frequency for the remaining devices in higher usage classes. Similarly, devices allocated to
• intermediate-use classes may be selected for less frequent annealing than those in higher-use classes.
• a programmable register is provided within the anneal controller (i.e., element 109 of Figure 1) and/or individual memory devices to permit definition of logical partitions and specification of a relative or absolute anneal frequency for each storage region.
• a relative anneal frequency may be specified, for example, by indicating an anneal priority (e.g., maximum, intermediate, low, disabled (no anneal)), or the total number of anneal cycles that are to be completed within the memory subsystem between each anneal directed to the subject storage region (a lower number indicating a more frequent anneal choice).
• An absolute anneal may be specified, for example, by programming a time interval and/or number of program/erase operations that are to elapse between anneal operations directed to the storage region.
• programmable selection or anneal frequency may be helpful is where the memory subsystem is populated by memory devices having different wear profiles (i.e., some devices wearing out faster than others for the same usage level).
• the anneal frequency may be specified directly (e.g., anneal once ever ⁇ ' N program/erase cycles) or indirectly (device type which may be looked up in a basic-input- output-service (BIOS) or other control program to determine appropriate anneal frequency) by information associated with each memory device and used to program the anneal sequence or frequency within the memory controller.
• BIOS basic-input- output-service
• information indicating the anneal frequency as a function of elapsed time, program/erase cycles and/or error threshold detection may be recorded within each memory device or recorded within an associated storage (e.g., as in a scrial-presence-detcct memory in a memory module) and thus may be read by the memory controller during system startup and used by the memory controller or host processor to program an anneal schedule (or anneal sequence) for each memory device.
• a dynamically determined anneal sequence may be helpful in applications where memory usage patterns vary substantially from device to device, but are difficult to predict or control
• memory devices may be selected dynamically, for example, selecting each device to be annealed in an order and/or time dictated by level of use (e.g., number of program/erase operations directed to the device).
• the usage metric may be an absolute count of the number of wear- inducing operations or may account for wear characteristics of the memory device.
• one thousand program/erase cycles in a memory device for which anneal-per two-thousand program/erase cycles is specified (or determined) may be prioritized over fifty thousand program/erase cycles a memory device bearing an anneal -per- hundred thousand
• the event used to trigger or initiate the anneal operation within each selected IC may similarly be predetermined, programmed or dynamically determined.
• Predetermined triggering includes, for example, initiating the anneal sequence for each selected device at regularly scheduled time intervals, initiating the anneal sequence continuously from device to device (e.g., proceeding without delay from the conclusion of an anneal operation for one memory device to the start of an anneal operation in the next memory device) or initiating the anneal sequence within a given device in response to detecting that one or more programmed thresholds (elapsed time, usage (e.g., program erase cycles), error rate or quantity (e.g., bit error rate, programming error rate, bad block quantity, etc.)) have been exceeded.
• programmed thresholds elapsed time, usage (e.g., program erase cycles), error rate or quantity (e.g., bit error rate, programming error rate, bad block quantity, etc.)
• Periodic or occasional (or uscr/softwarc- triggered) calibration/measurements may also be carried out to ascertain the need for maintenance (e.g., measuring data retention, number of program cycles or time required to program, or any other physical property (ies) of a device that may be used as empirical evidence of the need for anneal or other maintenance).
• the measured values or test results may be, for example, compared against absolute, dynamic or programmed benchmarks to determine whether an anneal or other maintenance operation is needed.
• measurement operations may be carried out by circuitry within the memory device under test (e.g., of the device's own volition or in response to measurement/calibration/test commands from a control device), by circuitry outside the memory device, or by a combination of circuitry and/or instructional logic within and outside the memory device.
• a controller e.g., implemented by dedicated circuitry, a software-controlled processor or any combination of dedicated circuitry and instruction-controlled logic
• a controller can periodically or at specific user instigation test a memory device to benchmark performance or changes in performance; for example, the number of program-verify ("PV") cycles for a memory device (e.g., average, maximum, minimum, or other measures) required for a successful write (program) operation or erase operation may be compared against one or more thresholds as indicating remaining life or excessive wear.
• PV program-verify
• any statistical derivation related to calibration/measurement data may also be used - as non-limiting examples, changes to average PV cycle requirements for a successful write or erase (e.g., for a page or block, or for randomly sampled cells in a memory device) may be used, either alone or with other measures (e.g., changes in standard deviation of PV cycles required for successful write or erase operations). Time required for successful write or erase operations may be measured using units other than PV cycles (e.g., using elapsed time). Many other benchmarks may also be used.
• Programmable trigger selection may include register-based selection (e.g., by a host processor executing BfOS or other program code) of the types of events (elapsed time, program/erase count, error detection, calibration/measurement etc.) and/or corresponding threshold levels that are to trigger anneal operations.
• Anneal operations may be triggered dynamically, for example, in response to detection of a threshold number or frequency of errors in retrieved data, failure to successfully execute program and/or erase operations, bad storage blocks, failure of measured/tested values to meet benchmarks, etc.
• an alternate storage device is selected or identified at 124a to establish a destination (repository) for evacuated data.
• the alternate storage may be one or more other memory devices within the annealable memory subsystem or may be a separate storage such as a scratch memory (i.e., a memory buffer provided for temporary data storage) within the memory controller, or an entirely separate storage within or external to the host system.
• the alternate storage may be predetermined or selected dynamically, As an example, in a number of embodiments described below, the alternate storage is the most recently annealed memory device (and therefore blank (empty) or presumed to be blank) within the anneal - capable (annealable) memory subsystem.
• the anneal target upon triggering an anneal operation, is evacuated via data transfer to the most recently annealed memory device (the prior anneal target) and then, after being annealed, itself becomes the alternate storage for the next anneal target,
• the anneal target In a dynamic selection of the alternate storage, other memory devices within the anncalablc memory subsystem or storage devices elsewhere in the host system may be evaluated with respect to the capacity required to support evacuation of the annealed device, and one or more of the memory devices or other storage devices selected to form the alternate storage. Examples of these embodiments are discussed in greater detail below.
• Data evacuation operations themselves may be categorized as passive or active depending on whether data is evacuated as part of natural traffic patterns (passive evacuation) and thus without requiring memory access operations merely to relocate the data, or through dedicated retrieval and relocation operations (active evacuation).
• passive evacuation a number of data evacuation operations themselves may be categorized as passive or active depending on whether data is evacuated as part of natural traffic patterns (passive evacuation) and thus without requiring memory access operations merely to relocate the data, or through dedicated retrieval and relocation operations (active evacuation).
• evacuation of data from the anneal target becomes progressively more aggressive as the need for anneal becomes more urgent, proceeding from a period of no data evacuation, to passive evacuation to active evacuation.
• data evacuation operations may initially be carried out in deference to host requested accesses (e.g., opportunistically performing evacuation operations when no host request is pending), followed by escalation to more aggressive competition with host requests for access to the memory subsystem.
• host requested accesses e.g., opportunistically performing evacuation operations when no host request is pending
• escalation to more aggressive competition with host requests for access to the memory subsystem e.g., opportunistically performing evacuation operations when no host request is pending
• operations may progress from less recently accessed pages to more recently accessed pages, thus actively evacuating the pages deemed less statistically likely to be evacuated as part of natural traffic patterns first.
• Passive evacuation operations are discussed in greater detail below.
• passive evacuation operations are memory access operation (generally requested by a host device) that enable data to migrate to alternate storage through logical- to-physical address re-mapping.
• Passive evacuation is generally effected through manipulation of page table entries and involves blocking write operations within the anneal target and re-directing those operations to the alternate storage instead, and retiring pages within the anneal target when they become stale.
• the updated page i.e., page of write data that is to replace a valid existing page within the anneal target
• the page table updated to invalidate the stale data page within the anneal target and to reflect the newly written page within the alternate storage.
• New writes are re-directed from the anneal target to the alternate storage in generally the same manner (writing the new data to the alternate storage), except that no page invalidation occurs with respect to the anneal target.
• blocking new writes within the anneal target generally involves data re-direction or bypass, the blocking operation prevents loading of data within the anneal target and thus may be viewed as part of passive evacuation.
• active evacuation operations involve host-independent (i.e., self-directed by the memory controller) data retrieval and re-write operations; removing (relocating) data from the anneal target to alternate storage.
• Active evacuation operations may be further characterized as deferential or competitive according to whether such operations defer to or compete with host-requested accesses to the ann ⁇ alable memory subsystem. In general, if the need for anneal is not urgent (i.e., deadline for executing an anneal is not imminent), active evacuation may be carried out during idle (spare) memory- access cycles or, if resources permit, in parallel with host-requested accesses.
• the anneal controller may undertake competitive evacuation, arbitrating with host-initiated memory access requests for available memory access cycles.
• the anneal controller may insert axmeal-relat ⁇ d data read and write operations ahead of host-requested memory accesses to ensure that the anneal target is evacuated prior to data loss or other device failure.
• the anneal controller will defer to host-requested accesses at least to the extent needed to ensure that core system operation may proceed, although at potentially lower performance for brief intervals,
• the anneal controller may initiate the actual anneal operation (restorative heating) operation with the anneal target as shown at 125.
• the anneal operation may be triggered immediately upon confirming evacuation, or the anneal controller may await a scheduled time or occurrence of a predetermined event before initiating the anneal operation.
• Figure 2B illustrates a specific instance of the generalized anneal sequence of Figure 2A.
• one of the memory devices within a memory subsystem is taken '"off-line'" (i.e., removed from the set of memory devices accessed in response to host read/write request) and replaced by an idle (formerly off-line) device.
• page table and I/O logic within the memory controller are updated to re-map incoming writes to the new device set (i.e., including the formerly off-line device and excluding the currently off-line device). Pages within the off-line device (the anneal target) are retired as new data is written (142).
• the off-line device is determined to be empty at decision block 143, then an anneal (i.e., heating) operation is performed as shown at 145 to complete the sequence of anneal events. Thereafter, at 146, another flash device is selected to be the anneal target (the next "off-line' ' device) and the anneal event sequence is repeated starting at 140.
• anneal i.e., heating
• a time limit or other threshold has been reached as determined at 147 (e.g., elapsed time, calendar time, error rate, error count, number of program-erase cycles, etc., has reached a
• anneal is commenced at 145, followed by selection of a new anneal target at 146 and a repetition of the anneal sequence starting at 140.
• FIG. 3 illustrates an exemplary background anneal cycle in which selection, passive evacuation, active evacuation and anneal phases are carried out in sequence for each memory device within a memory subsystem having five independently annealable memory devices.
• selection, passive evacuation, active evacuation and anneal phases are carried out in a sequence for memory devices in a subsystem, it is also possible, using the principles presented herein, to trigger these operations in response to a specifically-detected need, e.g., perceived remaining lifetime for any particular device.
• the five memory devices, M1 -M5 form a "rank'" of memory devices (anneal operations within other memory topologies such as multi-drop and multi-channel topologies, are discussed below). More specifically, a dynamically selected subset of four of the memory devices, referred to herein as the operational memory set (or operational set or device set) are accessed in parallel as a unit, each contributing a one-fourth share of each returned read data value and each storing a one- fourth share of each write data value. The remaining device, referred to as the standby memory (or spare memory), does not participate in memory read and write operations until an anneal target is selected from the operational memory and data evacuation is begun.
• the standby memory or spare memory
• memory device Ml constitutes the standby memory (153) and memory devices M2-M5 constitute the operational memory set 151.
• each memory device is assumed to have multiple storage banks 155 shaded to show valid content and un-shaded to designate blank or empty status.
• Background anneal cycle T begins with selection phase 16I 1 .
• M2 is selected as the new anneal target and standby memory Ml is selected as the alternate storage.
• the alternate storage is paired with the anneal target to form a virtual device (or virtual device pair) for purposes of data storage and retrieval during the upcoming evacuation phases. More specifically, during the passive evacuation phase 163,, incoming read requests that map to valid data (marked by shaded storage arrays) within the anneal target continue to be read from the anneal target while write operations (new writes and updates) are re-directed to the alternate storage. As the alternate storage becomes populated (i.e..).
• the alternate storage and anneal target (M 1 and M2) constitute a single virtual device that is responsible for the one- fourth share of a data value (the other three-fourths being provided by M3-M5) that would otherwise be provided by a single device.
• tbe anneal controller (leveraging command and I/O circuitry within the memory controller) issues commands, separate from those requested by the host controller/processor, to retrieve remaining data from the anneal target and write tbat data to the alternate storage.
• commands are indicated by the hashed regions within the storage arrays of M2 and Ml.
• the data transfer may take place in a unified read/write operation (i.e., read-from-target/write-to-alternate), or in component operations, first copying the data from the anneal target to a scratch memory within the memory controller (or another temporary storage source) and then later writing to the alternate storage.
• the active relocation operations may progress from less recently used (colder) pages to more recently used (hotter) pages. This progression is discussed in greater detail below.
• the memory controller includes either Ml or M2 in the operational memory set. Further, in one embodiment, the memory controller may exclude all memory devices but the virtual device (Ml and M2) from the relocation transactions (M2 to Ml ) in the active evacuation phase.
• the memory controller dynamically determines which subset of memory devices are to be accessed in a givers transaction based on the nature of the operation performed, the location of the data to be accessed and the operational memory set in place.
• this ability to dynamically select different memory devices or subsets of memory devices within the same rank is achieved through provision of separate chip-select lines from the memory controller to each memory device of the rank, thus enabling the memory controller to include only the desired subset of memory devices (which may be a single device) in any split-read, redirected write or relocation transaction.
• the memory controller may additionally include circuitry to dynamically switch the data lanes coupled to the operational memory set so as to preserve the order of significance of the individual portions of the overall data word written to or read from the operational memory set.
• the anneal controller determines that no valid data remains in the anneal target (i.e., the anneal target is fully evacuated or empty)
• the alternate storage (Ml) and other still -populated memory devices (M3-M5) form the new operational memory set and an anneal operation may be carried out within the anneal target (M2) in anneal phase 167;.
• the background anneal cycle is complete and the now-empty and annealed memory device (M2) constitutes the new standby memory and may be employed as the alternate storage in the next background anneal cycle, starting with the selection phase 161;.', .
• next background anneal cycle in which M3 is selected as the anneal target and M2 the alternate storage
• M3 is selected as the anneal target and M2 the alternate storage
• pre-selected events e.g., threshold usage, threshold error rate or number of errors, etc.
• a general goal in the background anneal process is to minimize impact on core system operation and thus render the anneal sequence transparent to the system user.
• the data evacuation effort is maintained as a passive operation for as iong as possible, then escalated from passive evacuation to increasingly active evacuation as the deadline for performing an anneal draws closer.
• the anneal controller may differentiate between data pages based on access frequency and/or recency. That is, as mentioned above, the anneal controller may relocate less frequently or recently accessed, "cold' * pages first in view of the higher statistical likelihood that the more frequently used, "hot" pages may yet be staled-out in an update operation.
• FIG. 4 illustrates an exemplary progression of evacuation states within an anneal controller (e.g., element 109 of Figure 1 ⁇ corresponding to escalating evacuation urgency.
• the evacuation state (represented by state variable ⁇ vacMode") is set to "Passive" at 181 and remains in that state until an event counter (e.g., an elapsed time counter, error counter, program/erase counter, etc.) updated at block 183 is determined to exceed a passive evacuation threshold at decision block 185.
• an event counter e.g., an elapsed time counter, error counter, program/erase counter, etc.
• the anneal controller while in the passive evacuation state, the anneal controller enables pages to be evacuated as part of natural (e.g., host-requested) traffic, re-directing updates to alternate storage and invalidating stale entries in the anneal target.
• natural e.g., host-requested
• the anneal controller Upon determining that the event counter has exceeded the passive evacuation threshold at decision block 185, the anneal controller enters an active evacuation phase 186 in which data is actively relocated with deference to host-requested memory accesses. More specifically, at 187, the evacuation mode is set to an active-deferential state (Active.Defer) to enable active but deferential data relocation.
• Active-deferential state Active-deferential state
• the anneal controller relocates data from the anneal target to the alternate storage.
• the anneal controller transitions to an active evacuation state 192 in which relocation operations contend with (i.e., compete with) host-requested memory accesses. More specifically, at block 193, the evacuation mode is transitioncd from Active. Defer to Active. Compete to specify the active-competitive state.
• the anneal controller While in the active-competitive state, the anneal controller competes with host requests for access to the anneaJablc memory subsystem, in at least some cases superseding host requests in an effort to ensure complete data evacuation prior to catastrophic data loss (or some event considered to define an evacuation deadline after which catastrophic data loss cannot be reliably avoided).
• the anneal controller Upon determining that the anneal target is fully evacuated (decision block 195), the anneal controller reverts to a non- evacuation state (NoEvac) at 197 and remains there until data evacuation is initiated within the next memory device to be annealed.
• NoEvac non- evacuation state
• the anneal controller may differentiate between hot and cold pages (pages being marked as hot or cold (or with even finer usage grades) in the page table, for example) and evacuate cold pages first. In one embodiment, for example, the anneal controller transitions through two or more sub-states (according to the number of data usage gradations) within each of the activ ⁇ .d ⁇ fer and active. compete states.
• This operation is shown at 194 for an embodiment having two data usage gradations, ''cold" pages and ''hot” pages, with cold pages being those less recently (or less frequently) accessed than "hot" pages.
• the axmeal controller progresses from an active.def ⁇ r.cold state in which, cold pages are evacuated, to an actrve.defer.hot state in which hot pages are evacuated.
• the anneal controller may similarly transition from an active.def ⁇ r.cold state in which, cold pages are evacuated, to an actrve.defer.hot state in which hot pages are evacuated.
• the anneal controller may similarly transition from an active.def ⁇ r.cold state in which, cold pages are evacuated, to an actrve.defer.hot state in which hot pages are evacuated.
• the anneal controller may similarly transition from an active.def ⁇ r.cold state in which, cold pages are evacuated, to an actrve.defer.hot state in which hot pages are evacuated
• the eoJd/hot sub-states may only be applied within the active-deferential or active-competitive states
• Figure 5 illustrates an exemplary operation of the anneal controller of Figure 1 (and the encompassing memory controller) during a memory access interval and in accordance with the evacuation states of Figure 4. If no memory access request is pending at the start of the memory access interval (determined at decision block 201), the memory subsystem is deemed idle for that interval and. if the anneal controller is an active evacuation state (Active.Defer or Active. Competitv ⁇ , both of which yield an affirmative determination in decision block 203;, then a data transfer from the anneal target to the alternate storage is carried out at block 205, with corresponding update to the page table to invalidate the data page in the anneal target and record the new entry in the alternate storage.
• Active evacuation state Active.Defer or Active. Competitv ⁇
• this operation may be effectuated by a switched coupling of the respective data paths to the affected memory devices, thus permitting efficient passage of the data retrieved from the anneal target to the alternate storage.
• ''shunt transfer operations may be carried out more quickly than disjoint read and write transactions within the source and destination memory devices.
• individual read and write transactions may be executed, retrieving data to a local buffer or scratch memory within the memory controller and then immediately thereafter, or in a later cycle, writing the retrieved data to the alternate storage.
• the memory controller indexes the page table using a logical address provided with the memory access request to determine the physical address to be read or written (209) and then reads data from or writes data to the specified storage location (21 1 ).
• the memory controller will invalidate page table entries corresponding to the logical address and create new entries to reflect the physical address for the newly written data (recalling that effectuating an overwrite in flash memory generally involves programming the updated data in a new page and flagging the preexisting and now stale page for eventual erase), As no data evacuation is ongoing, the same memory devices constituting the operational memory for the pre-existing page may be specified in the page table for the updated page.
• the page table is indexed using the logical address provided in the access request to determine either the physical address of the requested data (215) or the physical address of storage location to be written (223), depending on whether a read or write access is requested (determined in decision block 213).
• a scratch memory a temporary storage within the memory controller as discussed below
• active evacuation is ongoing (determined in decision block 217)
• data may be retrieved from the scratch memory and returned to the host concurrently with a shunt read/ write from the anneal target to the alternate storage (and corresponding page table update to reflect the relocated page).
• the memory controller may occasionally suspend execution of a host- requested memory read in favor of a relocation operation.
• the memory controller may insert one or more data relocation accesses (reads from axmeal target, writes to alternate storage) ahead of a pending host-requested read.
• a memory write operation if the write destination is within the anneal target or scratch memory (decision block 225), then the page table is updated to re-map the write destination to the alternate storage at block 233. Thereafter, host-supplied write data is written to the location within the alternate storage indicated by the page table (block 235). If the write destination is not within the anneal target or scratch memory (negative determination at decision block 225 ⁇ and the evacuation mode is active-competitive, then the write data may be written to the scratch memory in parallel with a relocation operation within the anneal target and alternate storage (e.g.. shunt read/write from the anneal target to the alternate storage), including page table update as indicated at 229. If the evacuation mode is passive or active-deferential (negative determination at decision block 227), data is written to the memory location indicated by the page table as shown at block 231.
• the evacuation mode is passive or active-deferential
• Figure 6A illustrates an embodiment of an annealable memory subsystem 270, including a generalized armeal-enabling memory controller 271 and the various annealable memory topologies it may support.
• the memory controller includes a host interface 275 (i.e., host input/output (I/O)), memory interface 277 (memory T/O), control logic 273 (including anneal controller 274), page table 279, scratch memory 281, read/write data path 287 and host-side and memory-side control paths, 283 and 285.
• Memory access requests are received within the memory controller 271 via the host interface 275 (by way of path 292) and delivered to the control logic 273 via host-side control path 283.
• the control logic 273 converts the incoming host access requests into corresponding memory access command, address and control signals, scheduling commands for output to the memory subsystem via memory-side control path 285 and memory interface 277 (and ultimately via memory channel 290), in the case of a host-requested write operation, write data is received via the host interface 275 and forwarded via internal data path 287 (also referred to herein as internal data lanes of the controller) to the memory interface 277 for transfer in association with one or more corresponding memory -write (or memory program) commands.
• the write data may be buffered or queued within the scratch memory 281 or other data buffer within the memory controller 271 as necessary to establish a desired timing relationship between command and data transmission via the memory channel 290.
• read data is received from the memory subsystem via the memory interface 277 and may be buffered as necessary in the scratch memory 281 or other data buffer within the memory controller 271 before being returned to the requesting host via host interface 275.
• read and write requests received via the host interface include (or are associated with) virtual address values thai are translated, via control logic and page table operation, into physical addresses that correspond to specific storage regions within the memory subsystem or other storage resources (e.g., the scratch memory 281),
• This virtual-to-physical address translation enables the control logic to map a given virtual address to physical storage locations as necessary to accommodate particularities of the underlying memory subsystem.
• the memory controller thus virtualizes the underlying storage arrangement, hiding complexity relating to management of the memory subsystem from the host requestor.
• the host requestor may simply issue virtual addresses without regard to the underlying details necessary for management of the various different types of memory to be accessed. More generally, even in a memory subsystem populated by a uniform type of memory, memory access complexity and constraints may be hidden from the access requestor.
• a sequence of host-requested write operations directed to the same virtual address may be mapped to a sequence of different physical memory addresses, with each successive write operation being effected by allocating a new physical address to hold the new write data while marking the physical address corresponding to the previously written data as invalid and to be eventually reclaimed through an erase operation.
• the redirection of data from anneal target to alternate storage may be effected by mapping (or remapping) virtual addresses that would otherwise be directed to the anneal target to a physical address within the alternate storage instead.
• mapping or remapping
• memory read requests may be split between the anneal target and the alternate storage according to the page table entry.
• data relocation may be effected by copying data from the anneal target to the alternate storage (either in a shunt read/wrile or through discrete read and write operations) and by updating the page table to invalidate the entry within the anneal target and designate the alternate storage as the repository of the data,
• all such data evacuation operations, as well as selection of the anneal target and alternate storage and initiation/termination of the anneal event itself are managed by the control logic 273, and more particularly by the anneal controller 274 within the control logic.
• the selection of the anneal target and alternate storage, as well as the movement of data between the target and alternate may vary significantly according to the topology of the memory system.
• the memory system may be classified as a single-channel system 301 or multi-channel system 330 according to the number of independently controllable memory channels coupled between the memory controller and respective memory subsystems.
• a single memory subsystem 302 is coupled to the memory controller in a single-channel system
• multiple memory subsystems (331 A, 33 IB, ...) are coupled to the memory controller in a multi-channel system.
• the memory devices that constitute a given memory subsystem may be coupled to the memory controller via a multi-drop memory channel (320) or point-to-point memory channel (303), and the memory channel itself may be sliced or unified.
• all the memory devices within the memory subsystem are coupled in common to the signaling lines that form the memory channel, whereas in the slic ⁇ d-channel configuration, memory devices or sets of memory devices within the memory subsystem are coupled to respective subsets of the signaling lines that constitute the memory channel.
• the command and address (CA) lines of the memory channel may be coupled in common to multiple memory devices (309;, 309?., ..., 309,,, m ), while a set of n data lines is sliced into subsets of m data lines (306 :, 306?, ..., 306 n/ ,,i) with each subset being coupled respectively to one of the n/m memory devices 309.
• a set of n data lines may be sliced into subsets of m lines (325 ⁇ , 325 ⁇ , ..., 325 n/m ), with each subset coupled in multi-drop fashion to a respective set of memory devices.
• the entire set of Ti data lines is coupled to a single memory device 317 (i.e., single memory TC), and thus coupled to an n-bit wide data interface (I/O 31 8) of the memory device to enable access to the memory core 319 therein.
• the memory devices coupled to respective m-bit slices of the data path may be operated as a parallel unit by delivering the same memory read and write command to each device (e.g., via a shared command path, not specifically shown).
• a set of memory devices operated as such a unit appears to the memory controller to be a unified device, with each memory device receiving a respective m-bit share of an n-bit write data value during a write operation and outputting a respective m-bit share of an n-bit read data value during a read operation.
• multiple ranks of memory devices may be defined, for example, through an addressing arrangement that enables only the devices of a selected rank (324; , 324;, ..., 324;.,/TM) to respond to a memory read or write command.
• respective chip-select lines or groups of chip select lines are coupled to each rank of memory devices 323 to enable a selected rank to respond to a memory access command provided via a command/address path that is coupled to all the memory devices within the memory subsystem.
• Figure 6B illustrates a generalized data processing system 335 in which the annealable memory system 270 of Figure 6A may be deployed.
• the data processing system 335 includes one or more processors 336 (or processor-cores) each of which may have various classes of internal storage (e.g., register file (Reg), level- 1 cache (Ll ), level 2 cache (L2), non-volatile storage (NV), etc.); a main memory 337 formed by address translation circuitry 342 (e.g., implemented by or within a memory controller such as memory controller 271 of Figure 6A) and one or more memory modules 341 ; and one or more secondary storage devices 338, such as a solid state disk (SSD), hard-disk drive, removable media (e.g., optical media, magnetic media, etc.).
• SSD solid state disk
• SSD hard-disk drive
• removable media e.g., optical media, magnetic media, etc.
• additional components may additionally be provided within the data processing system.
• additional components may each share the main memory 337 and/or secondary storage devices 338 with processors 336 and each additional component may also or alternatively include dedicated memories that may themselves be implemented by anneal able memory systems,
• the various categories or classes of memory within data processing system 335 may be viewed as forming a storage hierarchy with respect to a given data requestor.
• the register file, Ll cache, L2 cache, internal NV memory, main memory and secondary storage represent a hierarchy of storage devices, with each tier within the hierarchy generally exhibiting a larger storage capacity but slower access speed than the tier above it.
• the register file is smaller and faster than the Ll cache which is smaller and faster than the L2 cache which is smaller and faster than the main memory, etc.
• the tiers of memory may be distinguished by a number of characteristics beyond capacity and access speed.
• each tier of memory is generally disposed in closer physical proximity to the processing core than the next-larger/slower tier.
• each tier of memory that is addressed through a virtual addressing scheme i.e., address is translated by an address translation circuit such as 342 to generate the physical address applied to the memory devices themselves
• the address translation circuitry 342 provided within main memory 337 may be used to generate all physical addresses applied to the memory modules 341 (or other memory components in the event that memory devices are mounted directly to a motherboard or otherwise disposed within the system without provision of removable memory modules) so that the constituent memory devices may be deemed a "flat ' ' tier of memory from the perspective of the processor 336.
• the secondary storage devices 338 may be viewed as a single tier of memory or as separate tiers of memory, with each such device potentially including its own address translation circuitry.
• the address translation circuitry 342 translates the incoming "virtual" addresses into physical addresses that are, in turn, output to the memory modules 341 via command/address (CA) path 343.
• the address translation circuitry 342 may be associated with more general memory controller circuitry that also translates and/or queues incoming requests for subsequent issuance over the C/A path 343 as well as various control signals, such as chip-select signals, timing signals, etc.
• Data may be transferred between the processors 336 and main memory 337 via a data path 340 as shown, or time- multiplexed onto the request/address path, thus rendering the latter path as a generalized information signaling path.
• the data path 340 alone is depicted as extending to the secondary storage devices 338, a dedicated or shared command/address path may be coupled to each such device, or the data path 340 may be used to convey more generalized command/data information.
• an annealable memory system 270 may be used to implement any or all of the tiers of memory shown in Figure 6B, as well as any additional tiers or classes of memory not shown (e.g., graphics memory or additional tiers of memory inserted hierarchically between the main memory tier and secondary storage tier or between any others of the memory tiers).
• all or any subset of the individual integrat ⁇ d-eircuit memory devices (Ml-Mn) mounted to memory modules 341 may be implemented by flash memory devices or other types of memory devices that are annealed from time to time using the background and/or foreground annealing methods and supporting circuitry described herein.
• the individually anneal able memory devices may form slices of a rank as discussed in reference to Figure 6 A, with each rank of memory devices being coupled in multi-drop fashion to the sliced data path, or coupled point-to-point to respective sliced data paths.
• data path 340 is depicted as extending directly between the memory modules and the processor, data may pass through and/or be reorganized within a memory controller or other intermedial" ⁇ ' device.
• flash memory devices including NAND-typ ⁇ flash devices, NOR-type flash devices, as well as any other annealable memory devices may be employed in virtually any topology as discussed in reference to Figure 6A.
• the individual memory device(s) that constitute the annealable memory subsystem may be evacuated to one or more other devices within the same memory tier, or to one or more other tiers of memory devices.
• the selection or specification of the alternate storage to which data is evacuated may be fixed by design, determined dynamically according to system conditions, or programmatically controlled (e.g., specified by an initial configuration or by an operator when prompted during execution of a utility program or other software program relating to anneal).
• data evacuation is not limited to evacuation in preparation for anneal and instead may be carried out generally for maintenance operations or to render a device available for "hot" removal (i.e., removal under power) or for any other reason.
• Modifications of the generalized memory system 270 may be appropriate according to the application of the memory system 270 within a given memory tier.
• address translation circuitry may be omitted from any memory tier that is directly, physically addressed by a processor core (e.g., in an embodiment of the register file).
• the various tiers of memory may be implemented as discrete sets of one or more integrated-circuit dice (i.e., "chips") and systems of chips interco ⁇ pl ⁇ d by data signaling paths and control signaling paths, one or more tiers of memory may be integrated into a systern-on- chip (SOC) or within an integrated circuit package formed by one or more chips.
• SOC systern-on- chip
• FIG. 7A illustrates an example of a run-time annealabJe memory system formed by a flash controller 350 (a specific example of the more generic memory controller 271 shown in Figure 6A) coupled via respective point to point data iinks to a set of five flash memory devices (FDO, FD l, FD2, FD3 and FD4).
• the flash controller generally includes the memory controller components described above and thus contains, for example, control logic 351 (including an anneal controller 352). page table 353, scratch memory 355, command path 359 and memory interface 361.
• the constituent flash memory devices are coupled to a common command/address path (not shown in Figure 7A) and organized into a memory rank (i.e., a set of memory devices that receive and output respective slices of data, but that appear to the flash memory controller as a single unit for the purposes of host- requested read and write access), where the memory rank includes a single standby device and an operational memory set of four flash memory devices.
• a memory rank i.e., a set of memory devices that receive and output respective slices of data, but that appear to the flash memory controller as a single unit for the purposes of host- requested read and write access
• the memory rank includes a single standby device and an operational memory set of four flash memory devices.
• the arrangement is referred to herein as an NH memory rank (N devices per operational memory set, plus one spare) or NH architecture.
• the page table 353 is populated with address translation entries 354 that indicate not only logical to physical address translations, but also the operational memory set (i.e., subset of four memory devices to be accessed; also referred to herein as the "operational set") in which the data specified by a given logic address resides or is to be written. Accordingly, a logical address received in association with a host- r ⁇ quesl ⁇ d memory read or write may be applied by the control logic 351 to index the page table 353 and thus retrieve both the operational memory set and the physical address to be accessed therein.
• the op-set value (i.e., code indicating the operational set) is returned to the control logic 351 and used therein to assert the chip-select signals necessary to enable access to the specified subset (N) of the N+l attached memory devices.
• the physical address retrieved from the page table 353 is output onto the command/address path (not specifically shown in Figure 7A, but coupled to all N devices) which, due to the upset-controlled assertion of chip-select lines, is received within the operational set specified by the page table 353, thereby enabling the specified memory read or memory write operation to be carried out at the appropriate location within the appropriate subset of the attached flash memory devices.
• Figure 7B illustrates, within the context of the same flash controller 350 and flash memory subsystem as in Figure 7A, the pairing of an anneal target (flash device FDl) and an alternate storage (flash device FDO) to form a virtual device for purposes of data evacuation.
• new writes i.e., write operations directed a logical address not yet in the page table
• updates i.e., write operations directed to logical address already recorded in the page table and thus a request to overwrite data existing at that logical address
• FDO alternate storage
• a new entry corresponding to the host-supplied logical address is recorded in the page table and populated with the physical address at which data is stored and listing flash devices FDO, FD2, FD3 and FD4 as the operational set.
• the existing page table entry i.e., corresponding to the host-supplied logical address
• the existing page table entry is revised to reflect the physical address at which the updated data is written and to list flash devices FDO, FD2, FD3 and FD4 as the operational set.
• This revision of the existing page table entry effectively invalidates the pre-existing data entry within Ih ⁇ anneal target (i.e., effects a ''stale-out 5' of the data at the table-specified physical address within the anneal target).
• data may be read from an operational set that includes flash memory devices FDO. FD2, FD3 and FD4 (i.e., if the data sought has already been relocated to the alternate storage) or from an operational set that includes memory devices FDl , FD2, FD3 and FD4 (i.e., if the data has not yet been relocated within the alternate storage).
• Figure 8 illustrates exemplary changes in state within the memory subsystem of Figures 7 A and 7B as new-write operations and update operations are performed. For purposes of example, the following conditions are assumed at state 370:
• ® flash device FD3 is the current anneal target
• ® flash device FD2 is the alternate storage and thus flash devices FD2 and FD3 form a virtual pair for purposes of data evacuation;
• flash devices FDO, FDl , FD3 and FD4 constituted the operational memory set during the anneal of flash device FD2 and are striped with (i.e., contain at same physical addresses) data words A, B, D and C, such thai data slices AO, Al, A2 and A3 of data word A are stored within devices FDO, FDl, FD3 and FD4, respectively; data slices BO, Bl, B2 and B3 of data word B are stored within devices FDO, FDl , FD3 and FD4 respectively, and so forth; and ® an erase operation is ongoing within a physical address that is striped across all the memory devices, including device FD2.
• a programming operation i.e., writing in the flash memory subset
• a new data word 'F.' i.e., a new write
• flash device FD3 is the anneal target
• devices FDO, FDl, FD2 and FD4 are selected as the operational set and the programming operation is carried out to store data slices EO, Dl , E2 and E3 in respective unused storage regions of those flash devices.
• the flash controller asserts the chip select signals for the devices that constitute the operational set (FDO, FDl , FD2, FD4) so that the programming command is received only within those flash devices.
• the flash controller switches the four data slices that form data word 'E' (i.e., EO, El, E2, E3) onto the data paths coupled to the devices that form the operational set, thereby ensuring that the data slices arc delivered to the appropriate devices.
• the memory subsystem transitions from state 371 to state 372 in response to an update operation carried out to update existing data word 'B.'
• slices B0-B3 of the updated data word arc stored within an unused location of the operational set that includes the alternate storage (FD2), while the pre-existing data word is marked as invalid.
• data word 'B' is both updated (note that data word 'B' is stored in a new location within devices FDO, FDl. and FD4 as the data is written to a clean physical location instead of being overwritten in place) and slice D2 effectively transferred from the anneal target to the alternate storage by invalidation of the stale data word.
• the memory subsystem transitions from state 372 to state 373 in response to an update to data word 'D/
• slices D0-D3 of the updated data word are stored within an unused location of the operational set that includes the alternate storage (FD2), while the pre-existing and now stale data word is marked as invalid.
• the update to data word 'D' serves both to write the updated data within the memory subsystem and effectively to transfer slice D2 from the anneal target to the alternate storage.
• Figures 9A and 9B illustrate exemplar ⁇ ' data transfer operations that may be executed within the memory system of Figure 7 A to actively evacuate data from the anneal target to an alternate storage.
• Figure 9A illustrates a two-phase transfer in which data is initially transferred from the anneal target (FDl) to the scratch memory 355 within the flash controller 350 (phase 1), and then later transferred from the scratch memory 355 to the alternate storage, FDO (phase 2).
• the first-phase transfer may be earned out as part of a controller-initiated transfer (and thus be an active evacuation) or may be effected during a host-requested memory read operation, and thus be effected passively as part of normal operational traffic.
• the flash controller 350 may assert the chip-select line for only the anneal target for purposes of the read operation (i.e.. read command received only within the anneal target), thus avoiding unnecessary memory read operations within the other flash memory devices and thereby saving power.
• the flash controller may access all the memory devices within the operational set (i.e., flash devices FDl -FD4) and either ignore the data retrieved from devices other than the anneal target (i.e., not receive, save and/or forward the data from flash devices FD2-FD4) or save and/or forward the entire retrieved data word (i.e., effecting an internal or upstream caching operation).
• the data slice retrieved from the anneal target may be stored in the scratch memory 355 and the page table 353 updated to indicate that data from the specified address within the anneal target has been copied to the scratch memory 355 and is available for transfer to the alternate storage (e.g., setting the scratch memory flag V within the page table entry as shown in Figure 7D), ⁇
• the data slice to be relocated is retrieved from the anneal target as part of a host-requested read from the operational memory set.
• the entire data word read from the operational memory set including the slice from the anneal target, is forwarded to the host requestor via the memory T/0 361 and data read/write path 357, and the slice from the anneal target is additionally written to the scratch memory 355.
• the page table 353 is updated to indicate presence of the data from the anneal target within the scratch memory.
• the data is transferred from scratch memory 355 to the alternate storage (FDO in this example) as part of an active evacuation.
• a single-slice write operation is carried out to write data only to the alternate storage.
• the entire data word retrieved from the operational set that includes the anneal target i.e., the '"anneal set
• the ''alternate set is rewritten within the operational set that includes the alternate storage.
• Such '"multi-slice" operations are similar to the update operations described in reference to Figure 8, but are initiated by the flash controller rather than the host device.
• Figure 9B illustrates an exemplary single-phase data transfer between the anneal target and the alternate storage executed as part of an active evacuation.
• This operation is also referred to herein as a shunt read/write, as the flash controller 350 issues single-slice read instruction to the anneal target, FDl (i.e., chip-select raised for the anneal target only and not other devices of the N+ 1 rank), followed by a single-slice write to the alternate storage, FDO.
• FDl i.e., chip-select raised for the anneal target only and not other devices of the N+ 1 rank
• the data slice received from the anneal target is (i.e., the data being relocated) is looped (shunted) within the memory I/O circuit 361 from the receiver for the ann ⁇ al-target data path (i.e., the data lines coupled to the anneal target) to the output driver for the alternate-storage data path, thereby effecting a low-overhead data relocation from anneal target to alternate storage. That is, data is transferred between the anneal target and alternate storage without the delay or power consumption incurred in the transfer from memory 170 interface 361 to the internal read/write data path 357 and scratch memory 355 (i.e., without transfer to the memory controller core) as in the two-phase transfer of Figure 9 A.
• the m-bit data slices that form an ⁇ -bit data word within a sliced memory architecture 305 generally have a meaningful order with respect to one another.
• assignment of data slices to memory devices referred to herein as "slice assignment, * ' is static and established by the dedicated hardware connection between each internal data lane (i.e., slice conveying circuitry within the read/write path) of the memory controller and its corresponding external data path and memory device.
• correspondence between data slices and memory devices is dynamic, varying according to the operational memory set selected for a given memory access.
• This dynamic slice assignment presents a number of implementation challenges and options in terms of reallocating the slice assignment as each new anneal target is selected, and more generally in terms of the anneal-target progression itself (i.e., sequence of memory devices selected as anneal target).
• a round-robin anneal target progression is implemented (i.e., stepping from memory device 1 to 2 to 3 to ... N, in order, and then repeating the sequence starting again with memory device 0 ⁇ together with fixed slice assignment for each different anneal target.
• this approach provides the advantage of relatively low-overhead anneal-progression and slice-assignment logic, but at the cost of a substantial data relocation effort at the conclusion of each progression through the N+- memory rank (note the designation ''N+" is used herein to refer to a memory rank having one or more spare devices beyond the N devices needed to form an operational memory set matching the width of read and write data words).
• an oscillating progression reduces the data relocation effort relative to the round-robin/f ⁇ xed- slice-assignment approach, but yields an uneven number of anneal operations per progression cycle.
• round-robin anneal target progression is employed, but with multiple possible slice assign merits per anneal target (and thus increased complexity to determine and track the appropriate slice assignment) to reduce the data relocation volume.
• Figure 1OA illustrates an anneal progression and slice assignment according to the round-robin/fixed-slice-assignmcnt embodiment mentioned above.
• the example shown assumes a N+i architecture having four devices (and thus four slices) per operational memory set and a single spare device) formed by flash memory devices FDO, FDl, FD2, FD3 and FD5.
• anneal target FDl in selection state 'a'
• anneal target FD2 in selection state 'b
• anneal target FD3 in selection state 'c
• anneal target FD4 in selection state 'd
• anneal target FDO in selection state 'e.
• the operational memory set for each target selection state is shown below the memory rank as '"op set" (i.e., the flash memory devices of the rank less the anneal target) and the virtual device pair for each selection state is encompassed in dashed outline (with the anneal target and alternate storage being marked by bold and dashed device perimeters as shown in the legend at 365).
• the data slice assignment for each selection state is shown by arrows from each of the four data slices (dsO, dsl, ds2, ds3 ) to the memory devices that form the operational set in that selection state.
• alternate storage FDO and anneal target FDl form the virtual device pair
• flash devices FDO, FD2, FD3 and FD4 form the operational set for new writes and update operations (read operations are alternately directed to the alternate storage FDO or the anneal target FDl according to the data evacuation state indicated by the page table)
• the slice assignments are dsO/FDO, dsl/FD2, ds2/FD3 and ds3/FD4.
• the slice assignment is repeated below the memory rank under the designation ''slice asn" to explicate the data relocation effort required in the transition from one selection state to the next (note that the data slice is implicit in the notation shown so that a slice assignment of "0,2,3,4 " ' means that data slice 0 is assigned to FDO, data slice 1 is assigned to FD2, data slice 2 is assigned to FD3 and data slice 3 is assigned to FD4).
• all four of the data slices are reassigned as shown at 366.
• each data slice reassignment generally involves relocation of the data from the device to which the slice was previously assigned to the newly assigned device
• the reassignment of all four data slices in the single progression from state k e' to state 'a' involves four times as much data relocation as the tour previous selection state progressions.
• relatively low-overhead (i.e., small footprint) control circuitry may be provided to implement the round- robin target selection and fixed slice assignment, such low overhead control circuitry corn ⁇ s at the price of a relatively burdensome data relocation effort once per cycle of the selection state progression (i.e., once per anneal cycle),
• Sj Figure 1OB illustrates an embodiment of the osciliating-progression, fixed-s ⁇ ee- assignment approach mentioned above.
• the memory subsystem progresses through a predetermined sequence of anneal targets marked by selection states 'a' through 'h, " and having a fixed slice assignment for each selection state.
• the direction of the progression sequence is reversed so that any incremental displacement of the slice assignment resulting from the initial-direction progression is reversed by the return-direction progression.
• the end-of-rank devices may be determined based on physical disposition (e.g., end-of-rank devices are those physically disposed at the end of a row of memory devices as shown in Figure I OB) or logically assigned without regard to physical location. So long as the progression direction is reversed once for ever ⁇ ' N devices annealed, the reverse-direction (counter-direction) progression through the sequence of anneal targets will restore the slice- assignment displacement that occurred in the initial progression direction.
• the oscillating progression sequence may be viewed as a manner of ensuring that each new anneal target is always paired with a physically adjacent device to form the virtual device pair. Thus, upon reaching the device at either end of the rank, the anneal progression is reversed to ensure that the device next selected for anneal is disposed adjacent the end-of-rank device.
• each complete anneal cycle i.e., each complete oscillation from selection state a to selection state h
• the end-of-rank devices are annealed only once per anneal cycle (i.e., complete progression through selections state 'a' to 'h"), while the devices sandwiched between the end-of-rank devices (the 'interior" memory devices) are annealed twice per anneal cycle.
• Figure 1 OC illustrates an embodiment of the round-robin-progression, slice- assignment-lookup approach mentioned briefly above.
• the same round- robin target-selection sequence is used as in the round-robin/fixed-sJicc-assignmcnt approach of Figure 1 OA (thus, five device selection states, 'a' through 'e ⁇ for the 4+1 architecture shown), but a variable slice assignment is applied to avoid the once-per-cycle multi-slice data relocation effort.
• the slice assignment for any data write is dynamically determined (e.g., through look-up or algorithmic generation) according to whether the write operation involves a new-data write, or a revision to the location and/or data content of previously written data.
• New data writes (i.e., those not directed to a logical address having a pre-existing page table entry) are assigned a fixed slice assignment according to the selection state, thereby providing determinism in the new-writ ⁇ slice assignment.
• data slices are assigned in device-ascending order such that the least significant slice of the write data is stored in the next device to be annealed and each subsequent slice is stored in sequentially higher numbered devices, rolling over from device N to device 0 as necessary.
• slice assignment 2, 3, 4, 0 is applied for any new data write (i.e., dsO in FD2, dsl in FD3, ds2 in FD4 and ds3 in FDO).
• slice assignment 3, 4, 0, 1 is applied for any new data write; in selection state 'c' slice assignment 4,0,1 ,2 is applied and so forth.
• host-requested update operations or controller- directed data relocations i.e., updates or relocations, collectively referred to as revisions
• revisions are effected by retrieving the previously recorded slice assignment from the page table and then reassigning any data slice assigned to the anneal target for the current selection state (i.e., the "current anneal target' * ) to the alternate storage to produce a revised slice assignment.
• updates arc completed by writing the entire updated data word to the operational memory set indicated by the revised slice assignment, while relocations may be effected by writing only the reassigned data slice in accordance with the revised slice assignment (i.e., reassigned data slice is retrieved from the anneal target (or scratch memory) and written to the alternate storage without rewriting the other slices of the subject data word).
• each data word within the operational memory set may have one of multiple possible slice assignments depending on whether the data word was rewritten (i.e.. updated or relocated) or newly written during the selection state. More specifically, any data word rewritten within a given selection state may be stored with any one of the N revised slice assignments possible when slices are assigned in a device-ascending order. To appreciate this, consider the operation of an empty memory subsystem in selection state 'a' (as shown at 368).
• selection state 'b' after data evacuation is complete in selection state 'c', all entries within anneal target FD3 have been staled-oiit through host-requested update or controller-directed relocation. Accordingly, all entries in the page table will reflect one of the three slice assignments shown for selection state 'c'
• slice assignment 3.4,1,2 instead of 3,0,1,2 for data originally written in selection state 'c' then rewritten in selection state 'd' (once-rewritten data); slice assignment 2.3,4,1 instead of 2,3,0.1 for data originally written in selection state 'b' then rewritten in selection state 'c' and rewritten again in selection state 'd' (twice- rewritten data); and finally, slice assignment 1 ,2,3,4 instead of 1,2,3,0 for data originally written in selection state 'a r then rewritten in selection state 'b', again in selection state 'c' and again in selection state 'd r (thrice-rewritten data).
• all entries in the page table will reflect one of the five slice assignments shown for selection state 'e' (i.e., all entries within anneal target FDO have been staled-out through host-requested update or through controller-directed relocation).
• each different slice assignment within a given selection state corresponds to rotation (i.e., modulo N shift) of the new-write slice assignment, with the assignment for each data slice being rotated by the number of selection state progressions (and thus the number of state- to-state rewrites) since the original data write.
• rotation i.e., modulo N shift
• the slice reassignrnents for selection state 'e ⁇ if one selection state progression has occurred (i.e., data newly written in selection state "(T).
• the new-write and revised slice assignments for a given selection state may either be looked up in a table or generated algorithmically (e.g., generated by logic circuitry within the anneal controller). In either case, a round-robin selection state sequence may be applied (thus ensuring a uniform number of anneal operations per device in each complete anneal cycle) without need for burdensome, multi-device data relocations in any single progression of the anneal-target selection state. Instead, data relocation in each selection state requires data transfer from anneal target to alternate storage only.
• FIG. 1 A illustrates a more detailed embodiment of a flash controller 380 capable of managing the run-time- varying operational sets and corresponding slice assignments described in reference to Figures 1 OA-IOC.
• the control logic 351 includes a command logic circuit 385 that receives memory access commands 388 from the host (i.e., host-supplied access commands, "Host Cmd' * ) as well as self-generated relocation commands 386 ("Reloc Cmd' * ) from the anneal controller 352.
• each memory access command includes or is accompanied by a logical address 394 that is input to the page table 353 to enable retrieval of a corresponding page table entry 395, if any.
• the page table itself includes a page-table controller 389 (e.g., formed by a state machine (FSM) or other control logic) together with a lookup table 391 (LUT).
• the page- table controller 389 responds to the input logical address 394 by indexing the lookup table 391 to determine whether a corresponding entry has been recorded. If the command logic 385 indicates that a memory write is to be executed, and the lookup table 389 contains no entry corresponding to the logical address associated with the write command, the page-table controller 389 deems the write command to be a new data write and issues physical address, chip-select and slice-assignment signals accordingly.
• the page-table controller 389 receives a selection-state value from the control logic 351 (i.e., indicating the current anneal-target selection state) and applies that information to determine the operational memory set and a free physical address therein. Thereafter, the page-table controller 389 outputs the physical address 397 (e.g., at a desired time with respect output of the corresponding memory command 396 by the control logic 351) via a control path 359 and activates the subset of chip-select signals 398 (conveyed via chip-select lines that may be deemed part of control path 359) associated with the operational memory set to enable the appropriate subset of memory devices FD0-FD5 to receive the physical address and corresponding memory command.
• the control logic 351 i.e., indicating the current anneal-target selection state
• output drivers 327 within the memory interface 361 are provided to output the memory command 396, physical address 397 and chip select signals 398 onto respective sets of signal lines that form external control path 392.
• the command and address signal lines of control path 392 are coupled in common to command ("cmcT) and address ("addr") inputs of each of the flash memory devices (FD0-FD4) of an N-H memory rank, while the chip-select lines are coupled to chip-select inputs ("c-sel”) of respective flash memory devices to enable their independent selection.
• command and address signal lines may also be coupled to respective memory devices (i.e., dedicated command/address bus per flash memory device as opposed to having multiple flash memory devices coupled in common to the same command/address bus), and either the command, address, and/or chip-select lines may be eliminated and the signals otherwise conveyed thereon time-multiplexed onto another of the sets of signal lines,
• the page-table controller 389 also determines a slice assignment value 399 corresponding to the new-data write for the specified selection state, and outputs the slice assignment value to a slice-steering circuit 402 (e.g., implemented by a cross-bar switch, "xbar," in the embodiment of Figure 1 IAj provided within memory interface 361.
• a slice-steering circuit 402 e.g., implemented by a cross-bar switch, "xbar,” in the embodiment of Figure 1 IAj provided within memory interface 361.
• the slice-steering circuit 402 is coupled between internal data lanes 401 (part of internal read/write data path 357) and data rec ⁇ iver/output-driver circuits 404 within data I/O circuitry 403.
• the N internal data lanes 401 each corresponding to a respective data slice (ds ⁇ -ds3), are output onto the appropriate subset of the N+ 1 data path slices (i.e., slices of external data path 405) coupled respectively to the N H memory devices of memory rank 406.
• the exemplary five-device architecture described in the preceding figures is carried forward; thus five flash memory devices (FD0-FD4) are organized in an NH topology in which four flash memory devices constitute the operational memory set for a given memory access and the remaining flash memory device constitutes the anneal target (in the case of write operations) or alternate storage (in the case of a read that maps to the anneal target).
• Each of the memory devices in this topology are said to be connected in a point-lo-poinl arrangement, that is, such that data lanes of each memory device are dedicated only to that memory device and are not connected with other devices.
• four internal data lanes 401 are provided to convey data slices ds ⁇ -ds3 and are switchably coupled, via slice-steering circuit 402, to a respective subset of five data I/O circuits 404 according to the slice assignment value 399 provided from the page table 353.
• the four flash memory devices that constitute the operational memory set will be selected (via assertion of corresponding chip-select signals 398 ⁇ to receive the memory write command and corresponding physical address, and the four outgoing data slices will be routed to the selected operational set in an order controlled by the slice assignment value 399.
• the anneal controller specifies selection state 'a/ the page- table controller will apply that information to determine (by lookup or algorithmic generation) an operational set formed by devices 0.2,3,4 and slice assignment 0,2,3,4 (i.e., dsO assigned to FDO, dsl assigned to FD2, ds2 assigned to FD3 and ds3 assigned to FD4) in the case of the anneal progressions for Figures 1OA and 1OB, or slice assignment 2,3,4,0 in the case of the Figure 1OC anneal progression.
• an operational set formed by devices 0.2,3,4 and slice assignment 0,2,3,4 i.e., dsO assigned to FDO, dsl assigned to FD2, ds2 assigned to FD3 and ds3 assigned to FD4
• the page-table controller 389 inserts a new entry corresponding to the input logical address 394 into the lookup table 391 to specify the physical address, operational set and slice assignment of the newly written data. Note that the page table entry may be created and inserted into the lookup table 391 prior to, concurrently with, or after the data write operation itself.
• the page-table controller 389 determines that the incoming logical address 394 does match an existing page-table entry and a host-directed memory-write command is indicated, then an update operation is to be carried out with respect to the specified logical address. Tn that case, the page-table controller 389 may- operate differently according to whether the slice assignment per selection state is fixed or variable. In the case of the fixed slice-assignment approaches (i.e., as discussed in reference to Figures 1OA and 10B), the page-controller 389 determines (e.g., by table lookup and/or algorithmic generation) the operational set and corresponding slice assignment for the current selection state, as well as an available physical address within the operational set.
• the page-table controller 389 outputs the physical address 397 via the control path 359 and raises the chip-select signals 398 associated with the operational memory set, and provides the slice-assignment value 399 to the slice-steering circuitry 402.
• the page-table controller 389 effects a data update operation by generating the new slice assignment in accordance with the last-recorded operational set and slice assignment.
• the last-recorded operational set matches the operational set for the current selection state (i.e., the selection state has not progressed since the last data write operation for the specified logical address)
• the same slice assignment is applied, but at a different physical address (as data is not updated in place, at least in the case of the exemplary flash memory subsystem of Figure 1 IA).
• the slice assignment recorded for the last selection state is applied to lookup or generate a new slice assignment to be applied in the update operation.
• the first digit of the last-recorded slice assignment and the current selection state may be used to index a lookup table and thus retrieve the current slice assignment. For example, if the current selection state ;;: 'b, r then the four possible indices from prior selection state 'a' will yield new slice assignments as follows: 2->l ,3,4,0; 0->0, 1 ,3,4; 4->4,0, 1 ,3; 3->3,4,0, 1.
• the new slice assignment may be generated by rotating the n ⁇ w-write slice assignment by the progression count (i.e., number of anneal progressions since the data was originally written); a value determined in turn by, for example, the offset between last digit of the new-write slice assignment for the current selection state and the last digit of the previously recorded slice assignment.
• the progression count i.e., number of anneal progressions since the data was originally written
• logical address 394 is provided in connection with a relocation command 386, then a controller-directed relocation operation is carried out with respect to the specified logical address.
• the page-table controller 389 may operate differently according to whether the slice assignment per selection state is fixed or variable. In the case of the fixed slice-assignment approaches (i.e., as discussed in reference to Figures 1OA and 10B), the page-table controller 389 determines (by lookup and/or algorithmic generation) the operational set and corresponding slice assignment for the current selection state, as well as an available physical address within the operational set.
• the flash controller 380 outputs the physical address together with a memory read command and the chip-select signal(s) corresponding to the memory device(s) to be accessed in the relocation operation. Data to be relocated is thus read from one or more selected memory devices and returned to the flash controller 380 for a shunt write to the alternate storage or to be stored for later write back to the alternate storage.
• the flash controller 380 upon receiving the data being relocated from the anneal target (and other devices if multiple slices of data are being relocated), the flash controller 380 asserts the chip- select signal(s) associated with the memory device(s) to which, data is being relocated, and asserts a write command and physical address to be written to enable data write in the selected devices.
• the read and write operations may be carried out back-to-back (i.e., without intervening operations over the data path 405) and thus the page table may be updated upon completion of the operation by revising the page table entry to include the alternate storage in the slice assignment in place of the anneal target, but without requiring change to the physical address.
• the page table 353 may be updated following the data retrieval phase to reflect storage in a scratch memory (indicating the storage address within the scratch memory, although a FIFO or stack architecture may be used to avoid explicit scratch memory addressing requirements), and then updated again following the data write phase to reflect the completed relocation operation.
• detail view 369 illustrates a general embodiment of an anneal controller having a finite state machine 381 (or processor sequencer or other control circuitry), one or more operation counters 382 and one or more anneal control registers 383.
• the state machine 381 transitions between states to manage the event sequence within a given anneal operation as well as the progression through different anneal target selection states in an anneal cycle.
• the operation counters are used to track various operations and events within the memory subsystem (e.g., memory accesses, program/erase operations, bad-block or other error/failure detections, etc.) and thus may be evaluated by the state machine 381 in progressing through the evacuation phases of an anneal operation, performing device selection, triggering anneal, etc. For example, by counting program/erase cycles during each target selection state, separate program/erase counts may be tracked for each memory device and used to trigger an anneal in that device, and/or manage the progression from passive to active evacuation modes.
• the operation counter may also include multiple counters used to track operations in respective memory devices.
• the operation counters 382 include a program/erase counter and error counter for each memory device, as well as a global access counter used to track the net number of memory write operations (and/or a combination of memory read and write operations) within the memory subsystem.
• the device-specific program/erase counts may then be incremented for the program/erase operations performed in the corresponding memory device, and the error counters used to count error detections in the corresponding memory devices (e.g., program/erase failure, bit errors as indicated by error-correction-codes or other error information, bad-block detections, etc.).
• the global access counter may be used, for example, to enable the anneal controller to distinguish between hot and cold pages within the memory subsystem (e.g.. even read operations have a bearing on whether a page is hot or cold).
• the global access count value is stored within the page table entry whenever a memory page is updated, thus enabling determination of the recency of the data storage (e.g., by comparing the stored access count value with the current access count value) and thus a determination of whether the page is hot or cold (e.g., by comparing with a recency threshold).
• the anneal control registers 383 may be used to store host-programmed (or operator-programmed) values as well as variables or other state information for controlling operation of the anneal controller. Tn the embodiment shown, for example, the anneal control registers (which may be a single register or group of registers having multiple fields) are used
• TD identifier
• the anneal trigger mode e.g., elapsed or calendar time, error rate, error count, program/erase count, etc.
• the anneal progression mode e.g., round- robin progression with fixed slice-assignment, oscillating progression, round-robin progression with variable slice-assignment, etc
• the subsystem topology e.g., sliced, multidrop, multi-channel, etc.
• the nature of the alternate storage (composite or unified), whether anneal operations are to be carried out in the foreground or background, whether to enable burst and/or compressed anneal (both discussed below), hot/cold (or other data usage grade) thresholds and/or policy (e.g., whether recency of access, frequency of access and/or other criteria are to be used to distinguish between
• anneal control registers 383 are shown within anneal controller 352 (and within flash controller 380), such registers or any subset thereof may alternatively or additionally be disposed within one or more (or all) of the memory devices within the annealable memory subsystem, or in a separate integrated circuit device, remote from the anneal controller (and flash controller) and memory devices.
• Figure 1 IB illustrates a consequence of the shared-command/address architecture of Figure 1 IA. That is, in a memory subsystem populated by one or more ranks of devices that exhibit localized regions of defective or unreliable storage, the defective regions in one memory device restrict access to otherwise functional regions in other devices of the same rank due to their common physical address. For example, if it is desired to stripe data across plural flash memory devices by effectively striping data across all devices at physically- corresponding locations, bad blocks within a flash memory device (see Figure 1 IB) could render corresponding good blocks within other flash memory devices unavailable. That is to say, in such an implementation, access to the otherwise functional blocks is restricted as the bad block would be effectively striped across all the flash, devices of the memory rank, multiplying the amount of unavailable storage by the number of memory devices of the rank.
• Figure HC illustrates an alternative rank-based memory architecture in which a flash controller 415 is coupled to each flash memory device FD0-FD4 of a memory rank 406 by a dedicated command/address path 417p.-417 4 , Further, the page table 400 within the flash controller 415 is provided (along with state machine 408) with multiple address lookup tables (LUTs) 409 to enable each incoming logical address 394 to be independently (and therefore individually and differently) mapped to a physical address within a corresponding flash memory device,
• LUTs address lookup tables
• bad block information for each memory device can be separately managed, as an alternative to striping. For example, an incoming or self- generated (i.e...
• virtual address 394 may be used to index each of the address lookup tables 409 simultaneously, thus providing multiple physical addresses that are output via respective address paths 416 to corresponding flash devices. More specifically, as shown in detail view 419, a single (shared) memory command 396 may be output to each of the flash memory devices in conjunction with a respective physical address (e.g., PhysAddr FDO, PhysAddr FDl , ..., PhysAddr FD4) and chip select signal (ChipSel FDO - ChipSel FD4, determined according to the slice assignment 399). As shown, the command/address information for each memory device is driven onto a corresponding one of the dedicated command/address paths 417o-41?
• a respective physical address e.g., PhysAddr FDO, PhysAddr FDl , ..., PhysAddr FD4
• chip select signal ChipSel FDO - ChipSel FD4
• bad-block addresses for each of the flash memory devices may be omitted from the virtual-to-physical address mapping for that memory device only, thus avoiding the need to stripe the bad-block address across the entire memory rank.
• the freedom to apply different physical addresses to respective memory devices during a given memory access operation also permits different wear-leveling progressions to be applied within the different memory devices (e.g., to account for access differences within the different memory devices due to dynamic slice assignment or other effects).
• circuit blocks and signals shown in Figure 1 1 C correspond to and may generally be implemented in the same manner as like-numbered circuit blocks and signals described in reference to Figure 1 I A.
• independent control over the flash memory devices of a memory rank may also be achieved using the shared cornmand/addrcss, dedicated chip-select topology of Figure 1 I A.
• memory access commands and/or memory addresses may be time-multiplexed onto the shared command/address path (i.e., sent one after another), with the dedicated chip-select lines being used to control which of the memory devices is enabled to receive a given command/address.
• additional latency may be incurred in issuing commands and addresses to the individual devices under such a topology, the accesses to the individual memory devices may be pipelined so that throughput (i.e., memory bandwidth) will remain unaffected.
• Figure 1 ID illustrates an embodiment of a parallel-lookup page table 425 that may be used to implement page table 400 of Figure 11C and thus enable dedicated
• command/address paths to respective memory devices within a memory rank.
• an incoming logical address is compared with the contents of a logical address table 428 for the page table entries, yielding a hit/miss signal 437 (indicating whether a matching logical address has been detected) and an index 436 of the matching logical address, if any.
• the index 436 is applied to address each of a set of lookup tables that correspond to respective flash memory devices, FD0-FD4. By this operation, the index 436 is used to simultaneously read out multiple physical addresses that correspond to respective memory devices (as well as any other device- specific data). As shown, the index is also provided to a unified lookup table to obtain the operational memory set, slice assignment, hot/cold page information and any other device- independent data (i.e., data that defines the overall memory access and thus need not be stored in multiple instances in the devices-specific lookup table).
• the slice assignment retrieved from the table (“Read SA") is selected via multiplexer 434 to be the slice assignment 399 for the memory access.
• the write slice assignment (“Write SA ' ') may be supplied by the control logic (e.g., according to the target selection state and progression policy) via control port 432 and selected (in response to a read/write specifier, "Rd/Wr,” also supplied via control port 432) by multiplexer 434 as the slice assignment 399 for the memory access.
• hot/cold information is recorded for each virtual address by storing an access count value within the corresponding entry of table 430. More specifically, the access count value (i.e., maintained by a counter within the anneal controller as described in reference to Figure 1 IA) is stored within lookup table 430 as part of each memory read or write (or alternatively only for a memory write access) to the corresponding virtual address.
• the control logic or anneal controller therein may supply the current access count to the page table along with a virtual address and thus enable retrieval of the stored access count value and determination of a difference between the stored access count value and the current access count value within recency logic 433.
• recency logic 433 receives a hot/cold threshold (e.g., from an anneal control register as described in reference to Figure 1 IA) and compares the difference between the stored and current access count values with the threshold. By this operation, recency logic 433 may generate a hot/cold signal 439 that indicates whether the difference between the stored and current access count exceeds the threshold and, accordingly, whether the page is hot
• a hot/cold threshold e.g., from an anneal control register as described in reference to Figure 1 IA
• Figure 12 is a flow diagram illustrating an exemplar ⁇ ' sequence of operations that may be carried out by the flash controller of Figures 7A-7B (and Figures 1 1 A, 11 C) in response to host-requested or self-generated memory access commands.
• the memory access command is a memory read (determined in decision block 451 )
• the flash controller executes the operational sequence shown at 461. More specifically, at 462, the flash controller indexes the page table using the logical address supplied in the memory access command and thereby obtains the physical address, operational memory set and slice assignment for the requested data. Thereafter, at 464, the flash controller executes the memory read at the physical address within the operational memory set, steering the retrieved data slices on the internal controller data lines according to the slice assignment.
• the flash controller determines whether the supplied logical address (i.e., supplied as part of or in association with the memory access command) is in the page table at decision block 453. Otherwise, the flash controller deems the memory access to be either an update operation or a relocation operation according to, for example, a flag indicating whether the supplied address is a logical address (update operation) or physical address (relocation operation) and/or a flag indicating whether the command and/or logical address was self-generated or host-requested.
• a flag indicating whether the supplied address is a logical address (update operation) or physical address (relocation operation) and/or a flag indicating whether the command and/or logical address was self-generated or host-requested.
• the flash controller executes the operational sequence shown at 471. More specifically, at 472, the flash controller determines the operational set according to the current selection state and then determines the physical address and slice assignment of the data write operation according to the operational set. Tn the embodiment shown, the operational set, physical address and slice assignment are determined by respective lookup operations; for example, indexing a operational-set lookup table (OS-LookUp) using the current selection state, indexing a next-fr ⁇ e-address lookup table (Addr-LookUp) using the operational set, and indexing a slice assignment table (SA-LookUp) using the operational set.
• OS-LookUp operational-set lookup table
• Addr-LookUp next-fr ⁇ e-address lookup table
• SA-LookUp slice assignment table
• the flash controller may determine the operational memory set, physical address and/or slice assignment algorithraieally, As a specific example of algorithmic assignment, the slice assignment may match (or otherwise be derived from) the operational memory set itself, in any case, the flash controller executes a memory write operation as shown at 474, writing to the physical address within the operational memory set and steering the write data slices from the internal data lanes of the flash controller to respective memory devices of the operational set in the order specified by the slice assignment.
• the page table is updated during or at the conclusion of the write operation to reflect the address translation (logical-to-physical), operational memory set and slice assignment. Note that the slice assignment need not be recorded in the page table if fixed according to the operational memory set or otherwise determinable.
• the flash controller responds to an update command (i.e., a memory write command for which the logical address is found to match a pre-existing entry within the page table) by executing the operational sequence shown at 481.
• an update command i.e., a memory write command for which the logical address is found to match a pre-existing entry within the page table
• the flash controller indexes the page table using the logical address to obtain the physical address, operational memory set and, if necessary to determine the new slice assignment, the slice assignment of the pre-existing data. Because the updated data value (i.e., the data to be written and supplied in association with the logical address) will be written at a new physical address within a potentially new operational set, the flash controller marks the physical address for erasure for each device of the operational set at 484.
• the flash controller determines the operational set for the current selection state, then determines the next free physical address and the slice assignment for that operational set.
• the slice assignment for the write operation may be determined without regard to the past slice assignment in a fixed-slice-assignm ⁇ nt embodiment (e.g., as in Figures 1OB or 10(7), while the past slice assignment is referenced to determine the next slice assignment in a variable- sliee-assignrn ⁇ nt embodiment (e.g., as in Figure 10A),
• the flash controller executes a memory write at 488, writing the updated data at the physical address within the operational memory set, and steering slices of the updated data from the interna!
• the page table is updated to indicate the new physical address as well as the operational set and slice assignment (either of which may be derived from the other and thus implicitly indicated by the page- table entry) of the updated data.
• the flash controller executes the generalized operational sequence shown at 491, starting at 492 by determining source and destination memory sets (SrcSet, DestSet) and assigning corresponding source and destination addresses (SrcPhysAddr, DestPhysAddr) for the relocation operation.
• the source memory set is a set of one or more memory devices from which the data to be relocated (the "transit data") is to be retrieved using the source address
• the destination memory set is a counterpart set of one or more devices to which the transit data is to be relocated (i.e., written) at the destination address.
• the source and destination memory sets reduce to the anneal target and alternate storage, respectively, and the source and destination addresses may be the same physical address (i.e., the physical address supplied with the relocation command).
• the destination memory set includes all (or at least a plural subset) of the devices of the operational memory set for the current selection state
• the source memory set includes all (or at least a plural subset) of the devices of the operational memory set for the preceding selection state.
• the command-supplied physical address constitutes the source address, but a new destination address is determined and applied (as the physical storage addresses within the nieniory device being both read and written requires erasure before re-use).
• the flash controller executes a memory read at the source address (SrcPhysAddr) within the source memory set (SrcSet) to retrieve the transit data (i.e., data being relocated).
• the source memory set reduces to the anneal target (e.g., embodiments of Figures 1 OB, 1OC and all selection state progressions within embodiment of Figure 1OA except from state 'e' to 'a')
• the anneal target is accessed (i.e., asserting the chip-select line for the anneal target, but not the other memory devices of the rank) to effect a single slice retrieval.
• each of those devices is accessed to retrieve a multi-slice transit data value, applying the slice assignment value for the source memory set (e.g., the slice assignment for selection state 'e' in the approach of Figure 10A) to route the individual data slices in ascending order onto the internal data lanes of the flash controller (i.e. dsO onto internal lane O, dsl onto internal lane 1 , etc.).
• the slice assignment value for the source memory set e.g., the slice assignment for selection state 'e' in the approach of Figure 10A
• the relocation sequence 491 in the case of a two-phase relocation operation, the transit data retrieved from the source memory set is written to the scratch memory at 495 and the page table updated to reflect the new storage location. Thereafter, at 497, the flash controller retrieves the transit data from the scratch memory address indicated by the page table, and proceeds to execute a memory write at 496. In a single-phase relocation operation, the scratch memory write and read operations at 495 and 497 are skipped (hence their presentation in dashed outline within Figure 12), and the flash controller proceeds directly from data retrieval at 494 to the data write at 496.
• the transit data is written to the destination address (DestPhysAddr) within the destination memory set (D ⁇ stSet), applying any slice reassignment as necessary.
• DestPhysAddr the destination address
• D ⁇ stSet the destination memory set
• multiple slices are written to the destination memory set. with slices being routed from the internal data lanes of the flash controller to respective memory devices according to the slice assignment for the destination memory set (e.g., applying the slice assignment value for selection state 'a' in the approach of Figure K)A).
• the source address within the source memory set i.e., the anneal target in the case of a single-slice relocation, or multiple memory devices in the case of a multi-slice relocation
• the source address within the source memory set is marked for immediate or eventual erasure at 498.
• Anneal operations may be carried out within a multi-drop topology in generally the same sequence as in a sliced topology, with the memory controller progressing to a new anneal target and alternate storage for each of a sequence of target selection states.
• Figure 13 illustrates this operation, showing a flash memory controller having the same general structure described above (i.e., including control logic 501 with integral axmeal controller 503, page- table 505, scratch memory 507, infernal data path 509, control path 51 1 , host I/O (not shown) and memory I/O 513), but coupled to a set of flash memory devices 517 (FD0-FD4) via multidrop data path 516 instead of respective slices of a sliced data path.
• each of the flash memory devices 517 is part of a respective rank of memory devices (i.e., data path 516 is but one slice of a sliced data path as shown, for example, at 322 of Figure 6A) and each memory rank includes one or more spare memory devices
• the rank-based anneal management described above may be applied to anneal each memory device in each rank. More specifically, the spare device within a given rank may be used as the alternate storage and thus paired with the anneal target for the selected rank to form a virtual device for memory read, write and relocation purposes.
• the anneal target may be sequenced either by progressing through the memory devices of each rank before proceeding to the next rank, or by progressing through the memory devices coupled to a given slice of the data path (i.e., progressing through the '"slice" of memory devices) before proceeding to the next slice.
• flash devices FDl and FDO constitute the anneal target 530 and alternate storage 531 , respectively, of a virtual device 533.
• anneal operations within such an N+ slice architecture may be carried out by re-directing new writes and updates from the anneal target 530 to the alternate storage 531 , and directing memory
• -M- reads to either the alternate storage 531 or the anneal target 530 according to the device specified in the page table 505 for the address of interest. Also, if data is retrieved from the anneal target as part of a memory read, the data may be copied to the scratch memory 507, setting a flag in the corresponding page-table entry (and recording the scratch memory location in the page table entry or an associated location) to enable subsequent transfer from scratch memory 507 to alternate storage 531.
• entries within page table 505 may follow generally the same format as the page table entries for the N+l rank architecture, except that the physical address includes not only the internal memory address of the stored data, but also a device identifier or number (i.e., device address) to identify the physical storage device within the slice of devices.
• the page table may be updated after each write or update operation to show the alternate storage 531 as the repository of the updated or newly written data and, if an update, to invalidate (or mark for erasure) the corresponding entry within the anneal target 530.
• the source location within the anneal target may optionally be invalidated or marked for erasure (i.e. via page table update), thus leaving the scratch memory as the only valid source of the retrieved data.
• the source location within the anneal target 530 may not be invalidated/'marked-for-erasure until after the data is written to the alternate storage 531 to complete the relocation operation.
• the memory devices of a given slice are generally not simultaneously accessed and thus need not be constrained by slice-ordering considerations. Accordingly, any of the memory devices of the slice may be used as alternate storage 531 for a given anneal target 530, and the alternate storage selection may change from one anneal cycle to the next. More generally, as shown in Figure 14, the alternate storage need not be a single memory device, and instead may be a composite storage 540 that includes available storage regions in two or more other devices of the slice. Tn that case, the composite alternate storage need only have sufficient capacity to hold the valid content of the anneal target and thus may constitute a substantially smaller storage area than the total capacity of the anneal target.
• data relocated from anneal target to the composite storage may coexist with other data originally written within the memory devices that contribute the composite storage).
• shaded regions that denote the component regions of the composite storage two or more non-contiguous regions may be contributed by one memory device, and different amounts of storage may be contributed to the composite alternate storage by each participating device.
• Figures 15-19 illustrate anneal operations carried out within a multi-channel runtime annealable memory system.
• memory devices coupled to the memory controller via respective channels are generally independently controlled. In one embodiment, for example, separate
• command/address paths are provided to the devices of each channel (i.e., each set of same - channel devices share a command/address path that is separate from the command/address path coupled to another set of same-channel devices) so that entirely different memory transactions may be carried out concurrently (i.e., at least partly overlapping in time) within the different channels.
• a memory read at address X within flash device 3 of channel A i.e., device FD A3, coupled to flash controller 550 via memory channel 516 A
• FD BO memory write to flash device 0 of channel B
• the virtual device pair may be formed by devices from respective channels, In one embodiment, shown for example in Figure 15, devices from the same slice position (0, 1 , 2 or 3) but respective memory channels (516 A, 516B) are selected to form a sequence of virtual device pairs, annealing each of the devices of a given memory channel in turn before reversing the anneal target/alternate storage selection to progress through the devices of the other channel. In the embodiment of Figure 15, for example, the progression may be:
• FDAO/FDBO FDA 1/FDB1. ..., FD An-I /FDBn- 1 , FDBO/FDAO.
• FDB1/FDAI ..., FDBn- 1/FDAn- 1 , where the first device in the format FDi/FDj is the anneal target and the second is the alternate storage, and 'n' is the number of devices within each channel.
• the progression may alternate ("ping-pong") between the two memory channels as in
• one or more virtual device pairs in each progression may be formed without regard to device position.
• devices FDAl and FDB3 are paired.
• This embodiment may be useful, for example, where one channel exhibits spare capacity.
• thai device may be used as the alternate storage of not only the devices of channel A (e.g., evacuating from a given device within channel A into FDB3, and then reloading the evacuated device after it has been annealed so that FDB3 is free to receive the data of the next device in the progression), but also the other devices of channel B.
• the alternate storage within a multi-channel topology need not be a single memory device, and instead may be a composite storage that includes available storage regions in two or more other devices of the same and/or different channel Referring to Figure 17, tor example, the alternate storage is formed by storage regions within two same-channel flash devices (FD AO and FD A2), and two different- channel flash devices (FDB2, FDB3).
• FD AO and FD A2 two same-channel flash devices
• FDB2, FDB3 two different- channel flash devices
• the composite storage need only have sufficient capacity to hold the valid content of the anneal target and thus may constitute a substantially smaller storage area than the total capacity of the anneal target. Also, different amounts of storage may be "contributed" to the composite alternate storage by each participating device,
• FIG. 18 illustrates an embodiment of a multi-channel memory controller 570 (a flash controller in this example) having switched memory I/O circuitry 573 to facilitate channel-to- channel data transfer and thus speed data relocation operations.
• data is read from the anneal target (FD AIj via the channel-A I/O circuitry 572 A , then routed via a cross-bar switch 574 (though other switching technologies may be used) to the I/O circuitry 572 B for channel B to provide write data for a write into an alternate storage (FD B3) on that channel.
• FD AIj a cross-bar switch 574 (though other switching technologies may be used) to the I/O circuitry 572 B for channel B to provide write data for a write into an alternate storage (FD B3) on that channel.
• FD B3 alternate storage
• a relocation from an anneal target in channel A to an alternate storage in channel B is effected without requiring the data to be transferred to the controller core 571.
• an additional memory channel was present, for example, unfettered
• Figure 19 illustrates an alternative channel-to-channel data relocation operation within the multi-channel memory controller 570 in which data is read from the anneal target (FD A i) on channel A, stored in a scratch memory 585 within controller core 571 (and optionally returned to a host requestor via data read/write path 587) and then later written to an alternate storage (FD B3) on channel B.
• FD A i anneal target
• FD B3 alternate storage
• the data retrieval flow is shown by 583 (bold dashed path) and the data write flow as by 585 Tn this example, data operations within the two channels may be carried out concurrently such that another memory write or memory read is earned out (at least in part) in channel B while transit data (i.e., the data being relocated) is read from the anneal target in channel A, and then another memory write or memory read is carried out (at least in part) in channel A while the transit data is written to the alternate storage in channel B. That is, the host can be configured to access data (reading or writing) over only one of the memory channels at a time, thus freeing up the other channel for ann ⁇ al-related data evacuation operations without interfering with host- initiated traffic.
• the scratch memory 585 may serve as a temporary holding location for data being migrated (i.e., transit data) from a device on one channel to one or more devices on the other channel
• the scratch memory 585 may similarly be useful for temporarily storing transit data to be written back to an alternate storage on the same memory channel as the anneal target.
• the page table 583 may be updated to reflect the temporary location of the transit data so that any requests to read the transit data prior to relocation in alternate storage may effect data retrieval from the scratch memory 585 instead of requiring access to the anneal target.
• the page table is updated following completion of the data relocation operation to show the channel-B alternate storage as the new r repository of the data.
• Figures 20A-20E illustrate anneal operations within a N+2 memory architecture. That is, each rank or slice of devices includes two spare devices, thus permitting one spare device to be annealed (i.e., heated) simultaneously with data evacuation into the other spare device.
• the memory controller 600 (a flash controller in this example, as the memory rank is populated by flash memory devices FDO- FD5) includes control logic 601 (with integral anneal controller 602), page table 603, scratch memory 605, command path 61 1 , data read/write path 607, memory I/O circuitry 609 and controller I/O circuitry (not shown), all of which operate generally as described above.
• Respective slices of external data path 614 are coupled between the memory I/O circuitry 609 and each of the flash memory devices FD0-FD5, and a command/address path (not shown) is coupled in common to all the devices of the rank, except that individually controllable chip- select lines are coupled in dedicated fashion to each memory device to enable their independent selection as discussed in reference to Figure 11.
• each memory device may be coupled to a respective (i.e., separate, dedicated) address path and/or a respective command path, instead of being coupled to a common command/address path.
• Such an arrangement would allow each memory device to receive a different physical address within a given memory access and thus enable tolerance for bad blocks at different physical locations within the memory devices and provide support for device-specific wear-leveling.
• evacuation operations and anneal operations may be pipelined as shown in Figure 2OB, with stepwise progression of the evacuation device (ev), anneal device (xx) and alternate storage (as) from selection state to selection state (progressing from state 'a' through state T in each anneal cycle) as shown in Figure 2OC.
• flash device 0 Flash Dev 0
• flash device 2 constitute the alternate storage and evacuation device, respectively, of a virtual device, while flash device 1 is annealed
• Figure 2OE illustrates another advantageous characteristic of a system having two spare devices. That is, after an anneal operation is complete within a given memory device, that device together with the new anneal target may be omitted from the set of devices accessed during a given memory access cycle (e.g.. during memory writes) and thus available for a concurrent data evacuation operation.
• a data relocation operation is carried out with respect to just- annealed device FDO and the new anneal target FDl, concurrently with a host-requested memory access within one or more of memory devices FD2-FD5.
• a shunt transfer from FDl to FDO is effected through the memory I/O circuitry, thus enabling collision-free communication between the two devices concurrently with the host-requested traffic.
• Figures 21A-21D illustrate an anneal progression, and alternative anneal pipelines that may be employed in an N+3 sliced or multi-drop architecture.
• the anneal operation applied to a given device within memory rank 640 extends over two successive selection states in an anneal progression and that two devices are concurrently annealed in each selection state.
• FIG. 21B illustrates an anneal pipeline corresponding to the selection-state progression of Figure 21 A 5 demonstrating the staggered, but concurrent anneal of multiple memory devices within the same rank or slice,
• the anneal progression and pipeline of Figures 21 A and 2 IB is particularly applicable where the anneal operation is expected to take longer (at least on occasion) than device evacuation operation and where a higher anneal frequency is desired. That is, on average (N+3)*2 devices are annealed per execution of each complete anneal cycle (i.e., progression through all selection states).
• N+3*2 devices are annealed per execution of each complete anneal cycle (i.e., progression through all selection states).
• the evacuation operation is expected to take longer (at least occasionally) than the anneal operation, it may be desirable to evacuate two memory devices concurrently during the anneal of another.
• FIG. 21C The anneal pipeline for such an embodiment is shown in Figure 21C, showing that, for each selection state, one device is annealed (the anneal device), while the evacuation of another device (the leading evacuee device) is concluded and the evacuation of a third device (the trailing evacuee) is initiated.
• Figure 21D illustrates this approach in the context of an N+-3 memory system that includes controller 630 and an anneal-capable memory subsystem populated by seven memory devices, FD0-FD6. As shown, over the interval in which FDO is being annealed (i.e., heated to an annealing temperature), FDl is evacuated and then FD2 is evacuated, while devices FD3-FD6 form the operational memory set.
• Figure 2 IE illustrates yet another evacuation methodology that may applied within an N-'- 3 memory system.
• two memory devices within rank 640 are evacuated concurrently while annealing a third memory device. More specifically, in the embodiment shown, the memory device next to be annealed is actively evacuated (as indicated by "ae") into a first alternate storage (as) during an interval in which another of the memory devices is passively evacuated into a second alternate storage (as).
• the device being heated during the passive and active evacuation of the evacuee devices is marked by ''xx.' *
• the memory device actively evacuated in selection state 'a' becomes the anneal device (xx)
• the memory device passively evacuated in selection state 'a' becomes the actively-evacuated device (ae)
• the device annealed in selection state 'a * becomes the alternate storage (as) for a device newly selected for passive evacuation (pc)
• the evacuation devices and their respective alternate storage devices constitute virtual device pairs that form part of the operational memory set.
• the operational memory set consists of two individual memory devices ('"op") and two virtual device pairs (ae/ ' as and pe/as).
• Different numbers of memory devices may be included within the memory rank in alternative embodiments and the number of concurrently evacuated devices (and thus the number of virtual device pairs included in the operational memory set) may be extended to any practicable number.
• progression arrangements, operational pipelines and architectures other than those shown in Figures 21 A-21E may be employed, and the concepts demonstrated in the N+3 sliced or multi-drop architecture may be extended to architectures having device counts of
• Figure 22 illustrates a generalized sequence of operations that may be carried out in a foreground anneal 130; a run-time anneal operation in which the core operation of the system is suspended while the annealable memory subsystem (or subsystems) is annealed and then resumed when the anneal is complete.
• a foreground anneal sequence begins in response to occurrence of a triggering event.
• the triggering event may be an automatically determined/detected condition or set of conditions 651 or operator-initiated 655.
• the triggering event may be deterministic (e.g., detecting that a predetermined or programmed amount of time has elapsed since the last anneal operation, or that a particular point in time has been reached (i.e., scheduled)) or non- dctcrmimsric (e.g., system idle for prcdeterrnined''programmcd period of time, threshold number of program/erase operations since last anneal, threshold number/rate of errors, etc.) or any combination thereof.
• dctcrmimsric e.g., system idle for prcdeterrnined''programmcd period of time, threshold number of program/erase operations since last anneal, threshold number/rate of errors, etc.
• a human or artificial operator of the system may specifically instinct anneal operation, for example, through use of an operating system utility or other user-interface selection, thus affirmatively choosing to place the system in a suspended state pending completion or termination of the anneal operation.
• a human operator may determine (through indicators or otherwise; that the system performance may be improved by annealing the memory subsystem and choose a control panel utility or other maintenance control interface in which to launch an anneal operation.
• the user interface may be used to track the progress of the anneal, showing the device or devices currently under anneal, the time remaining on the current anneal operation, the device or devices remaining to be annealed (and an estimated time for all or each) as well as the device or devices already annealed.
• Any problematic memory e.g., bad or unreliable sector, cluster device, rank, etc.
• memory avoidance e.g., marking the problematic memory as unreliable and not to be used
• the foreground anneal operation proceeds in generally the same manner in each case: evacuating data from the memory to be annealed to a backup
• -7Q- storage or spare annealable device(s) 133 executing the anneal (i.e., device heating) operation 135, and then re-ioading the annealed memory with any evacuated data 137.
• anneal i.e., device heating
• data evacuation is to be performed prior to anneal at 135, then data may be evacuated to a backup storage that is distinct from the annealable memory subsystem as shown at 661 (i.e.. into another storage within or separate from the host system containing the annealable memory subsystem), or may be evacuated into spare storage of the annealable memory subsystem itself as shown in 663.
• the various approaches employed for evacuating an anneal target for purposes of background anneal may likewise be employed to evacuate anneal targets as part of a foreground anneal sequence.
• the constituent memory devices of the annealable memory subsystem may either be annealed all at once (referred to herein as a "parallel" anneal) as shown at 671, or sequentially by device or group of devices as shown at 673.
• a parallel anneal any data evacuation may be earned out with respect to the anneal target and one or more alternate storage devices as described above in the context of background anneal.
• the selection between parallel anneal and sequential anneal may be fixed according to the application, or programmatically established. In either case, a number of factors may be taken into account and/or programmatically specified as impacting the selection between parallel and sequential anneal.
• parallel anneal may be enabled only when the device is docked or otherwise powered by line power (i.e., not battery powered). Conversely, if line power is available, then the provision of limited device functionality (or concurrent execution of other maintenance operations) may be weighed against the urgency of the anneal operation in selecting between parallel and sequential anneal.
• the memory re-loading operation at 137 is optional and, if desired or necessary, may be explicitly executed (e.g., transferring evacuated data back into annealable memory subsystem) and/or effected through a data migration that occurs as normal operational traffic, in either case, after any required memory re-load has been completed, the core operation of the host system may be resumed.
• FIGS 23A-23C illustrate a type of foreground anneal referred to herein as a "burst anneal.'"
• the general sequence of a burst anneal is shown in the exemplary flow diagram of Figure 23 A. That is, the memory subsystem or appliance containing the memory subsystem is taken off-line at 701.
• anneals are executed according to the background anneal methodology described above except that, (i) the anneal controller proceeds directly to active evacuation, evacuating a memory device concurrently with annealing (heating) of another memory device, (ii) migrations are performed through shunt transfer (e.g., transfer through cross-bar or other switching fabric within the input/output circuitry of the memory controller) and thus without temporary storage in scratch memory, and/or (iii) multiple anneals and/or data evacuations are performed in parallel in respective threads of an anneal sequence.
• the memory subsystem or appliance is placed back online at 705, enabling memory access operations to resume.
• Figure 23B illustrates a burst-anneal within a flash memory system having a flash controller 720, and set of flash devices ("Flash' ' ),
• the flash controller itself includes a crossbar T/'O circuit 726, scratch memory 721 , page table 723 and read/write data path
• Read/Write Path' * each of which function generally as described above.
• data is actively evacuated from one flash device to the most recently annealed flash device concurrently (i.e., at least partly overlapping in time) with an annealing (heating) operation within a third flash device.
• the fully -evacuated flash is annealed (heated) concurrently with active evacuation of another flash device into the most recently annealed device, as shown at 727.
• Figure 23C illustrates an alternate burst anneal sequence within the same flash memory system as Figure 23B, but with multiple anneal (heating) operations and multiple data evacuations carried out simultaneously.
• Such multiple simultaneous anneal/evacuation operations are referred to herein as anneal threads, as each operation may be carried out as an instance of the single-threaded burst anneal operation described in reference to Figure 23B.
• Both the singular and mu Hi -threaded anneal operation leverage concurrency in the component operations of the individual anneal threads (and the multithreaded operation further leverages concurrency between two or more such individual anneal threads) to accelerate annealing of the overall memory rank, enabling the entire memory rank to be more quickly annealed and returned to service,
• Figure 24 illustrates an exemplary compressed anneal operation that may be carried out to provide a spare memory device within a memory subsystem otherwise having no spare device, or to provide additional spare memory devices for purposes of leveraging the multiple-spare techniques described above (e.g., N+2, N- ⁇ -3, etc. as well as the off-line burst anneal techniques described in reference to Figures 23A-23C).
• the run-tirn ⁇ ann ⁇ alabl ⁇ memory system or appliance including same is tak ⁇ n off-line.
• the data storage requirement is reduced to free up one or more spare (or additional spare) memory devices.
• a number of different techniques may be carried out individually or in combination to reduce the net data storage requirement of the data stored w r ithin the memory system (i.e., the data volume) including encoding all or part of the data volume, for example, using vector quantization, data dc-dupiication or other lossless or lossy techniques; moving a portion of the data out of the memory system; and/or invalidating data confirmed or known to be redundantly stored in other storage systems, such as disk, main memory, cache, etc (e.g., merging two or more semiconductor-based RAID -5 memory subsystems such that a memory device (or devices) used to maintain parity data for one of the subsystems is at least temporarily not needed for that purpose and thus available as a spare memory device).
• the data selected for transfer may be identified according to a predetermined or programmatically- sel ⁇ cted policy (e.g., stored within an anneal control register as discussed in Figure 1 IA) that is biased toward transference of data less likely to be valuable (needed) within the memory subsystem (or host flash appliance), such as cold pages instead of hot pages.
• a predetermined or programmatically- sel ⁇ cted policy e.g., stored within an anneal control register as discussed in Figure 1 IA
• the now-smaller data volume is compressed into a reduced number of memory devices at 755, yielding one or more spare or additional spare memory devices as in the N+ architectures described above (i.e., N-'- 1 , N ⁇ -2, N-t-3, etc.).
• anneal operations may be executed at 757 according to the background anneal methodology and burst anneal methodology described above even if the original storage arrangement included no spare devices.
• data may be decoded or transferred back into the memory subsystem as necessary (as shown at 759) to enable resumed operation when the memory subsystem or host flash appliance is placed back online at 761,
• Figure 25A illustrates an exemplary user interface 680 that may be presented to a human operator of a run-time anncalable system in connection with foreground and/or background anneal operations. More specifically, upon launching an "Anneal Manager * ' program (i.e., instructing the system processor(s) to execute a sequence of instructions that display user interface 680 on an operator-viewable display, and that carries out the anneal operation in accordance with operator input), the system may gather status information relating to the anneal (e.g..
• the Anneal Manager user-interface 680 includes exemplary pull-down menus captioned "File,” “Action,” and "Help.”
• the File menu provides user-selectable menu items including “Preferences,” “Print Report” and “Exit” as shown at 683, while the Action menu provides menu items “Select Targei(s),” “Start .Anneal,” “Abort Anneal,” and “Exit” as shown at 685.
• Menu items for the Help menu are not shown, but may include various help and status items to enable the user to gather information about the anneal operation and the various options available via the pull-down menus and general user interface.
• Preferences may be selected to obtain a preferences window for operator entry of various options and preferences (parameters) relating to the anneal. For example, the operator may be prompted to select/enable: whether to enable background anneal whether to perform parallel anneal or sequential anneal; various visible/audible warnings that the system may issue to indicate the need for anneal; thresholds relating to passive and active evacuation stages; triggering event criteria/policy, device selection criteria/policy (i.e., how system will select anneal target and/or alternate storage, including programmable selection sequences), page table management options, topology- specific choices/ options, anneal progression methodology, scratch memory usage policy, and so forth. More generally, any of the optional and variable anneal-related parameters (and/or selections, operations, etc.) described herein may be presented to the user via the preferences window (or other user-interface display available in connection with the Anneal Manager program).
• the '"Print Report” menu item may be selected to print various reports relating to past or ongoing anneal operations, and the "Exit” menu item may be selected to terminate execution of the Anneal Manager program (utility).
• menu item "Select Memory” may be used to specify specific memory subsystems and/or specific memory devices within a given memory subsystem to be annealed.
• the start anneal and abort anneal menu items may be selected to initiate and abort, respectively, anneal of the selected memory devices (or targets captured by the preferences settings). Additional/different menu items and/or pull-down menus may be provided within the anneal manager program according to application needs.
• a volume window 691 is provide to list each of the armealable memory subsystems present in the system and to show the anneal status (e.g., anneal required, anneal in progress, up to date), storage capacity, unused storage, life remaining (e.g., a gauge showing lifetime left in a non-volatile memory device), date/time of last anneal operation and estimated time to complete an anneal operation (e.g., time remaining if anneal in progress, or total time required if anneal not in progress) of each volume.
• the anneal status e.g., anneal required, anneal in progress, up to date
• storage capacity e.g., a gauge showing lifetime left in a non-volatile memory device
• date/time of last anneal operation e.g., estimated time to complete an anneal operation (e.g., time remaining if anneal in progress, or total time required if anneal not in progress) of each volume.
• such values may be dead-reckoned based on stored parameters, determined based upon periodic measurements. Such determinations and measurements may also be used to initiate maintenance events (i.e., software-initiated, maintenance events involving data migration). Also, a user may be prompted to optionally initiate an anneal when a predetermined (e.g., programmed) amount of life left within a given device or volume, and/or anneal may be automatically triggered when the life remaining has reached a critical threshold (e.g., 5% or 3%— again, a predetermined or programmed threshold).
• a critical threshold e.g., 5% or 3%
• An anneal progress window 693 is also presented to show the status of any anneal in progress.
• the anneal operation in progress within subsystem X is 67% complete and, to this point, has reclaimed (i.e., recovered from a previously unusable state) 124 gigabits (Gb) of storage capacity as shown at 697.
• User-selectable control buttons i.e., display images that may be activated or "c Helved" through user operation of a mouse or other pointer device
• control button 701 controls the operator to start a foreground anneal operation
• control button 703 abort an ongoing anneal operation
• control button 705 to display an anneal status window
• the control buttons 701 , 703, 705 may be rnodally available or unavailable to the operator according to the state of the system.
• the ''start anneal button 701 is unavailable (as shown by it's shadowed outline) as an anneal is already ongoing (note that multiple subsystems may be annealed concurrently in the background and/or foreground, if system resources permit).
• An exemplary anneal status window 71 1 is shown at 710, and includes an image 715 of the memory subsystem undergoing anneal (subsystem X in this case), showing the constituent devices for which the anneal is complete, those for which anneal is ongoing and those not yet annealed.
• the storage volume to which the status display pertains and the estimated time remaining are repeated in the example shown. Numerous other status values and detail may be presented, including test results following anneal, memory devices that remain suspect or known to be fully or partly inoperable, and so forth.
• the anneal status window 711 corresponds to a foreground anneal operation in which numerous memory devices (but not all) are annealed concurrently. The anneal status window may also present status with regard to any ongoing background anneal operations.
• the user- interface of Figure 25A may be presented through execution of an operating- system utility (e.g., a Microsoft Windows ("Microsoft" and "Windows” may be protected by one or more trademarks of Microsoft Corporation) control panel application or like software-implemented utility.
• an operating- system utility e.g., a Microsoft Windows ("Microsoft" and "Windows” may be protected by one or more trademarks of Microsoft Corporation) control panel application or like software-implemented utility.
• such software-implemented utility may, when executed, monitor the need for a memory maintenance event, where the memory is a element (or component) of a multiple element storage system and interact with a user regarding selection of data evacuation options.
• the software utility may initially prompt a user with a set of one or more options for rerouting data from the memory at 767 and, in response to user selections/specifications, passively evacuate the memory during run-time of the host system. Thereafter, the software utility may prompt the user a second time at 769 (e.g., after a predetermined condition relating to evacuation has been satisfied) with a set of one or more additional options including, for example and without limitation, (i) performing system- supervised performance of the maintenance event and/or performing active evacuation of the memory, in which contents of the memory are moved and remapped to another of the elements of the storage system.
• the above-described user-interface and related functions may be implemented by computer/processor execution of a software program (i.e., executable instructions that may be stored on a machine readable media).
• a software program i.e., executable instructions that may be stored on a machine readable media.
• the anneal manager may be part of a configuration/management tool provided in connection with an operating system, in at least one embodiment, the anneal manager may permit a user to periodically assess estimated remaining device lifetime and/or implement a user-directed anneal without awaiting automated detection of the need for a maintenance event.
• the anneal circuitry used to heat selected int ⁇ grated-circuit devices to annealing temperatures may be implemented on-die, in- package or in-system.
• Figure 26 illustrates an embodiment of a flash memory system, for example, having a combination in-system and in-package anneal circuitry. More specifically, a flash controller 780 is provided with an anneal-control interface 782 to a heater/controller device 781 (e.g., itself an integrated circuit) that provides heating control and/or power to a set of in-package heaters 787 via control path 784, with each in-package heater being used to heat one or more flash memory devices ("Flash'") within the package sequentially or
• a flash controller 780 is provided with an anneal-control interface 782 to a heater/controller device 781 (e.g., itself an integrated circuit) that provides heating control and/or power to a set of in-package heaters 787 via control path 784, with each in-package heater being used to heat one or
• the heater/controller and/or in-package heaters may be powered by higher voltages (V A ) than those used to power logic-level integrated-circuits (or at least by dedicated or isolated supplies and/or by supplies capable of powering heater loads), and thus specially equipped to deliver the power necessary for annealing operations.
• V A voltages
• the heater/controller and/or in-package heaters may be powered by higher voltages (V A ) than those used to power logic-level integrated-circuits (or at least by dedicated or isolated supplies and/or by supplies capable of powering heater loads), and thus specially equipped to deliver the power necessary for annealing operations.
• a dynamic random access memory (DRAM) 785 or any other type of memory may be coupled to the flash controller via interface 786 to provide a scratch memory (the scratch memory may alternatively be included within the flash controller), and one or more high-speed interfaces (for example and without limitation, a second generation, extended peripheral component interconnect bus (PCIe Gen2)) 106 may be provided to enable one or more host devices to access the flash controller and thus the flash memory system.
• Command, address and data paths 108a may be coupled between the flash memory devices and flash controller 780 according to any topology appropriate to the system application.
• FIG. 27 illustrates an embodiment of a muiti-die flash memory package 800 having in-paekage heating structures and that may be used within the flash memory system of Figure 26.
• the flash memory package includes a set of four flash memory dice 803 1 -803 4 (more or fewer dice may be provided) stacked on a multi-layer (laminated) substrate 801 and attached via respective die-attach heating blocks 805.
• the multi-layer substrate includes conductive distribution structures to couple die-interconnect pads at an exposed surface of the substrate to solder balls 815 or other circuit- board interconnect structures on the underside of the substrate, with wire -bonds 807 or other die-interconnect structures coupling the die-interconnect pads to counterpart pads on the memory dice themselves.
• the stacked dice 803 and die-interconnect structures 807 may be encapsulated by a molding 809 or other housing.
• each die attach heating block 805 may be implemented as shown at detail view 810. That is, a top layer 812 may be formed by a thermally-conductive die-attach tape or film approximately 12.5 to 25 microns thick and having a thermal conductivity constant ("k") of approximately 2 watts per meter* degree Kelvin (W/mK).
• k thermal conductivity constant
• a heating block or element itself forms the middle layer 814 and may be implemented by any one (or combination) of a silicon spacer (e.g., forming a heat-producing resistive element) approximately 25 to 50 microns thick; a polyimide/FEP ThermofoilTM heater (approximately 75 microns thick) or a silicon-on-insulator (SOT) layer approximately 1 to 5 microns thick.
• a silicon spacer e.g., forming a heat-producing resistive element
• a polyimide/FEP ThermofoilTM heater approximately 75 microns thick
• SOT silicon-on-insulator
• -8Q- may be protected by one or more trademarks of Minco Corporation of Minneapolis,
• a bottom layer 816 may be formed by a thermally-insulatrve die-attach film or tape having a thickness in the range of 12.5 o 25 microns and a thermal conductivity constant of approximately 0.2 W/mK. Note that the thicknesses, thermal conductivity constants, quantity and implementing material/structures of the layers of each die attach heating block arc provided as examples only and may be varied in alternative embodiments without departing from the intended scope of the disclosure herein. More generally, the stacked-die package shown in Figure 27 is but one example of an in-package heating arrangement that may be used within the system of Figure 26 (or any others of the memory systems disclosed herein). Other heating arrangements, including in- system and on-die heating arrangements as discussed above may be employed in alternative embodiments.
• the user- interface 680 of Figure 23 and the underlying program code (i.e., sequences of instructions and related data) used to present the user- interface, receive user-input and execute the underlying anneal management operations may be recorded on one or more computer-readable media for later retrieval and execution within one or more processors of a special purpose or general purpose computer system or consumer electronic device or appliance.
• Computer-readable media in which such instructions and data may be embodied include, but are not limited to, machine-readable storage media in various forms.
• machine-readable media include, for example and without limitation, optical, magnetic or semiconductor storage media such as floppy disk, CD, DVD, active RAM within a computer, or any other type of physical device from which the instructions may be extracted under full or partial control of a machine.
• machine-readable media include, for example and without limitation, optical, magnetic or semiconductor storage media such as floppy disk, CD, DVD, active RAM within a computer, or any other type of physical device from which the instructions may be extracted under full or partial control of a machine.
• the various circuits disclosed herein may be described using computer aided design tools and expressed (or represented), as data and/or instructions embodied in various computer-readable media, in terms of their behavioral register transfer, logic component, transistor, layout geometries, and/or other characteristics.
• Formats of files and other objects in which such circuit expressions may be implemented include, but are not limited to, formats supporting behavioral languages such as C, Verilog, and VHDL, formats supporting register level description languages like RTL, and formats supporting geometry description languages such as GDSTL GDSlU, GDSiV, CiF, MEBES and any other suitable formats and languages.
• Computer-readable media in which such formatted data and/or instructions may be embodied include, but are not limited to, various forms of machine-readable storage media as discussed above (e.g., optica!, magnetic or semiconductor storage media).
• [00182j When received within a computer system via one or more computer-readable media, such data and/or instruction -based expressions of the above described circuits may be processed by a processing entity (e.g., one or more processors) within the computer system in conjunction with execution of one or more other computer programs including, without limitation, net-list generation programs, place and route programs and the like, to generate a representation or image of a physical manifestation of such circuits. Such representation or image may thereafter be used in device fabrication, for example, by enabling generation of one or more masks that are used to form various components of the circuits in a device fabrication process.
• a processing entity e.g., one or more processors
• Such representation or image may thereafter be used in device fabrication, for example, by enabling generation of one or more masks that are used to form various components of the circuits in a device fabrication process.
• signals described or depicted as having active-high or active-low logic levels may have opposite logic levels in alternative embodiments.
• Component circuitry within integrated circuit devices may be implemented using metal oxide semiconductor (MOS) technology, bipolar technology or any other technology in which logical and analog circuits may be implemented.
• MOS metal oxide semiconductor
• a signal is said to be "asserted'" when the signal is driven to a low or high logic state (or charged to a high logic state or discharged to a low logic state) to indicate a particular condition.
• a signal is said to be “deasserted” to indicate that the signal is driven (or charged or discharged) to a state other than the asserted state (including a high or low " logic state, or the floating state that may- occur when the signal driving circuit is transitioned to a high impedance condition, such as an open drain or open collector condition).
• a signal driving circuit is said to "output" a signal to a signal receiving circuit when the signal driving circuit asserts (or deasserts, if explicitly stated or indicated by context) the signal on a signal line coupled between the signal driving and signal receiving circuits.
• a signal line is said to be “activated” when a signal is asserted on the signal line, and “deactivated”' when the signal is deasseried.
• the prefix symbol "/" attached to signal names indicates that the signal is an active low signal (i.e., the asserted state is a logic low state).
• a line over a signal name e.g., ' ⁇ signal name > '
• the term “coupled” is used herein to express a direct connection as well as a connection through one or more intervening circuits or structures.
• Integrated circuit device "programming” may include, for example and without limitation, loading a control value into a register or other storage circuit within the device in response to a host instruction and thus controlling an operational aspect of the device, establishing a device configuration or controlling an operational aspect of the device through a one-time programming operation (e.g., blowing fuses within a configuration circuit during device production), and/or connecting one or more selected pins or other contact structures of the device to reference voltage lines (also referred to as snapping) to establish a particular device configuration or operation aspect of the device.
• exemplary " ' is used to express an example, not a preference or requirement.
• passive evacuation techniques presented above may be used to prepare a device (e.g., non-volatile memory) for later replacement or other maintenance
• techniques applicable to such evacuation may include a progressively escalated evacuation including a passive evacuation followed by an active evacuation, if desired, triggered in response to milestones or user-directed implementation.
• These techniques may be managed by hardware (such as an anneal controller), with some, most or all of these functions offloaded to software.
• Other features from the processes mentioned above may be further combined with these techniques, as appropriate.
• a system may comprise a memory controller, multiple annealable devices (e.g., integrated circuits having heating elements) and an anneal controller having control paths for the heating element circuitry for each of the annealable devices.
• the anneal controller may regulate a relatively high voltage supply used to drive each anneal process and serve the function of heater control circuitry.
• the high voltage supply can either be integrated with the anneal controller or implemented via a separate chip (IC), and the anneal controller may be combined with a memory controller, or may be implemented in a separate IC.
• any of the techniques mentioned above may be implemented in a device that monitors the need for a memory maintenance event, where the memory is one clement of a multiple element storage system, that permits a user to specify actions upon detection of the need for the maintenance event, and that permits a user to take actions at predetermined or programmable milestones (e.g., as the need for annealing or other maintenance becomes more acute).
• a system may include a genera! purpose computer system running appropriate software.
• a user may be provided with a set of one or more options for rerouting data from the memory, to thereby passively or actively evacuate the memory during run-time of a system; for example, a user may select a methodology for evacuation, including evacuation to other devices within the same tier (e.g., flash devices in main memory), secondary storage, or temporary storage in controller "scratch" memory, or cache.
• these functions may be provided as part of a computer operating system or as software distributed for installation on existing computer systems.
• the annealable devices may include multiple flash devices that make up part or all of main memory, while other tiers of memory may include processor cache and an optional secondary hard disk storage.
• Such an implementation facilitates the development of portable devices rooted in nonvolatile main memory (e.g., a handheld or laptop computer) with the use of anneal or memory replacement to extend portable device effective lifetime.
• these techniques may also be used with a software utility that displays to the user an indication or gauge (e.g., a pictograph) of remaining device lifetime, based on maximum permitted write cycles, a time period, measured criteria, or using other parameters as discussed above. Such a gauge may then be further employed with logic that selectively performs data evacuation or which otherwise prepares mcmo ⁇ y for replacement or maintenance.
• a software utility can also be distributed as part of an operating system or as separately distributed software, either with or without anneal capabilities.
• an evacuation step may be performed prior to an anneal; in other embodiments (e.g., where a backup of data already exists, such as in a RAID system), the evacuation step may be replaced with a post anneal (or post maintenance step), i.e., with no prior evacuation, with redundancy being reestablished through a refill step in lieu of evacuation.
• post anneal or post maintenance step

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Engineering & Computer Science (AREA)
• General Physics & Mathematics (AREA)
• Techniques For Improving Reliability Of Storages (AREA)
• Semiconductor Memories (AREA)
• Non-Volatile Memory (AREA)
• Hardware Redundancy (AREA)

Priority Applications (4)

Applications Claiming Priority (4)

Publications (1)

Family

ID=43607275

Family Applications (1)

Country Status (5)

Cited By (17)

Families Citing this family (10)

Citations (3)

Family Cites Families (8)

• 2010
  • 2010-06-29 CN CN2010800429958A patent/CN102576569A/zh active Pending
  • 2010-06-29 KR KR1020127007342A patent/KR20120059569A/ko not_active Ceased
  • 2010-06-29 EP EP10810328.4A patent/EP2467855A4/en not_active Withdrawn
  • 2010-06-29 WO PCT/US2010/040322 patent/WO2011022123A1/en active Application Filing
  • 2010-06-29 JP JP2012525570A patent/JP2013502647A/ja active Pending

Patent Citations (3)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events