WO2015034954A1 - Procédé et appareil pour réduire la capacité déclarée d'un dispositif de stockage par réduction d'une plage d'adresses logiques - Google Patents

Procédé et appareil pour réduire la capacité déclarée d'un dispositif de stockage par réduction d'une plage d'adresses logiques Download PDF

Info

Publication number
WO2015034954A1
WO2015034954A1 PCT/US2014/053941 US2014053941W WO2015034954A1 WO 2015034954 A1 WO2015034954 A1 WO 2015034954A1 US 2014053941 W US2014053941 W US 2014053941W WO 2015034954 A1 WO2015034954 A1 WO 2015034954A1
Authority
WO
WIPO (PCT)
Prior art keywords
storage device
storage
host
memory
volatile memory
Prior art date
Application number
PCT/US2014/053941
Other languages
English (en)
Inventor
Allen Samuels
Warren Fritz Kruger
Linh Tien Truong
Original Assignee
Sandisk Technologies Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US14/470,580 external-priority patent/US9519577B2/en
Priority claimed from US14/470,596 external-priority patent/US9442670B2/en
Application filed by Sandisk Technologies Inc. filed Critical Sandisk Technologies Inc.
Publication of WO2015034954A1 publication Critical patent/WO2015034954A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F12/00—Accessing, addressing or allocating within memory systems or architectures
    • G06F12/02—Addressing or allocation; Relocation
    • G06F12/0223—User address space allocation, e.g. contiguous or non contiguous base addressing
    • G06F12/023—Free address space management
    • G06F12/0238—Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory
    • G06F12/0246—Memory management in non-volatile memory, e.g. resistive RAM or ferroelectric memory in block erasable memory, e.g. flash memory
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
    • G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
    • G06F11/1008—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's in individual solid state devices
    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11C—STATIC STORES
    • G11C16/00—Erasable programmable read-only memories
    • G11C16/02—Erasable programmable read-only memories electrically programmable
    • G11C16/06—Auxiliary circuits, e.g. for writing into memory
    • G11C16/34—Determination of programming status, e.g. threshold voltage, overprogramming or underprogramming, retention
    • G11C16/349—Arrangements for evaluating degradation, retention or wearout, e.g. by counting erase cycles
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/10—Providing a specific technical effect
    • G06F2212/1016—Performance improvement
    • G06F2212/1024—Latency reduction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/10—Providing a specific technical effect
    • G06F2212/1032—Reliability improvement, data loss prevention, degraded operation etc
    • G06F2212/1036—Life time enhancement
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/10—Providing a specific technical effect
    • G06F2212/1041—Resource optimization
    • G06F2212/1044—Space efficiency improvement
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2212/00—Indexing scheme relating to accessing, addressing or allocation within memory systems or architectures
    • G06F2212/72—Details relating to flash memory management
    • G06F2212/7205—Cleaning, compaction, garbage collection, erase control
    • G—PHYSICS
    • G11—INFORMATION STORAGE
    • G11C—STATIC STORES
    • G11C2211/00—Indexing scheme relating to digital stores characterized by the use of particular electric or magnetic storage elements; Storage elements therefor
    • G11C2211/56—Indexing scheme relating to G11C11/56 and sub-groups for features not covered by these groups
    • G11C2211/564—Miscellaneous aspects
    • G11C2211/5641—Multilevel memory having cells with different number of storage levels

Definitions

  • the disclosed embodiments relate generally to memory systems, and in particular, to triggering a process to reduce declared capacity of a storage device (e.g., comprising one or more flash memory devices).
  • a storage device e.g., comprising one or more flash memory devices.
  • Flash memory typically utilize memory cells to store data as an electrical value, such as an electrical charge or voltage.
  • a flash memory cell for example, includes a single transistor with a floating gate that is used to store a charge representative of a data value.
  • Flash memory is a non- volatile data storage device that can be electrically erased and reprogrammed. More generally, non- volatile memory (e.g., flash memory, as well as other types of non-volatile memory implemented using any of a variety of technologies) retains stored information even when not powered, as opposed to volatile memory, which requires power to maintain the stored information.
  • Increases in storage density have been facilitated in various ways, including increasing the density of memory cells on a chip enabled by manufacturing developments, and transitioning from single-level flash memory cells to multi-level flash memory cells, so that two or more bits can be stored by each flash memory cell.
  • an amelioration trigger for reducing declared capacity of non-volatile memory of a storage device in a storage system is detected, and in accordance with the detected amelioration trigger, an amelioration process to reduce declared capacity of the non-volatile memory of the storage device is performed.
  • Figure 1 A is a block diagram illustrating an implementation of a data storage system, in accordance with some embodiments.
  • Figure IB is a block diagram illustrating an implementation of a data storage system, in accordance with some embodiments.
  • Figure 1C is a block diagram illustrating an implementation of a data storage system, in accordance with some embodiments.
  • Figure 2A-1 is a block diagram illustrating an implementation of a
  • Figure 2A-2 is a block diagram illustrating an implementation of a
  • Figure 2B-1 is a block diagram illustrating an implementation of a system management module, in accordance with some embodiments.
  • Figure 2B-2 is a block diagram illustrating an implementation of a system management module, in accordance with some embodiments.
  • Figure 2C-1 is a block diagram illustrating an implementation of a cluster management module, in accordance with some embodiments.
  • FIG. 2C-2 is a block diagram illustrating an implementation of a cluster management module, in accordance with some embodiments.
  • Figure 2D is a block diagram illustrating an implementation of an amelioration module included in Figures 2A-1 and 2A-2, in accordance with some embodiments.
  • FIG. 3 is a block diagram of a logical address space, and more specifically a logical block address (LBA) space, in accordance with some embodiments.
  • LBA logical block address
  • Figure 4 is a block diagram of a mapping table and physical address space, in accordance with some embodiments.
  • Figure 5 A is a prophetic diagram of voltage distributions that may be found in a single-level flash memory cell (SLC) over time, in accordance with some embodiments.
  • SLC single-level flash memory cell
  • Figure 5B is a prophetic diagram of voltage distributions that may be found in a multi-level flash memory cell (MLC) over time, in accordance with some embodiments.
  • MLC multi-level flash memory cell
  • Figure 6 illustrates a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • Figures 7A-7D illustrate a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • Figures 8A-8D illustrate a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • FIGS 9A-9D illustrate a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • Figures 1 OA- IOC illustrate a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • FIGS 1 lA-11C illustrate a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • Figures 12A-12C illustrate a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • Figures 13A-13D illustrate a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • Figures 14A-14C illustrate a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • Figures 15A-15D illustrate a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • Figures 16A-16C illustrate a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • Figures 17A-17C illustrate a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • Figures 18A-18B illustrate a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • Figures 19A-19B illustrate a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • Figures 20A-20D illustrate a flowchart representation of a method of managing a storage system, in accordance with some embodiments.
  • Figure 21 is a block diagram illustrating an implementation of a data storage system in accordance with some embodiments.
  • Figure 22A is a block diagram illustrating an implementation of a
  • Figure 22B is a diagram illustrating a characterization vector table included in
  • FIG. 22A in accordance with some embodiments.
  • Figure 22C is a diagram illustrating a characterization vector included in
  • Figures 23A-23C are block diagrams of a storage medium divided into a plurality of logical chunks in accordance with some embodiments.
  • Figure 24A is a block diagram of a storage medium divided into a plurality of logical chunks in accordance with some embodiments.
  • Figure 24B is a diagram of an indirection table corresponding to the plurality of logical chunks in Figure 24 A in accordance with some embodiments.
  • Figure 24C is a block diagram of a storage medium divided into a plurality of logical chunks in accordance with some embodiments.
  • Figure 24D is a diagram of an indirection table corresponding to the plurality of logical chunks in Figure 24C in accordance with some embodiments.
  • Figures 25-27 illustrate flowchart representations of methods of triggering a migration process in accordance with some embodiments.
  • Figures 28A-28C illustrate a flowchart representation of a method of managing a storage system in accordance with some embodiments.
  • a multi-level flash cell When a multi-level flash cell has reached its wear limit it typically still has charge retention capability sufficient to store a reduced number of charge levels. Often it is the case that a substantial number of erasure and reprogramming cycles can be performed on a wear-limited multi-level flash cell with full recovery of the stored data, provided that a reduced number of charge levels is used and expected.
  • a flash memory device operating in 3 bits per cell mode typically can perform between 500 and 1500 erasure and reprogramming cycles before being considered worn out. However, at that point in time it will typically still have sufficient charge storage capability to operate in the single bit per cell mode (SLC) for an additional 10,000 to 20,000 erasure and reprogramming cycles before the SLC wear limit is encountered.
  • the various embodiments described herein include systems, methods and/or devices used to enable triggering a process to reduce declared capacity of a storage device. Some embodiments include systems, methods and/or devices to detect a trigger condition in accordance with one or more metrics of a storage device and enable an amelioration process associated with the detected trigger condition, the amelioration process to reduce declared capacity of non- volatile memory of the storage device.
  • some embodiments include a method of managing a storage system.
  • the method includes, at a storage device of the storage system: (1) generating one or more metrics of the storage device, the storage device including non- volatile memory, (2) detecting a trigger condition in accordance with the one or more metrics of the storage device, and (3) enabling an amelioration process associated with the detected trigger condition, the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • enabling the amelioration process associated with the detected trigger condition includes notifying a host to which the storage device is operatively coupled of the trigger condition.
  • enabling the amelioration process associated with the detected trigger condition includes: (1) receiving a query from a host to which the storage device is operatively coupled, and (2) in response to receiving the query, reporting the trigger condition.
  • enabling the amelioration process associated with the detected trigger condition includes: (1) receiving a command from a host to which the storage device is operatively coupled, and (2) in response to receiving the command, sending a response to the command and a notification of the trigger condition.
  • enabling the amelioration process associated with the detected trigger condition includes: (1) receiving a command from a host to which the storage device is operatively coupled, and (2) in response to receiving the command, sending a response to the command and a notification that prompts the host to obtain information with respect to the trigger condition.
  • the host includes a client on behalf of which data is stored in the storage system.
  • the host includes a storage system controller of the storage system.
  • the host includes a cluster controller of the storage system.
  • enabling the amelioration process associated with the detected trigger condition includes scheduling the amelioration process to be performed on the storage device.
  • enabling the amelioration process associated with the detected trigger condition includes determining one or more parameters for the amelioration process.
  • enabling the amelioration process associated with the detected trigger condition further includes reporting at least a subset of the one or more parameters for the amelioration process.
  • generating one or more metrics of the storage device includes generating at least one metric, of the one or more metrics, for each memory portion of a plurality of memory portions of the storage device.
  • the one or more metrics of the storage device include one or more status metrics corresponding to the storage device's ability to retain data.
  • the one or more metrics of the storage device include one or more performance metrics corresponding to performance of the storage device.
  • the one or more metrics of the storage device include one or more wear metrics corresponding to wear on the storage device.
  • the one or more metrics of the storage device include one or more time metrics.
  • the one or more metrics of the storage device include values of the one or more metrics from more than one time.
  • the method further includes, after enabling the amelioration process, (1) re-evaluating the trigger condition in accordance with the one or more metrics of the storage device, and (2) in accordance with a determination that the trigger condition is no longer valid, aborting the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • the amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes a process to reduce utilization of the non-volatile memory of the storage device.
  • the storage device comprises one or more flash memory devices.
  • a storage device includes (1) non- volatile memory
  • controller memory e.g., non-volatile memory or volatile memory in or coupled to the controller
  • programs which when executed by the one or more processors cause the storage device to perform or control performance of any of the methods Al to A20 described herein.
  • any of the methods Al to A20 described above are performed by a storage device including means for performing any of the methods described herein.
  • a storage system includes (1) a storage medium
  • memory e.g., non-volatile memory or volatile memory in the storage system
  • programs which when executed by the one or more processors cause the storage system to perform or control performance of any of the methods Al to A20 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage device, the one or more programs including instructions for performing any of the methods described herein.
  • the various embodiments described herein include systems, methods and/or devices used to enable triggering, at a host, a process to reduce declared capacity of a storage device.
  • Some embodiments include systems, methods and/or devices to detect a trigger condition in accordance with one or more metrics of a storage device and enable an amelioration process associated with the detected trigger condition, the amelioration process to reduce declared capacity of non- volatile memory of the storage device.
  • some embodiments include a method of managing a storage system.
  • the method includes, at a host to which a storage device of the storage system is operatively coupled: (1) obtaining one or more metrics of the storage device, the storage device including non-volatile memory, (2) detecting a trigger condition in accordance with the one or more metrics of the storage device, and (3) enabling an amelioration process associated with the detected trigger condition, the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • the host includes a client on behalf of which data is stored in the storage system.
  • the host includes a storage system controller of the storage system.
  • the host includes a cluster controller of the storage system.
  • enabling the amelioration process associated with the detected trigger condition includes scheduling the amelioration process to be performed on the storage device.
  • enabling the amelioration process associated with the detected trigger condition includes determining one or more parameters for the amelioration process.
  • enabling the amelioration process associated with the detected trigger condition further includes conveying at least a subset of the one or more parameters for the amelioration process to the storage device.
  • obtaining one or more metrics of the storage device includes obtaining at least one metric, of the one or more metrics, for each memory portion of a plurality of memory portions of the storage device.
  • the one or more metrics of the storage device include one or more status metrics corresponding to the storage device's ability to retain data.
  • the one or more metrics of the storage device include one or more performance metrics corresponding to performance of the storage device.
  • the one or more metrics of the storage device include one or more wear metrics corresponding to wear on the storage device.
  • the one or more metrics of the storage device include one or more time metrics.
  • the one or more metrics of the storage device include values of the one or more metrics from more than one time.
  • the method further includes, after enabling the amelioration process, (1) re-evaluating the trigger condition in accordance with the one or more metrics of the storage device, and (2) in accordance with a determination that the trigger condition is no longer valid, aborting the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • the amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes a process to reduce utilization of the non-volatile memory of the storage device.
  • the storage device comprises one or more flash memory devices.
  • a storage system includes (1) one or more storage devices (e.g., comprising one or more non-volatile storage devices, such as flash memory devices), (2) a host to which the one or more storage devices are operatively coupled, (3) one or more processors, and (4) controller memory storing one or more programs, which when executed by the one or more processors cause the host to perform or control performance of any of the methods Bl to B16 described herein.
  • one or more storage devices e.g., comprising one or more non-volatile storage devices, such as flash memory devices
  • any of the methods Bl to B16 described above are performed by a host system, coupled to one or more storage devices, the host system including means for performing any of the methods described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage system, the one or more programs including instructions for performing any of the methods described herein.
  • a host system includes (1) an interface for operatively coupling to a storage system, (2) one or more processors, and (3) controller memory storing one or more programs, which when executed by the one or more processors cause the host system to perform or control performance of any of the methods Bl to B16 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a host system, the one or more programs including instructions for performing any of the methods described herein.
  • the various embodiments described herein include systems, methods and/or devices used to enable triggering a process to reduce declared capacity of a storage device in a multi-storage-device storage system. Some embodiments include systems, methods and/or devices to detect a trigger condition in accordance with one or more metrics of a respective storage device of a plurality of storage devices of a storage system and enable an
  • amelioration process associated with the detected trigger condition the amelioration process to reduce declared capacity of non- volatile memory of the respective storage device.
  • some embodiments include a method of managing a storage system.
  • the method includes: (1) obtaining, for each storage device of a plurality of storage devices of the storage system, one or more metrics of the storage device, the storage device including non-volatile memory, (2) detecting a trigger condition in accordance with the one or more metrics of a respective storage device of the plurality of storage devices of the storage system, and (3) enabling an amelioration process associated with the detected trigger condition, the amelioration process to reduce declared capacity of the non-volatile memory of the respective storage device.
  • enabling the amelioration process associated with the detected trigger condition includes notifying a host to which the respective storage device is operatively coupled of the trigger condition.
  • enabling the amelioration process associated with the detected trigger condition includes: (1) receiving a query from a host to which the respective storage device is operatively coupled, and (2) in response to receiving the query, reporting the trigger condition.
  • enabling the amelioration process associated with the detected trigger condition includes: (1) receiving a command from a host to which the respective storage device is operatively coupled, and (2) in response to receiving the command, sending a response to the command and a notification of the trigger condition.
  • enabling the amelioration process associated with the detected trigger condition includes: (1) receiving a command from a host to which the respective storage device is operatively coupled, and (2) in response to receiving the command, sending a response to the command and a notification that prompts the host to obtain information with respect to the trigger condition.
  • the host includes a client on behalf of which data is stored in the storage system.
  • the host includes a storage system controller of the storage system.
  • the host includes a cluster controller of the storage system.
  • enabling the amelioration process associated with the detected trigger condition includes scheduling the amelioration process to be performed on the respective storage device.
  • enabling the amelioration process associated with the detected trigger condition includes determining one or more parameters for the amelioration process.
  • enabling the amelioration process associated with the detected trigger condition further includes reporting at least a subset of the one or more parameters for the amelioration process.
  • obtaining one or more metrics of the respective storage device includes obtaining at least one metric, of the one or more metrics, for each memory portion of a plurality of memory portions of the respective storage device.
  • the one or more metrics of the respective storage device include one or more status metrics
  • the one or more metrics of the respective storage device include one or more performance metrics corresponding to performance of the respective storage device.
  • the one or more metrics of the respective storage device include one or more wear metrics
  • the one or more metrics of the respective storage device include one or more time metrics.
  • the one or more metrics of the respective storage device include values of the one or more metrics from more than one time.
  • the method further includes, after enabling the amelioration process: (1) re-evaluating the trigger condition in accordance with the one or more metrics of the respective storage device, and (2) in accordance with a determination that the trigger condition is no longer valid, aborting the amelioration process to reduce declared capacity of the non-volatile memory of the respective storage device.
  • the respective storage device comprises one or more flash memory devices.
  • a storage system includes (1) non- volatile memory,
  • controller memory storing one or more programs, which when executed by the one or more processors cause the storage system to perform or control performance of any of the methods CI to C20 described herein.
  • any of the methods CI to C20 described above are performed by a storage system including means for performing any of the methods described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage system, the one or more programs including instructions for performing any of the methods described herein.
  • a storage system includes (1) a plurality of storage devices, (2) one or more subsystems having one or more processors, and (3) memory storing one or more programs, which when executed by the one or more processors cause the one or more subsystems to perform or control performance of any of the methods CI to C20 described herein.
  • a host system includes (1) an interface for operatively coupling to a storage system, (2) one or more processors, and (3) controller memory storing one or more programs, which when executed by the one or more processors cause the host system to perform or control performance of any of the methods CI to C20 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a host system, the one or more programs including instructions for performing any of the methods described herein.
  • the various embodiments described herein include systems, methods and/or devices used to enable notification of a trigger condition to reduce declared capacity of a storage device. Some embodiments include systems, methods and/or devices to notify a host to which a storage device is operatively coupled of a trigger condition for reducing declared capacity of non- volatile memory of the storage device. [00123] (Dl) More specifically, some embodiments include a method of managing a storage system.
  • the method includes, at a storage device of the storage system, the storage device including non- volatile memory: (1) detecting a trigger condition for reducing declared capacity of the non- volatile memory of the storage device, and (2) notifying a host to which the storage device is operatively coupled of the trigger condition for reducing declared capacity of the non- volatile memory of the storage device, the trigger condition for enabling performance of an amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • the amelioration process is performed, at least in part, by an apparatus other than the storage device (e.g., performed at least in part by the host, or by a storage system controller or by a cluster controller of a data storage system that includes at least one storage device distinct from the storage device).
  • the host includes a client on behalf of which data is stored in the storage system.
  • the host includes a storage system controller of the storage system.
  • the host includes a cluster controller of the storage system.
  • notifying the host of the trigger condition for reducing declared capacity of the non- volatile memory of the storage device includes notifying the host with an unsolicited communication.
  • notifying the host of the trigger condition for reducing declared capacity of the non- volatile memory of the storage device includes: (1) receiving a query from the host, and (2) in response to receiving the query, reporting the trigger condition.
  • notifying the host of the trigger condition for reducing declared capacity of the non- volatile memory of the storage device includes: (1) receiving a command from the host, and (2) in response to receiving the command, sending a response to the command and a notification of the trigger condition.
  • notifying the host of the trigger condition for reducing declared capacity of the non- volatile memory of the storage device includes: (1) receiving a command from the host, and (2) in response to receiving the command, sending a response to the command and a notification that prompts the host to obtain information with respect to the trigger condition.
  • notifying the host of the trigger condition for reducing declared capacity of the non- volatile memory of the storage device further includes notifying the host that the storage device is in read-only mode.
  • the trigger condition is detected by the storage device in accordance with one or more metrics of the storage device.
  • the method further includes, after notifying the host of the trigger condition for reducing declared capacity of the non- volatile memory of the storage device: (1) re-evaluating the trigger condition in accordance with the one or more metrics of the storage device, and (2) in accordance with a determination that the trigger condition is no longer valid, notifying the host of an absence of the trigger condition for reducing declared capacity of the non-volatile memory of the storage device.
  • the amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes a process to reduce utilization of the non-volatile memory of the storage device.
  • the storage device comprises one or more flash memory devices.
  • a storage device includes (1) non-volatile memory
  • controller memory e.g., non-volatile memory or volatile memory in or coupled to the controller
  • programs which when executed by the one or more processors cause the storage device to perform or control performance of any of the methods Dl to D13 described herein.
  • a storage system includes (1) a storage medium
  • non- volatile storage devices such as flash memory devices
  • processors e.g., one or more processors
  • memory e.g., non-volatile memory or volatile memory in the storage system
  • programs which when executed by the one or more processors cause the storage system to perform or control performance of any of the methods Dl to D13 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage device, the one or more programs including instructions for performing any of the methods described herein.
  • the various embodiments described herein include systems, methods and/or devices used to enable notification of a trigger condition to reduce declared capacity of a storage device in a multi-storage-device storage system. Some embodiments include systems, methods and/or devices to notify a host to which a respective storage device of a plurality of storage devices of a storage system is operatively coupled of a trigger condition for reducing declared capacity of non-volatile memory of the respective storage device.
  • some embodiments include a method of managing a storage system.
  • the method includes: (1) obtaining, for each storage device of a plurality of storage devices of the storage system, one or more metrics of the storage device, the storage device including non-volatile memory, (2) detecting a trigger condition for reducing declared capacity of the non-volatile memory of a respective storage device of the plurality of storage devices of the storage system, the trigger condition detected in accordance with the one or more metrics of two or more of the storage devices of the plurality of storage devices in the storage system, and (3) notifying a host to which the respective storage device is operatively coupled of the trigger condition for reducing declared capacity of the non-volatile memory of the respective storage device, the trigger condition for enabling performance of an amelioration process to reduce declared capacity of the nonvolatile memory of the respective storage device.
  • the notification of the trigger condition causes performance of the
  • the host includes a client on behalf of which data is stored in the storage system.
  • the host includes a storage system controller of the storage system.
  • the host includes a cluster controller of the storage system.
  • notifying the host of the trigger condition for reducing declared capacity of the non- volatile memory of the respective storage device includes notifying the host with an unsolicited communication.
  • notifying the host of the trigger condition for reducing declared capacity of the non- volatile memory of the respective storage device includes: (1) receiving a query from the host, and (2) in response to receiving the query, reporting the trigger condition.
  • notifying the host of the trigger condition for reducing declared capacity of the non- volatile memory of the respective storage device includes: (1) receiving a command from the host, and (2) in response to receiving the command, sending a response to the command and a notification of the trigger condition.
  • notifying the host of the trigger condition for reducing declared capacity of the non- volatile memory of the respective storage device includes: (1) receiving a command from the host, and (2) in response to receiving the command, sending a response to the command and a notification that prompts the host to obtain information with respect to the trigger condition.
  • notifying the host of the trigger condition for reducing declared capacity of the non- volatile memory of the respective storage device includes notifying the host that the respective storage device is in read-only mode.
  • the method further includes, after notifying the host of the trigger condition for reducing declared capacity of the non- volatile memory of the respective storage device: (1) re-evaluating the trigger condition in accordance with the one or more metrics of the two or more storage devices of the plurality of storage devices in the storage system, and (2) in accordance with a determination that the trigger condition is no longer valid, notifying the host of an absence of the trigger condition for reducing declared capacity of the non-volatile memory of the respective storage device.
  • the obtaining, the notifying, or both the obtaining and the notifying are performed by one or more subsystems of the storage system distinct from the plurality of storage devices.
  • the amelioration process to reduce declared capacity of the non-volatile memory of the respective storage device includes a process to reduce utilization of the non-volatile memory of the respective storage device.
  • the respective storage device comprises one or more flash memory devices.
  • a storage system includes (1) non- volatile memory
  • controller memory e.g., non-volatile memory or volatile memory in or coupled to a controller of the storage system
  • any of the methods El to E13 described above are performed by a storage system including means for performing any of the methods described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage system, the one or more programs including instructions for performing any of the methods described herein.
  • a storage system includes (1) a plurality of storage devices, (2) one or more subsystems having one or more processors, and (3) memory (e.g., non- volatile memory or volatile memory in the storage system) storing one or more programs, which when executed by the one or more processors cause the one or more subsystems to perform or control performance of any of the methods El to El 3 described herein.
  • a host system includes (1) an interface for operatively coupling to a storage system, (2) one or more processors, and (3) controller memory storing one or more programs, which when executed by the one or more processors cause the host system to perform or control performance of any of the methods El to El 3 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a host system, the one or more programs including instructions for performing any of the methods described herein.
  • the various embodiments described herein include systems, methods and/or devices used to reduce declared capacity of a storage device in accordance with a detected amelioration trigger. Some embodiments include systems, methods and/or devices to detect an amelioration trigger for reducing declared capacity of non- volatile memory of a storage device of the storage system, and perform, in accordance with the detected amelioration trigger, an amelioration process to reduce declared capacity of non- volatile memory of the storage device.
  • some embodiments include a method of managing a storage system.
  • the method includes: (1) detecting an amelioration trigger for reducing declared capacity of non-volatile memory of a storage device of the storage system, and (2) in accordance with the detected amelioration trigger, performing an amelioration process to reduce declared capacity of the non-volatile memory of the storage device, including reducing a range of logical addresses of a logical address space available to a host.
  • the host includes a client on behalf of which data is stored in the storage system.
  • the host includes a storage system controller of the storage system.
  • the host includes a cluster controller of the storage system.
  • the detecting, the performing, or both the detecting and the performing are performed by the storage device [00166]
  • the detecting, the performing, or both the detecting and the performing are performed by one or more subsystems of the storage system distinct from the storage device.
  • reducing the range of logical addresses of the logical address space available to the host includes reducing the range of logical addresses of the logical address space available to the host in accordance with one or more parameters for the amelioration process.
  • reducing the range of logical addresses of the logical address space available to the host includes removing a contiguous portion of the range of logical addresses of the logical address space available to the host.
  • reducing the range of logical addresses of the logical address space available to the host includes altering one or more logical address entries of a mapping table, the mapping table used to translate logical addresses in the logical address space to physical addresses in a physical address space of the storage device.
  • altering one or more logical address entries of the mapping table includes moving the one or more logical address entries without moving data stored at the one or more physical addresses associated with the one or more logical address entries.
  • the method includes, prior to altering the one or more logical address entries of the mapping table, selecting the one or more logical address entries to be altered so as to minimize performance degradation.
  • the method includes, prior to altering the one or more logical address entries of the mapping table, selecting the one or more logical address entries to be altered so as to minimize overhead from garbage collection.
  • the method includes, prior to altering the one or more logical address entries of the mapping table, selecting the one or more logical address entries to be altered so as to minimize a number of logical address entries to move.
  • performing the amelioration process to reduce declared capacity of the non-volatile memory of the storage device further includes advertising a reduced declared capacity of the non-volatile memory of the storage device.
  • the method includes, after beginning performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device, detecting an indication to abort the reduction in declared capacity of the non-volatile memory of the storage device; and in response to detecting the indication to abort the reduction in declared capacity of the nonvolatile memory of the storage device, aborting performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • performing the amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes reducing utilization of the non- volatile memory of the storage device.
  • reducing the range of logical addresses of the logical address space available to the host includes copying data stored at a first set of physical addresses associated with a first set of logical address entries in a mapping table to a second set of physical addresses associated with a second set of logical address entries in the mapping table, and updating the first set of logical address entries in the mapping table, the mapping table used to translate logical addresses in the logical address space to physical addresses in a physical address space of the storage device.
  • updating the first set of logical address entries in the mapping table includes invalidating one or more logical address entries of the first set of logical address entries.
  • the storage device comprises one or more flash memory devices.
  • a storage device includes (1) non- volatile memory, (2) one or more processors, and (3) controller memory (e.g., a non-transitory computer readable storage medium in the storage device) storing one or more programs, which when executed by the one or more processors cause the storage device to perform or control performance of any of the methods F1-F5 and F8-F20 described herein.
  • controller memory e.g., a non-transitory computer readable storage medium in the storage device
  • any of the methods F1-F5 and F8-F20 described above are performed by a storage device including means for performing any of the methods described herein.
  • a storage system includes (1) a storage medium
  • memory e.g., a non-transitory computer readable storage medium in the storage system
  • programs which when executed by the one or more processors cause the storage system to perform or control performance of any of the methods F1-F20 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage device, the one or more programs including instructions for performing any of the methods described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage system, the one or more programs including instructions for performing any of the methods described herein.
  • a storage system includes (1) one or more storage devices, (2) one or more subsystems having one or more processors, and (3) memory (e.g., a non-transitory computer readable storage medium in the one or more of the subsystems) storing one or more programs, which when executed by the one or more processors cause the one or more subsystems to perform or control performance of any of the methods F1-F4, F6, and F8-F18 described herein.
  • memory e.g., a non-transitory computer readable storage medium in the one or more of the subsystems
  • a host system includes (1) an interface for operatively coupling to a storage system, (2) one or more processors, and (3) controller memory (e.g., a non-transitory computer readable storage medium in the host system) storing one or more programs, which when executed by the one or more processors cause the host system to perform or control performance of any of the methods F1-F4 and F6-F18 described herein.
  • controller memory e.g., a non-transitory computer readable storage medium in the host system
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a host system, the one or more programs including instructions for performing any of the methods described herein.
  • the various embodiments described herein include systems, methods and/or devices used to reduce declared capacity of a storage device in accordance with a detected amelioration trigger. Some embodiments include systems, methods and/or devices to detect an amelioration trigger for reducing declared capacity of non- volatile memory of a storage device in a storage system and perform, in accordance with the detected amelioration trigger, an amelioration process to reduce declared capacity of non-volatile memory of the storage device.
  • some embodiments include a method of managing a storage system, where the method includes: (1) detecting an amelioration trigger for reducing declared capacity of non-volatile memory of a storage device of the storage system; and, (2) in accordance with the detected amelioration trigger, performing an amelioration process to reduce declared capacity of the non-volatile memory of the storage device, including making specific logical addresses of a logical address space unavailable to a host.
  • the host includes a client on behalf of which data is stored in the storage system.
  • the host includes a storage system controller of the storage system.
  • the host includes a cluster controller of the storage system.
  • the detecting, the performing, or both the detecting and the performing are performed by one or more subsystems of the storage system distinct from the storage device.
  • making specific logical addresses of the logical address space unavailable to the host includes making specific logical addresses of the logical address space unavailable to the host in accordance with one or more parameters for the amelioration process.
  • making specific logical addresses of the logical address space unavailable to the host includes enumerating one or more portions of the logical address space that are unavailable to the host.
  • the one or more enumerated portions of the logical address space that are unavailable to the host are determined in accordance with an algorithmic definition of which logical addresses of the logical address space are unavailable.
  • the one or more enumerated portions of the logical address space that are unavailable to the host are determined in accordance with a determination of which logical addresses of the logical address space are unused.
  • making specific logical addresses of the logical address space unavailable to the host includes: (i) specifying a first list of logical addresses of the logical address space that are in use; and (ii) specifying a second list of logical addresses of the logical address space that are available for use, where logical addresses of the logical address space not specified on the first list or on the second list are logical addresses of the logical address space unavailable to the host.
  • the first list and/or the second list is maintained at the host.
  • the first list and/or the second list is maintained at the storage device.
  • the first list and/or the second list is maintained external to the storage device.
  • the host selects the specific logical addresses of the logical address space to make unavailable to the host.
  • the specific logical addresses of the logical address space unavailable to the host are selected to minimize performance degradation.
  • the specific logical addresses of the logical address space unavailable to the host are selected to minimize overhead from garbage collection.
  • performing the amelioration process to reduce declared capacity of the non-volatile memory of the storage device further includes advertising a reduced declared capacity of the non- volatile memory of the storage device.
  • the method further includes: after beginning performance of the amelioration process to reduce declared capacity of the non- volatile memory of the storage device, detecting an indication to abort the reduction in declared capacity of the non-volatile memory of the storage device, and, in response to detecting the indication to abort the reduction in declared capacity of the nonvolatile memory of the storage device, aborting performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • performing the amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes reducing utilization of the non-volatile memory of the storage device.
  • making specific logical addresses of the logical address space unavailable to the host includes copying data stored at a first set of physical addresses associated with a first set of logical address entries in a mapping table to a second set of physical addresses associated with a second set of logical address entries in the mapping table, and updating the first set of logical address entries in the mapping table, the mapping table used to translate logical addresses in the logical address space to physical addresses in a physical address space of the storage device.
  • updating the first set of logical address entries in the mapping table includes invalidating one or more logical address entries of the first set of logical address entries.
  • the storage device comprises one or more flash memory devices.
  • a storage device includes (1) non-volatile memory,
  • controller memory e.g., a non-transitory computer readable storage medium in the storage device
  • controller memory e.g., a non-transitory computer readable storage medium in the storage device
  • a storage device includes means for performing or causing performance of the method of any of G1-G5 and G8-G27 described herein.
  • a storage system includes (1) a storage medium
  • memory e.g., a non-transitory computer readable storage medium in the storage system
  • programs which when executed by the one or more processors cause the storage system to perform or control performance of the method of any of G1-G27 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage device, the one or more programs including instructions for performing or causing performance of the method of any of G1-G5 and G8- G27 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage system, the one or more programs including instructions for performing or causing performance of the method of any of G1-G27 described herein.
  • a storage system includes (1) one or more storage devices, (2) one or more subsystems having one or more processors, and (3) memory (e.g., a non-transitory computer readable storage medium in the one or more of the subsystems) storing one or more programs, which when executed by the one or more processors cause the one or more subsystems to perform or control performance of the method of any of G1-G27 described herein.
  • memory e.g., a non-transitory computer readable storage medium in the one or more of the subsystems
  • a host system includes (1) an interface for operatively coupling to a storage system, (2) one or more processors, and (3) controller memory (e.g., a non-transitory computer readable storage medium in the host system) storing one or more programs, which when executed by the one or more processors cause the host system to perform or control performance of the method of any of G1-G4 and G7-G27 described herein.
  • controller memory e.g., a non-transitory computer readable storage medium in the host system
  • a storage system includes means for performing or causing performance of the method of any of G1-G27 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a host system, the one or more programs including instructions for performing or causing performance of the method of any of G1-G4 and G6-G27 described herein.
  • the various embodiments described herein include systems, methods and/or devices used to reduce declared capacity of a storage device in accordance with a detected amelioration trigger. Some embodiments include systems, methods and/or devices to detect a an amelioration trigger for reducing declared capacity of non- volatile memory of a storage device in a storage system and perform, in accordance with the detected amelioration trigger, an amelioration process to reduce declared capacity of non-volatile memory of the storage device.
  • some embodiments include a method of managing a storage system.
  • the method includes: (1) detecting an amelioration trigger for reducing declared capacity of non-volatile memory of a storage device of the storage system; and, (2) in accordance with the detected amelioration trigger, performing an amelioration process to reduce declared capacity of the non-volatile memory of the storage device, including reducing a count of logical addresses of a logical address space available to a host.
  • the host includes a client on behalf of which data is stored in the storage system.
  • the host includes a storage system controller of the storage system.
  • the host includes a cluster controller of the storage system.
  • the detecting, the performing, or both the detecting and the performing are performed by one or more subsystems of the storage system distinct from the storage device.
  • the detecting, the performing, or both the detecting and the performing are performed by the host.
  • reducing the count of logical addresses of the logical address space available to the host includes reducing the count of logical addresses of the logical address space available to the host in accordance with one or more parameters for the amelioration process.
  • reducing the count of logical addresses of the logical address space available to the host includes maintaining a count of logical addresses of the logical address space that are currently in use.
  • the host enforces that a count of logical addresses in use by the host of the logical address space does not exceed the count of logical addresses of the logical address space available to the host.
  • the storage device enforces that a count of logical addresses in use by the host of the logical address space does not exceed the count of logical addresses of the logical address space available to the host.
  • the method further includes returning an error to the host, in accordance with a determination that a write command from the host would cause the count of logical addresses in use by the host to exceed the count of logical addresses available to the host.
  • performing the amelioration process to reduce declared capacity of the non-volatile memory of the storage device further includes advertising a reduced declared capacity of the non- volatile memory of the storage device.
  • the method further includes: after beginning performance of the amelioration process to reduce declared capacity of the non- volatile memory of the storage device, detecting an indication to abort the reduction in declared capacity of the non-volatile memory of the storage device, and, in response to detecting the indication to abort the reduction in declared capacity of the nonvolatile memory of the storage device, aborting performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • performing the amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes reducing utilization of the non-volatile memory of the storage device.
  • the storage device comprises one or more flash memory devices.
  • a storage device includes (1) non- volatile memory,
  • controller memory e.g., a non-transitory computer readable storage medium in the storage device
  • controller memory e.g., a non-transitory computer readable storage medium in the storage device
  • a storage device includes means for performing or causing performance of the method of any of H1-H5 and H8-H16 described herein.
  • a storage system includes (1) a storage medium
  • HI -HI 6 e.g., comprising one or more non- volatile storage devices, such as flash memory devices
  • processors e.g., one or more processors
  • memory e.g., a non-transitory computer readable storage medium in the storage system
  • programs which when executed by the one or more processors cause the storage system to perform or control performance of the method of any of HI -HI 6 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage device, the one or more programs including instructions for performing or causing performance of the method of any of H1-H5 and H8- H16 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage system, the one or more programs including instructions for performing or causing performance of the method of any of HI -HI 6 described herein.
  • a storage system includes (1) one or more storage devices, (2) one or more subsystems having one or more processors, and (3) memory (e.g., a non-transitory computer readable storage medium in the one or more of the subsystems) storing one or more programs, which when executed by the one or more processors cause the one or more subsystems to perform or control performance of the method of any of H1-H4 and H6-H16 described herein.
  • memory e.g., a non-transitory computer readable storage medium in the one or more of the subsystems
  • a host system includes (1) an interface for operatively coupling to a storage system, (2) one or more processors, and (3) controller memory (e.g., a non-transitory computer readable storage medium in the host system) storing one or more programs, which when executed by the one or more processors cause the host system to perform or control performance of the method of any of H1-H4 and H7-H16 described herein.
  • controller memory e.g., a non-transitory computer readable storage medium in the host system
  • a storage system including means for performing or causing performance of the method of any of HI -HI 6 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a host system, the one or more programs including instructions for performing or causing performance of the method of any of H1-H4 and H6-H16 described herein.
  • the various embodiments described herein include systems, methods and/or devices used to reduce declared capacity of a storage device in accordance with a detected amelioration trigger. Some embodiments include systems, methods and/or devices to detect a trigger condition in accordance with one or more metrics of a storage device and enable an amelioration process associated with the detected trigger condition, the amelioration process to reduce declared capacity of non- volatile memory of the storage device.
  • some embodiments include a method of managing a storage system.
  • the method includes: (1) detecting an amelioration trigger for reducing declared capacity of non-volatile memory of a storage device of the storage system, and (2) in accordance with the detected amelioration trigger, performing an amelioration process to reduce declared capacity of the non-volatile memory of the storage device, including altering an encoding format of at least a portion of the non-volatile memory of the storage device, and reducing declared capacity of the non-volatile memory of the storage device.
  • the detecting, the performing, or both the detecting and the performing are performed by the storage device
  • the detecting, the performing, or both the detecting and the performing are performed by one or more subsystems of the storage system distinct from the storage device.
  • the detecting, the performing, or both the detecting and the performing are performed by the host.
  • the host includes a client on behalf of which data is stored in the storage system.
  • the host includes a storage system controller of the storage system.
  • the host includes a cluster controller of the storage system.
  • altering the encoding format of at least a portion of the non-volatile memory of the storage device includes altering the encoding format of at least a portion of the non-volatile memory of the storage device in accordance with one or more parameters for the amelioration process.
  • altering the encoding format of at least a portion of the non-volatile memory of the storage device includes altering the encoding format from a higher-density physical encoding format to a lower-density physical encoding format.
  • altering the encoding format of at least a portion of the non-volatile memory of the storage device includes altering a number of states per memory cell from a higher number of states to a lower number of states.
  • altering the encoding format of at least a portion of the non-volatile memory of the storage device includes altering the encoding format from a Triple-Level Cell (TLC) format to a Multi-Level Cell (MLC) format.
  • TLC Triple-Level Cell
  • MLC Multi-Level Cell
  • altering the encoding format of at least a portion of the non-volatile memory of the storage device includes altering the encoding format from a Triple-Level Cell (TLC) format to a Single- Level Cell (SLC) format.
  • TLC Triple-Level Cell
  • SLC Single- Level Cell
  • altering the encoding format of at least a portion of the non-volatile memory of the storage device includes altering the encoding format from a Multi-Level Cell (MLC) format to a Single- Level Cell (SLC) format.
  • MLC Multi-Level Cell
  • SLC Single- Level Cell
  • performing the amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes reducing utilization of the non- volatile memory of the storage device.
  • altering the encoding format of at least a portion of the non-volatile memory of the storage device includes altering the encoding format of a memory portion of a plurality of memory portions of the non-volatile memory of the storage device.
  • altering the encoding format of at least a portion of the non-volatile memory of the storage device includes altering the encoding format of all client data of the non-volatile memory of the storage device.
  • performing the amelioration process to reduce declared capacity of the non-volatile memory of the storage device further includes advertising a reduced declared capacity of the non-volatile memory of the storage device.
  • the method further comprises (1) after beginning performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device, detecting an indication to abort the reduction in declared capacity of the non-volatile memory of the storage device, and (2) in response to detecting the indication to abort the reduction in declared capacity of the non-volatile memory of the storage device, aborting performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • the storage device comprises one or more flash memory devices.
  • a storage device includes (1) non- volatile memory, (2) one or more processors, and (3) controller memory storing one or more programs, which when executed by the one or more processors cause the storage device to perform or control performance of any of the methods 11-119 described herein.
  • any of the methods 11-119 described above are performed by a storage device including means for performing any of the methods described herein.
  • a storage system includes (1) a storage medium
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a host system, the one or more programs including instructions for performing any of the methods 11-119 described herein.
  • a storage system includes (1) one or more storage devices, (2) one or more subsystems having one or more processors, and (3) memory storing one or more programs, which when executed by the one or more processors cause the one or more subsystems to perform or control performance of any of the methods 11-119 described herein.
  • a host system includes (1) an interface for operatively coupling to a storage system, (2) one or more processors, and (3) controller memory storing one or more programs, which when executed by the one or more processors cause the host system to perform or control performance of any of the methods 11-119 described herein.
  • the various embodiments described herein include systems, methods and/or devices used to reduce declared capacity of a storage device in accordance with a detected amelioration trigger. Some embodiments include systems, methods and/or devices to detect a an amelioration trigger for reducing declared capacity of non- volatile memory of a storage device in a storage system and perform, in accordance with the detected amelioration trigger, an amelioration process to reduce declared capacity of non-volatile memory of the storage device.
  • some embodiments include a method of managing a storage system.
  • the method is performed at a storage device of the storage system.
  • the method includes: (1) detecting an amelioration trigger for reducing declared capacity of non-volatile memory of the storage device of the storage system; and, (2) in accordance with the detected amelioration trigger, performing an amelioration process to reduce declared capacity of the non-volatile memory of the storage device, including: (i) trimming, in accordance with a trim command received from a host, at least a portion of a set of logical addresses in a logical address space; and (ii) reducing declared capacity of the non-volatile memory of the storage device.
  • the host includes a client on behalf of which data is stored in the storage system.
  • the host includes a storage system controller of the storage system.
  • the host includes a cluster controller of the storage system.
  • trimming at least a portion of the set of logical addresses includes trimming at least a portion of the set of logical addresses in accordance with one or more parameters for the amelioration process.
  • trimming at least a portion of the set of logical addresses includes: (i) receiving, from the host, information specifying the set of logical addresses in the logical address space; and (ii) invalidating one or more logical address entries, associated with the set of logical addresses, of a mapping table, the mapping table used to translate logical addresses in the logical address space to physical addresses in a physical address space of the storage device.
  • reducing declared capacity of the non- volatile memory of the storage device includes making a number of logical addresses, less than or equal to a number of logical addresses corresponding to the invalidated logical address entries, unavailable to the host.
  • reducing declared capacity of the non- volatile memory of the storage device includes making a number of logical addresses, less than or equal to a number of logical addresses corresponding to the invalidated logical address entries, unavailable to the host.
  • reducing declared capacity of the non-volatile memory of the storage device includes making a number of logical addresses, greater than or equal to a number of logical addresses corresponding to the invalidated logical address entries, unavailable to the host.
  • performing the amelioration process to reduce declared capacity of the non-volatile memory of the storage device further includes advertising a reduced declared capacity of the non-volatile memory of the storage device.
  • the method further includes: after beginning performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device, detecting an indication to abort the reduction in declared capacity of the non-volatile memory of the storage device; and in response to detecting the indication to abort the reduction in declared capacity of the nonvolatile memory of the storage device, aborting performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • performing an amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes reducing utilization of the non- volatile memory of the storage device.
  • the storage device comprises one or more flash memory devices.
  • a storage device includes (1) non- volatile memory, (2) one or more processors, and (3) controller memory (e.g., a non-transitory computer readable storage medium in the storage device) storing one or more programs, which when executed by the one or more processors cause the storage device to perform or control performance of the method of any of Jl-Jl 1 described herein.
  • controller memory e.g., a non-transitory computer readable storage medium in the storage device
  • a storage device including means for performing or causing performance of the method of any of Jl-Jl 1 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage device, the one or more programs including instructions for performing or causing performance of the method of any of Jl-Jl 1 described herein.
  • a storage system includes (1) a storage medium
  • memory e.g., a non-transitory computer readable storage medium in the storage system
  • programs which when executed by the one or more processors cause the storage system to perform or control performance of the method of any of Jl-Jl 1 described herein.
  • a storage system including means for performing or causing performance of the method of any of Jl-Jl 1 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage system, the one or more programs including instructions for performing or causing performance of the method of any of Jl-Jl 1 described herein.
  • some embodiments include a method of managing a storage system.
  • the method includes: (1) detecting an amelioration trigger for reducing declared capacity of non-volatile memory of a storage device of the storage system, and (2) in accordance with the detected amelioration trigger, performing an amelioration process to reduce declared capacity of the non-volatile memory of the storage device, including deleting from the storage device discardable data that is used by a host, and reducing declared capacity of the non- volatile memory of the storage device.
  • the host includes a client on behalf of which data is stored in the storage system.
  • the host includes a storage system controller of the storage system.
  • the host includes a cluster controller of the storage system.
  • the detecting, the performing, or both the detecting and the performing are performed by one or more subsystems of the storage system distinct from the storage device.
  • deleting discardable data that is used by the host includes deleting discardable data that is used by the host in accordance with one or more parameters for the amelioration process.
  • deleting discardable data that is used by the host includes deleting temporary data.
  • deleting discardable data that is used by the host includes deleting data that can be rebuilt when needed.
  • deleting discardable data that is used by the host includes deleting snapshots of data.
  • deleting discardable data that is used by the host includes deleting data that is pre-marked by the host as data that is discardable.
  • deleting discardable data that is used by the host includes: invalidating one or more logical address entries, associated with the discardable data, of a mapping table, the mapping table used to translate logical addresses in a logical address space to physical addresses in a physical address space of the storage device.
  • reducing declared capacity of the non-volatile memory of the storage device includes making a number of logical addresses, less than or equal to a number of logical addresses corresponding to the invalidated logical address entries, unavailable to the host.
  • performing the amelioration process to reduce declared capacity of the non-volatile memory of the storage device further includes advertising a reduced declared capacity of the non- volatile memory of the storage device.
  • the method further comprises (1) after beginning performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device, detecting an indication to abort the reduction in declared capacity of the non-volatile memory of the storage device, and (2) in response to detecting the indication to abort the reduction in declared capacity of the non-volatile memory of the storage device, aborting performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • the storage device comprises one or more flash memory devices.
  • a storage device includes (1) non- volatile memory,
  • controller memory storing one or more programs, which when executed by the one or more processors cause the storage device to perform or control performance of any of the methods K2-K17 described herein.
  • any of the methods K2-K17 described above are performed by a storage device including means for performing any of the methods described herein.
  • a storage system includes (1) a storage medium
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage device, the one or more programs including instructions for performing any of the methods K1-K17 described herein.
  • a storage system includes (1) one or more storage devices, (2) one or more subsystems having one or more processors, and (3) memory storing one or more programs, which when executed by the one or more processors cause the one or more subsystems to perform or control performance of any of the methods K2-K17 described herein.
  • a host system includes (1) an interface for operatively coupling to a storage system, (2) one or more processors, and (3) controller memory storing one or more programs, which when executed by the one or more processors cause the host system to perform or control performance of any of the methods K2-K17 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a host system, the one or more programs including instructions for performing any of the methods described herein.
  • some embodiments include a method of managing a storage system.
  • the method includes: (1) detecting an amelioration trigger for reducing declared capacity of non-volatile memory of a first storage device of the storage system, and (2) in accordance with the detected amelioration trigger, performing an amelioration process to reduce declared capacity of the non-volatile memory of the first storage device, including moving a portion of data that is used by a host from the first storage device to another storage device of the storage system, and reducing declared capacity of the non-volatile memory of the first storage device.
  • the host includes a client on behalf of which data is stored in the storage system.
  • the host includes a storage system controller of the storage system.
  • the host includes a cluster controller of the storage system.
  • the detecting, the performing, or both the detecting and the performing are performed by the host.
  • the detecting, the performing, or both the detecting and the performing are performed by one or more subsystems of the storage system distinct from the first storage device.
  • moving the portion of data that is used by the host includes moving the portion of data that is used by the host in accordance with one or more parameters for the amelioration process.
  • moving the portion of data that is used by the host includes moving data associated with one or more virtual logical addresses, including updating a virtual address mapping module.
  • moving the portion of the data includes selecting one or more logical addresses of that data so as to minimize performance degradation.
  • moving the portion of the data includes selecting one or more logical addresses of that data so as to minimize overhead from garbage collection.
  • performing the amelioration process to reduce declared capacity of the non-volatile memory of the first storage device further includes: invalidating one or more logical address entries, associated with the portion of data, of a mapping table, the mapping table used to translate logical addresses in the logical address space to physical addresses in a physical address space of the first storage device.
  • reducing declared capacity of the non-volatile memory of the first storage device includes trimming logical addresses associated with the portion of data moved from the first storage device.
  • performing the amelioration process to reduce declared capacity of the non-volatile memory of the first storage device further includes advertising a reduced declared capacity of the non- volatile memory of the first storage device.
  • the method further comprises: (1) after beginning performance of the amelioration process to reduce declared capacity of the non-volatile memory of the first storage device, detecting an indication to abort the reduction in declared capacity of the non- volatile memory of the first storage device; and (2) in response to detecting the indication to abort the reduction in declared capacity of the non-volatile memory of the first storage device, aborting
  • the first storage device comprises one or more flash memory devices.
  • a host system includes: (1) an interface for operatively coupling to a storage system, (2) one or more processors, and (3) controller memory storing one or more programs, which when executed by the one or more processors cause the first storage device to perform or control performance of any of the methods L2- L15 described herein.
  • any of the methods L2-L15 described above are performed by a system including means for performing any of the methods described herein.
  • a storage system includes (1) a storage medium
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage device, the one or more programs including instructions for performing any of the methods LI -LI 5 described herein.
  • a storage system includes (1) one or more storage devices, (2) one or more subsystems having one or more processors, and (3) memory storing one or more programs, which when executed by the one or more processors cause the one or more subsystems to perform or control performance of any of the methods L2-L15 described herein.
  • the various embodiments described herein include systems, methods and/or devices used to reduce declared capacity of a storage device in accordance with a detected amelioration trigger.
  • Some embodiments include systems, methods and/or devices to detect an amelioration trigger for reducing declared capacity of non- volatile memory of a storage device in a storage system and perform, in accordance with the detected amelioration trigger, an amelioration process to reduce declared capacity of non-volatile memory of the storage device.
  • some embodiments include a method of managing a storage system.
  • the method is performed at a host that is operatively coupled with a storage device of a storage system.
  • the method includes: (1) detecting an amelioration trigger for reducing declared capacity of non- volatile memory of the storage device of the storage system; and, (2) in accordance with the detected amelioration trigger, performing an amelioration process to reduce declared capacity of the non- volatile memory of the storage device, including: (i) trimming at least a portion of a set of logical addresses in a logical address space; and (ii) reducing declared capacity of the nonvolatile memory of the storage device.
  • the host includes a client on behalf of which data is stored in the storage system.
  • the host includes a storage system controller of the storage system.
  • the host includes a cluster controller of the storage system.
  • trimming at least a portion of the set of logical addresses includes trimming at least a portion of the set of logical addresses in accordance with one or more parameters for the amelioration process.
  • trimming at least a portion of the set of logical addresses includes: (i) obtaining information specifying the set of logical addresses in the logical address space; and (ii) invalidating one or more logical address entries, associated with the set of logical addresses, of a mapping table, the mapping table used to translate logical addresses in the logical address space to physical addresses in a physical address space of the storage device.
  • reducing declared capacity of the non- volatile memory of the storage device includes making a number of logical addresses, less than or equal to a number of logical addresses corresponding to the invalidated logical address entries, unavailable to the host. Alternatively, in some
  • reducing declared capacity of the non- volatile memory of the storage device includes making a number of logical addresses, greater than or equal to a number of logical addresses corresponding to the invalidated logical address entries, unavailable to the host.
  • performing the amelioration process to reduce declared capacity of the non-volatile memory of the storage device further includes advertising a reduced declared capacity of the non-volatile memory of the storage device.
  • the method further includes: after beginning performance of the amelioration process to reduce declared capacity of the non- volatile memory of the storage device, detecting an indication to abort the reduction in declared capacity of the non-volatile memory of the storage device; and in response to detecting the indication to abort the reduction in declared capacity of the nonvolatile memory of the storage device, aborting performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • performing an amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes reducing utilization of the non-volatile memory of the storage device
  • the storage device comprises one or more flash memory devices.
  • a host system includes (1) non- volatile memory, (2) one or more processors, and (3) memory (e.g., a non-transitory computer readable storage medium in the host system) storing one or more programs, which when executed by the one or more processors cause the host system to perform or control performance of the method of any of Ml -Ml 1 described herein.
  • memory e.g., a non-transitory computer readable storage medium in the host system
  • a host system including means for performing or causing performance of the method of any of Ml -Ml 1 described herein.
  • Ml 4 In yet another aspect, some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a host system, the one or more programs including instructions for performing or causing performance of the method of any of Ml -Ml 1 described herein.
  • a storage system includes (1) one or more storage devices, (2) a host to which the one or more storage devices are operatively coupled, (3) one or more processors, and (4) memory (e.g., a non-transitory computer readable storage medium in the storage system) storing one or more programs, which when executed by the one or more processors cause the host to perform control performance of the method of any of M 1 -M 11 described herein.
  • memory e.g., a non-transitory computer readable storage medium in the storage system
  • a storage system including means for performing or causing performance of the method of any of Ml -Ml 1 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage system, the one or more programs including instructions for performing or causing performance of the method of any of Ml -Ml 1 described herein.
  • the various embodiments described herein include systems, methods and/or devices used to reduce declared capacity of a storage device in accordance with a detected amelioration trigger. Some embodiments include systems, methods and/or devices to detect an amelioration trigger for reducing declared capacity of non- volatile memory of a storage device in a storage system and perform, in accordance with the detected amelioration trigger, an amelioration process to reduce declared capacity of non-volatile memory of the storage device.
  • some embodiments include a method of managing a storage system.
  • the method includes: (1) detecting an amelioration trigger for reducing declared capacity of non-volatile memory of a storage device of the storage system; and, (2) in accordance with the detected amelioration trigger, performing an amelioration process to reduce declared capacity of the non-volatile memory of the storage device, including: (i) obtaining a target capacity of the non-volatile memory of the storage device and a current utilization of the non-volatile memory of the storage device, wherein the target capacity is lower than a current capacity of the non-volatile memory of the storage device and the current utilization corresponds to allocated logical addresses in a logical address space; (ii) determining whether the target capacity of the non- volatile memory of the storage device is greater than the current utilization of the non- volatile memory of the storage device; (iii) in accordance with a determination that the target capacity is greater than the current utilization, forgoing trimming of the allocated logical addresses; (ii)
  • the host includes a client on behalf of which data is stored in the storage system.
  • the host includes a storage system controller of the storage system.
  • the host includes a cluster controller of the storage system.
  • N5 In some embodiments of the method of any of Nl to N4, the detecting, the performing, or both the detecting and the performing are performed by the storage device
  • the detecting, the performing, or both the detecting and the performing are performed by one or more subsystems of the storage system distinct from the storage device.
  • N7 In some embodiments of the method of any of Nl to N4, the detecting, the performing, or both the detecting and the performing are performed by the host.
  • trimming at least a portion of the allocated logical addresses includes trimming at least a portion of the allocated logical addresses in accordance with one or more parameters for the amelioration process.
  • trimming at least a portion of the allocated logical addresses includes: (i) obtaining information specifying a set of logical addresses in the logical address space; and (ii) invalidating one or more logical address entries, associated with the set of logical addresses, of a mapping table, the mapping table used to translate logical addresses in the logical address space to physical addresses in a physical address space of the storage device.
  • reducing declared capacity of the non- volatile memory of the storage device includes making a number of logical addresses, less than or equal to a number of logical addresses corresponding to the invalidated logical address entries, unavailable to the host. Alternatively, in some
  • reducing declared capacity of the non-volatile memory of the storage device includes making a number of logical addresses, greater than or equal to a number of logical addresses corresponding to the invalidated logical address entries, unavailable to the host.
  • performing the amelioration process to reduce declared capacity of the non-volatile memory of the storage device further includes advertising a reduced declared capacity of the non- volatile memory of the storage device.
  • the method further includes: after beginning performance of the amelioration process to reduce declared capacity of the non- volatile memory of the storage device, detecting an indication to abort the reduction in declared capacity of the non-volatile memory of the storage device; and in response to detecting the indication to abort the reduction in declared capacity of the nonvolatile memory of the storage device, aborting performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • performing an amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes reducing utilization of the non- volatile memory of the storage device.
  • the storage device comprises one or more flash memory devices.
  • a storage device includes (1) non- volatile memory,
  • controller memory e.g., a non-transitory computer readable storage medium in the storage device
  • controller memory e.g., a non-transitory computer readable storage medium in the storage device
  • some embodiments include a storage device including means for performing or causing performance of the method of any of N1-N5 and N8-N14 described herein. [00375] (N17) In yet another aspect, some embodiments include a storage system including means for performing or causing performance of the method of any of N1-N14 described herein.
  • a storage system includes (1) a storage medium
  • memory e.g., a non-transitory computer readable storage medium in the storage system
  • programs which when executed by the one or more processors cause the storage system to perform or control performance of the method of any of N1-N14 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage device, the one or more programs including instructions for performing or causing performance of the method of any of N1-N5 and N8- N14 described herein.
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a storage system, the one or more programs including instructions for performing or causing performance of the method of any of N1-N14 described herein.
  • a storage system includes (1) one or more storage devices, (2) one or more subsystems having one or more processors, and (3) memory (e.g., a non-transitory computer readable storage medium in the one or more subsystems) storing one or more programs, which when executed by the one or more processors cause the one or more subsystems to perform or control performance of the method of any of N1-N14 described herein.
  • memory e.g., a non-transitory computer readable storage medium in the one or more subsystems
  • a host system includes (1) optionally, an interface for operatively coupling to a storage system, (2) one or more processors, and (3) memory (e.g., a non-transitory computer readable storage medium in the host system) storing one or more programs, which when executed by the one or more processors cause the host system to perform or control performance of the method of any of N1-N4 and N6-N14 described herein.
  • memory e.g., a non-transitory computer readable storage medium in the host system
  • some embodiments include a non-transitory computer readable storage medium, storing one or more programs configured for execution by one or more processors of a host system, the one or more programs including instructions for performing or causing performance of the method of any of N1-N4 and N6-N14 described herein.
  • a storage controller is configured to divide a plurality of flash memory devices into a plurality of logical chunks. After detecting a predefined trigger condition as to a respective flash memory device of the plurality of flash memory devices, the storage controller is configured to initiate a migration process whereby at least one logical chunk on the respective flash memory device is migrated to a second flash memory device different from the respective flash memory device. As such, logical chunks are migrated off of a flash memory device as trigger conditions are detected as to the flash memory device thereby slowly phasing out the flash memory device instead of the entire flash memory device failing and consequently replicated at once.
  • the various embodiments described herein include devices and/or methods that may improve the endurance of a storage system including a storage medium (e.g., comprising one or more flash memory devices). Some embodiments include methods and/or systems for managing a storage system by migrating data between flash memory devices of the storage system.
  • a storage medium e.g., comprising one or more flash memory devices.
  • some embodiments include a method of managing a storage system that comprises a storage controller and a plurality of flash memory devices each with a plurality of flash memory blocks.
  • the method is performed at the storage controller.
  • the method includes: dividing each of the plurality of flash memory devices into a plurality of logical chunks, where each logical chunk includes one or more flash memory blocks; and detecting a predefined trigger condition with respect to a respective flash memory device of the plurality of flash memory devices.
  • the method includes: selecting one of the plurality of logical chunks of the respective flash memory device for migration in accordance with predefined selection criteria; and storing a replicated logical chunk, comprising a copy of the selected logical chunk, at a second flash memory device different from the respective flash memory device.
  • the method further includes: remapping an address of the selected logical chunk to a physical location of the replicated logical chunk; and decreasing a number of logical chunks associated with the respective flash memory device.
  • Some embodiments include a storage controller comprising: one or more processors; a host interface configured to couple the storage controller to a host; a storage medium interface configured to couple the storage controller to a storage medium including a plurality of flash memory devices, each with a plurality of flash memory blocks; and a storage controller storing instructions, which when executed by the one or more processors, cause the storage controller to perform the operations of any of the methods described herein.
  • Some embodiments include a non-transitory computer readable storage medium, storing one or more programs for execution by one or more processors of a storage controller coupled to a storage medium including a plurality of flash memory devices, where each flash memory device includes a plurality of flash memory blocks, the one or more programs including instructions that when executed by the one or more processors cause the storage controller to perform the operations of any of the methods described herein.
  • Data storage systems use a variety of techniques to avoid data loss caused by a variety of failure mechanisms, including storage media failure, communication failures, and failures at the system and subsystem level.
  • a common feature of these mechanisms is the use of data redundancy to protect data, to compensate for actual and potential data errors (e.g., media errors, lost data, transmission errors, inaccessible data, etc.).
  • One class of redundancy mechanisms is known as error correction codes (ECCs).
  • ECCs error correction codes
  • Numerous types of error correction codes are well known (e.g., BCH, LDPC, Reed-Solomon, etc.), as are numerous schemes for storing them with or in conjunction with the data that is being protected.
  • Another class of redundancy mechanisms is erasure codes (e.g., pyramid, fountain, partial MDS, locally repairable, simple regenerating, etc.)
  • RAID redundancy mechanism
  • RAID redundant array of independent disks
  • RAID schemes providing different levels of data protection (e.g., RAID-1, RAID-5, RAID-6, RAID-10, etc.).
  • stripes of data stored in multiple distinct storage locations are treated as a set, and stored with sufficient redundant data that any data in a stripe that would have been lost, in a partial or complete failure of any one of the storage locations, is recovered using the other data in the stripe, possibly including the redundant data.
  • a third type of redundancy mechanism is replication of data to multiple storage locations, typically in distinct failure domains.
  • Systems implementing this type of redundancy mechanism typically store three or more replicas of each data set or data item. Typically either each replica is in a distinct failure domain from the other replicas, or at least one replica is in a distinct failure domain from the other replicas.
  • the embodiments described below work in conjunction with the data redundancy mechanisms described above (used alone or in combination).
  • Some of the data storage systems described below have an architecture or configuration designed to implement a particular redundancy mechanism.
  • some of the embodiments described below may utilize more than one of the redundancy mechanisms described above, either alone or in combination.
  • some of the embodiments are able to store data encoded with different redundancy mechanisms simultaneously.
  • the selection of parameters i.e., codeword size relative to data size
  • altering the redundancy mechanism directly affects the amount of data stored and in turn the utilization.
  • FIG. 1A is a block diagram illustrating data storage system 100, in accordance with some embodiments. While some example features are illustrated, various other features have not been illustrated for the sake of brevity and so as not to obscure pertinent aspects of the example embodiments disclosed herein. To that end, as a non- limiting example, data storage system 100 includes a storage device 120, which includes a storage controller 124 and a storage medium 130, and is used in conjunction with or includes a computer system 110.
  • storage medium 130 is a single flash memory device while in other embodiments storage medium 130 includes a plurality of flash memory devices.
  • storage medium 130 is NAND-type flash memory or NOR- type flash memory.
  • storage medium 130 includes one or more three- dimensional (3D) memory devices, as further defined herein.
  • storage controller 124 is a solid-state drive (SSD) controller.
  • SSD solid-state drive
  • other types of storage media may be included in accordance with aspects of a wide variety of embodiments (e.g., PCRAM, ReRAM, STT-RAM, etc.).
  • a flash memory device includes one or more flash memory die, one or more flash memory packages, one or more flash memory channels or the like.
  • data storage system 100 can contain one or more storage device 120s.
  • Computer system 110 is coupled to storage controller 124 through data connections 101.
  • computer system 110 includes storage controller 124, or a portion of storage controller 124, as a component and/or a subsystem.
  • storage controller 124 may be any suitable computer device, such as a computer, a laptop computer, a tablet device, a netbook, an internet kiosk, a personal digital assistant, a mobile phone, a smart phone, a gaming device, a computer server, or any other computing device.
  • Computer system 110 is sometimes called a host, host system, client, or client system.
  • computer system 110 is a server system, such as a server system in a data center.
  • computer system 110 includes one or more processors, one or more types of memory, a display and/or other user interface components such as a keyboard, a touch screen display, a mouse, a track-pad, a digital camera and/or any number of supplemental devices to add functionality.
  • computer system 110 does not have a display and other user interface components.
  • Storage medium 130 is coupled to storage controller 124 through connections
  • Connections 103 are sometimes called data connections, but typically convey commands in addition to data, and optionally convey metadata, error correction information and/or other information in addition to data values to be stored in storage medium 130 and data values read from storage medium 130.
  • storage controller 124 and storage medium 130 are included in the same device (i.e., an integral device) as components thereof.
  • storage controller 124 and storage medium 130 are embedded in a host device (e.g., computer system 110), such as a mobile device, tablet, other computer or computer controlled device, and the methods described herein are performed, at least in part, by the embedded memory controller.
  • Storage medium 130 may include any number (i.e., one or more) of memory devices including, without limitation, non-volatile semiconductor memory devices, such as flash memory device(s).
  • flash memory device(s) can be configured for enterprise storage suitable for applications such as cloud computing, for database applications, primary and/or secondary storage, or for caching data stored (or to be stored) in secondary storage, such as hard disk drives.
  • flash memory device(s) can also be configured for relatively smaller-scale applications such as personal flash drives or hard-disk replacements for personal, laptop, and tablet computers.
  • storage medium 130 includes one or more three-dimensional (3D) memory devices, as further defined herein.
  • Storage medium 130 is divided into a number of addressable and individually selectable blocks, such as selectable portion 131.
  • the individually selectable blocks are the minimum size erasable units in a flash memory device.
  • each block contains the minimum number of memory cells that can be erased simultaneously.
  • Each block is usually further divided into a plurality of pages and/or word lines, where each page or word line is typically an instance of the smallest individually accessible (readable) portion in a block.
  • the smallest individually accessible unit of a data set is a sector, which is a subunit of a page. That is, a block includes a plurality of pages, each page contains a plurality of sectors, and each sector is the minimum unit of data for reading data from the flash memory device.
  • error control coding can be utilized to limit the number of uncorrectable errors that are introduced by electrical fluctuations, defects in the storage medium, operating conditions, device history, write -read circuitry, etc., or a combination of these and various other factors.
  • storage controller 124 includes a management module
  • Storage controller 124 may include various additional features that have not been illustrated for the sake of brevity and so as not to obscure pertinent features of the example embodiments disclosed herein, and a different arrangement of features may be possible.
  • Host interface 129 provides an interface to computer system 110 through data connections 101.
  • storage medium I/O 128 provides an interface to storage medium 130 though connections 103.
  • storage medium I/O 128 includes read and write circuitry, including circuitry capable of providing reading signals to storage medium 130 (e.g., reading threshold voltages for NAND-type flash memory).
  • management module 121-1 includes one or more processing units (CPUs, also sometimes called processors) 122-1 configured to execute instructions in one or more programs (e.g., in management module 121-1). In some embodiments, the one or more CPUs 122-1 are shared by one or more components within, and in some cases, beyond the function of storage controller 124. Management module 121-1 is coupled to host interface 129, additional module(s) 125 and storage medium I/O 128 in order to coordinate the operation of these components. In some embodiments, one or more modules of management module 121-1 are implemented in management module 121-2 of computer system 110. In some embodiments, one or more processors of computer system 110 (not shown) are configured to execute instructions in one or more programs (e.g., in management module 121-2). Management module 121-2 is coupled to storage device 120 in order to manage the operation of storage device 120.
  • CPUs also sometimes called processors
  • Management module 121-2 is coupled to storage device 120 in order to manage the operation of storage device 120.
  • Additional module(s) 125 are coupled to storage medium I/O 128, host interface 129, and management module 121-1.
  • additional module(s) 125 may include an error control module to limit the number of uncorrectable errors inadvertently introduced into data during writes to memory or reads from memory.
  • additional module(s) 125 are executed in software by the one or more CPUs 122-1 of management module 121-1, and, in other embodiments, additional module(s) 125 are implemented in whole or in part using special purpose circuitry (e.g., to perform encoding and decoding functions).
  • additional module(s) 125 are implemented in whole or in part by software executed on computer system 110.
  • an error control module included in additional module(s) 125, includes an encoder and a decoder.
  • the encoder encodes data by applying an error control code to produce a codeword, which is subsequently stored in storage medium 130.
  • the decoder applies a decoding process to the encoded data to recover the data, and to correct errors in the recovered data within the error correcting capability of the error control code.
  • each type or family of error control codes may have encoding and decoding algorithms that are particular to the type or family of error control codes.
  • some algorithms may be utilized at least to some extent in the decoding of a number of different types or families of error control codes.
  • an exhaustive description of the various types of encoding and decoding algorithms generally available and known to those skilled in the art is not provided herein.
  • host interface 129 receives data to be stored in storage medium 130 from computer system 110.
  • the data received by host interface 129 is made available to an encoder (e.g., in additional module(s) 125), which encodes the data to produce one or more codewords.
  • the one or more codewords are made available to storage medium I/O 128, which transfers the one or more codewords to storage medium 130 in a manner dependent on the type of storage medium being utilized.
  • a read operation is initiated when computer system
  • (host) 110 sends one or more host read commands (e.g., via data connections 101, or alternatively a separate control line or bus) to storage controller 124 requesting data from storage medium 130.
  • Storage controller 124 sends one or more read access commands to storage medium 130, via storage medium I/O 128, to obtain raw read data in accordance with memory locations (addresses) specified by the one or more host read commands.
  • Storage medium I/O 128 provides the raw read data (e.g., comprising one or more codewords) to a decoder (e.g., in additional module(s) 125). If the decoding is successful, the decoded data is provided to host interface 129, where the decoded data is made available to computer system 110. In some embodiments, if the decoding is not successful, storage controller 124 may resort to a number of remedial actions or provide an indication of an irresolvable error condition.
  • a storage medium e.g., storage medium 130
  • each block is optionally
  • the smaller subunit of a block consists of multiple memory cells (e.g., single-level cells or multi-level cells).
  • programming is performed on an entire page.
  • a multi-level cell (MLC) NAND flash typically has four possible states per cell, yielding two bits of information per cell.
  • MLC NAND has two page types: (1) a lower page (sometimes called fast page), and (2) an upper page (sometimes called slow page).
  • a triple-level cell (TLC) NAND flash has eight possible states per cell, yielding three bits of information per cell.
  • the encoding format of the storage media is a choice made when data is actually written to the storage media.
  • an event, condition, or process that is said to set the encoding format, alter the encoding format of the storage media, etc. It should be recognized that the actual process may involve multiple steps, e.g., erasure of the previous contents of the storage media followed by the data being written using the new encoding format and that these operations may be separated in time from the initiating event, condition or procedure.
  • pages in the storage medium may contain invalid (e.g., stale) data, but those pages cannot be overwritten until the whole block containing those pages is erased.
  • the pages (if any) with valid data in that block are read and re -written to a new block and the old block is erased (or put on a queue for erasing).
  • This process is called garbage collection.
  • the new block contains the pages with valid data and may have free pages that are available for new data to be written, and the old block can be erased so as to be available for new data to be written. Since flash memory can only be programmed and erased a limited number of times, the efficiency of the algorithm used to pick the next block(s) to re -write and erase has a significant impact on the lifetime and reliability of flash-based storage systems.
  • FIG. 1B is a block diagram illustrating data storage system 140, in accordance with some embodiments. While some example features are illustrated, various other features have not been illustrated for the sake of brevity and so as not to obscure pertinent aspects of the example embodiments disclosed herein. To that end, as a non- limiting example, data storage system 140 (sometimes called a scale-up storage system, a single node storage system, etc.) includes a plurality of storage devices 160 (e.g., storage devices 160-1 to 160-m) and a storage system controller 150, and is used in conjunction with a computer system 142.
  • storage devices 160 e.g., storage devices 160-1 to 160-m
  • storage devices 160 include management modules 161 (e.g., storage device 160-1 includes management module 161-1 and storage device 160-m includes management module 161-m).
  • management modules 161 e.g., storage device 160-1 includes management module 161-1 and storage device 160-m includes management module 161-m.
  • Computer system 142 is coupled to storage system controller 150 through connections 141. However, in some embodiments computer system 142 includes a part of or the entire storage system controller 150 as a component and/or a subsystem. For example, in some embodiments, some or all of the functionality of storage system controller 150 is implemented by software executed on computer system 142.
  • Computer system 142 may be any suitable computer device, such as a computer, a laptop computer, a tablet device, a netbook, an internet kiosk, a personal digital assistant, a mobile phone, a smart phone, a gaming device, a computer server, or any other computing device. In some embodiments, computer system 142 is a server system, such as a server system in a data center.
  • Computer system 142 is sometimes called a host, host system, client, or client system.
  • computer system 142 includes one or more processors, one or more types of memory, a display and/or other user interface components such as a keyboard, a touch screen display, a mouse, a track-pad, a digital camera and/or any number of supplemental devices to add functionality.
  • computer system 142 does not have a display and other user interface components.
  • storage system controller 150 includes a system management module 151-1, and additional module(s) 155.
  • Storage system controller 150 may include various additional features that have not been illustrated for the sake of brevity and so as not to obscure pertinent features of the example embodiments disclosed herein, and a different arrangement of features may be possible.
  • storage system controller 150 additionally includes an interface for each of the storage devices 160 coupled to storage system controller 150.
  • Storage devices 160 are coupled to storage system controller 150 through connections 143 (e.g., storage device 160-1 is coupled to storage system controller 150 through connections 143-1 and storage device 160-m is coupled to storage system controller 150 through connections 143-m).
  • connections 143-1 through 143-m are implemented as a communication media over which commands and data are communicated using a protocol such as SCSI, SAT A, Infiniband, Ethernet, Token Ring, or the like.
  • system management module 151-1 includes one or more processing units (CPUs, also sometimes called processors) 152-1 configured to execute instructions in one or more programs (e.g., in system management module 151-1).
  • the one or more CPUs 152-1 are shared by one or more components within, and in some cases, beyond the function of storage system controller 150.
  • management module 151-1 is coupled to additional module(s) 155 in order to coordinate the operation of these components.
  • one or more modules of system management module 151-1 are implemented in system management module 151-2 of computer system 142 (sometimes called a host, host system, client, or client system).
  • one or more processors (sometimes called CPUs or processing units) of computer system 142 are configured to execute instructions in one or more programs (e.g., in system management module 151-2).
  • System management module 151-2 is coupled to storage system controller 150 in order to manage the operation of storage system controller 150.
  • Additional module(s) 155 are coupled to system management module 151-1.
  • additional module(s) 155 are executed in software by the one or more CPUs 152-1 of system management module 151-1, and, in other embodiments, additional module(s) 155 are implemented in whole or in part using special purpose circuitry. In some embodiments, additional module(s) 155 are implemented in whole or in part by software executed on computer system 142.
  • storage system controller 150 receives data to be stored in storage devices 160 from computer system 142 (sometimes called a host, host system, client, or client system). In some embodiments, storage system controller 150 maps a virtual logical address from computer system 142 to an address, which determines or identifies the one or more of storage devices 160 to which to write the data.
  • computer system 142 sometimes called a host, host system, client, or client system.
  • storage system controller 150 maps a virtual logical address from computer system 142 to an address, which determines or identifies the one or more of storage devices 160 to which to write the data.
  • FIG. 1C is a block diagram illustrating an implementation of data storage system 170, in accordance with some embodiments. While some example features are illustrated, various other features have not been illustrated for the sake of brevity and so as not to obscure pertinent aspects of the example embodiments disclosed herein.
  • data storage system 170 (sometimes called a scale-out storage system, a multiple node storage system or a storage cluster system) includes a plurality of storage subsystems 192 (e.g., storage subsystems 192-1 to 192-s) and a cluster controller 180, and is used in conjunction with a computer system 172.
  • storage subsystems 192 include storage system controllers 190 and storage devices 194 (e.g., storage subsystem 192-1 includes storage system controller 190-1 and storage devices 194-1 through 194-n).
  • Computer system 172 is coupled to cluster controller 180 through connections
  • computer system 172 includes cluster controller 180 as a component and/or a subsystem.
  • cluster controller 180 may be any suitable computer device, such as a computer, a laptop computer, a tablet device, a netbook, an internet kiosk, a personal digital assistant, a mobile phone, a smart phone, a gaming device, a computer server, or any other computing device.
  • computer system 172 is a server system, such as a server system in a data center.
  • Computer system 172 is sometimes called a host, host system, client, or client system.
  • computer system 172 includes one or more processors, one or more types of memory, a display and/or other user interface components such as a keyboard, a touch screen display, a mouse, a track-pad, a digital camera and/or any number of supplemental devices to add functionality. In some embodiments, computer system 172 does not have a display and other user interface components.
  • cluster controller 180 includes a cluster management module 181-1, and additional module(s) 185.
  • Cluster controller 180 may include various additional features that have not been illustrated for the sake of brevity and so as not to obscure pertinent features of the example embodiments disclosed herein, and a different arrangement of features may be possible.
  • cluster controller 180 additionally includes an interface for each of the storage subsystems 192 coupled to cluster controller 180.
  • Storage subsystems 192 are coupled to cluster controller 180 through connections 173 (e.g., storage subsystem 192-1 is coupled to cluster controller 180 through connections 173-1 and storage subsystem 192-s is coupled to cluster controller 180 through connections 173-s).
  • connections 173 may be
  • a shared communication network e.g., Token Ring, Ethernet, Infmiband, etc.
  • cluster management module 181-1 includes one or more processing units (CPUs, also sometimes called processors) 182-1 configured to execute instructions in one or more programs (e.g., in cluster management module 181-1).
  • the one or more CPUs 182-1 are shared by one or more components within, and in some cases, beyond the function of cluster controller 180.
  • Cluster management module 181-1 is coupled to additional module(s) 185 in order to coordinate the operation of these components.
  • one or more modules of cluster management module 181-1 are implemented in cluster management module 181-2 of computer system 172 (sometimes called a host, host system, client, or client system).
  • one or more processors (sometimes called CPUs or processing units) of computer system 172 (not shown) are configured to execute instructions in one or more programs (e.g., in cluster management module 181-2).
  • Cluster management module 181-2 is coupled to cluster controller 180 in order to manage the operation of cluster controller 180.
  • Additional module(s) 185 are coupled to cluster management module 181-1.
  • additional module(s) 185 are executed in software by the one or more CPUs 182-1 of cluster management module 181-1, and, in other embodiments, additional module(s) 185 are implemented in whole or in part using special purpose circuitry.
  • additional module(s) 185 are implemented in whole or in part by software executed on computer system 172.
  • cluster controller 180 receives data to be stored in storage subsystems 192 from computer system 172 (sometimes called a host, host system, client, or client system).
  • cluster controller 180 maps a virtual logical address from computer system 172 to an address format understandable by storage subsystems 192 and to identify a storage subsystem of storage subsystems 192 to which to write the data.
  • cluster controller 180 may convert the data to be stored into a plurality of sets of data, each set of data is stored on one storage subsystem of storage subsystems 192.
  • the conversion process may be as simple as a partitioning of the data to be stored.
  • the conversion process may redundantly encode the data to be stored so as to provide enhanced data integrity and access in the face of failures of one or more storage subsystems of storage subsystems 192 or communication thereto.
  • a read operation is initiated when computer system 172 sends one or more host read commands to cluster controller 180 requesting data from storage subsystems 192.
  • cluster controller 180 maps a virtual logical address from computer system 172 to an address format understandable by storage subsystems 192, to determine or identify the storage subsystem of storage subsystems 192 from which to read the requested data.
  • more than one storage subsystem of storage subsystems 192 may have data read in order to satisfy the read operation, e.g. for data reconstruction.
  • host or "host system” may be construed to mean
  • a computer system e.g., computer system 110, Figure 1A, computer system 142, Figure IB, or computer system 172, Figure 1C
  • a storage system e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, or data storage system 170, Figure 1C
  • a storage system controller e.g., storage system controller 150, Figure IB
  • a cluster controller e.g., cluster controller 180, Figure 1C of a storage system (e.g., data storage system 170, Figure 1C)
  • any computing entity e.g., a computer, a process running on a computer, a mobile phone, an internet kiosk, a tablet computer, a laptop computer, a desktop computer, a server computer, etc.
  • any computing entity e.g., a computer, a process running on a computer, a mobile phone, an internet kiosk, a tablet computer, a laptop computer, a desktop computer, a server computer, etc.
  • any computing entity e
  • the term “host” may refer to computer system 142 or storage system controller 150, depending on the context.
  • the term “host” may refer to computer system 172 or cluster controller 180, depending on the context.
  • the host is or includes a client or client system, on behalf of which data is stored in a storage system.
  • FIG. 2A-1 is a block diagram illustrating a management module 121-1, in accordance with some embodiments, as shown in Figure 1A.
  • Management module 121-1 typically includes one or more processing units (sometimes called CPUs or processors) 122-1 for executing modules, programs and/or instructions stored in memory 206-1 and thereby performing processing operations, memory 206-1 (sometimes called controller memory), and one or more communication buses 208-1 for interconnecting these components.
  • the one or more communication buses 208-1 optionally include circuitry (sometimes called a chipset) that interconnects and controls communications between system components.
  • Management module 121-1 is coupled to host interface 129, additional module(s) 125, and storage medium I/O 128 by the one or more communication buses 208-1.
  • Memory 206-1 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and may include non- volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 206-1 optionally includes one or more storage devices remotely located from the CPU(s) 122-1. Memory 206-1, or alternatively the non- volatile memory device(s) within memory 206-1, comprises a non-transitory computer readable storage medium. In some embodiments, memory 206-1, or the non-transitory computer readable storage medium of memory 206-1 stores the following programs, modules, and data structures, or a subset or superset thereof:
  • translation table 212-1 that is used for mapping logical addresses to physical
  • translation table 212-1 includes mapping table 402, Figure 4);
  • data read module 214-1 that is used for reading data from one or more codewords, pages or blocks in a storage medium (e.g., storage medium 130, Figure 1A);
  • data write module 216-1 that is used for writing data to one or more codewords, pages or blocks in a storage medium (e.g., storage medium 130, Figure 1A);
  • data erase module 218-1 that is used for erasing data from one or more blocks in a storage medium (e.g., storage medium 130, Figure 1A);
  • garbage collection module 220-1 that is used for garbage collection for one or more blocks in a storage medium (e.g., storage medium 130, Figure 1A);
  • metrics module 222-1 that is used for generating and/or obtaining one or more metrics of a storage device (e.g., storage device 120, Figure 1A);
  • trigger detection module 224-1 that is used for detecting a trigger condition (e.g., in accordance with one or more metrics of a storage device);
  • enabling module 226-1 that is used for enabling an amelioration process associated with a trigger condition (e.g., detected by trigger detection module 224-1); • notification module 228-1 that is used for notifying a host to which a storage device is operatively coupled of a trigger condition (e.g., detected by trigger detection module 224-1) and/or of an absence of the trigger condition;
  • amelioration module 230-1 that is used for performing an amelioration process to reduce declared capacity of non-volatile memory of a storage device (e.g., storage device 120, Figure 1A), optionally including: o detection module 231-1 that is used for detecting an amelioration trigger for reducing declared capacity of the non-volatile memory of the storage device; o utilization module 232-1 that is used for reducing utilization of the nonvolatile memory of the storage device; and o capacity module 234-1 that is used for reducing declared capacity of the nonvolatile memory of the storage device.
  • o detection module 231-1 that is used for detecting an amelioration trigger for reducing declared capacity of the non-volatile memory of the storage device
  • o utilization module 232-1 that is used for reducing utilization of the nonvolatile memory of the storage device
  • o capacity module 234-1 that is used for reducing declared capacity of the nonvolatile memory of the storage device.
  • Each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above.
  • the above identified modules or programs i.e., sets of instructions
  • memory 206-1 may store a subset of the modules and data structures identified above.
  • memory 206-1 may store additional modules and data structures not described above.
  • the programs, modules, and data structures stored in memory 206-1, or the non-transitory computer readable storage medium of memory 206-1 provide instructions for implementing some of the methods described below.
  • some or all of these modules may be implemented with specialized hardware circuits that subsume part or all of the module functionality.
  • Figure 2A-1 shows management module 121-1 in accordance with some embodiments
  • Figure 2A-1 is intended more as a functional description of the various features which may be present in management module 121-1 than as a structural schematic of the embodiments described herein.
  • the programs, modules, and data structures shown separately could be combined and some programs, modules, and data structures could be separated.
  • Figure 2A-2 is a block diagram illustrating a management module 121-2, in accordance with some embodiments.
  • Management module 121-2 typically includes one or more processing units (sometimes called CPUs or processors) 122-2 for executing modules, programs and/or instructions stored in memory 206-2 and thereby performing processing operations, memory 206-2, and one or more communication buses 208-2 for interconnecting these components.
  • the one or more communication buses 208-2 optionally include circuitry (sometimes called a chipset) that interconnects and controls communications between system components.
  • Management module 121-2 is coupled to storage device 120 by the one or more communication buses 208-2.
  • Memory 206-2 (sometimes called host memory) includes highspeed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and may include non- volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 206-2 optionally includes one or more storage devices remotely located from the CPU(s) 122-2. Memory 206-2, or alternatively the non-volatile memory device(s) within memory 206-2, comprises a non-transitory computer readable storage medium. In some embodiments, memory 206-2, or the non-transitory computer readable storage medium of memory 206-2 stores the following programs, modules, and data structures, or a subset or superset thereof:
  • translation table 212-2 that is used for mapping logical addresses to physical
  • translation table 212-2 includes mapping table 402, Figure 4;
  • data read module 214-2 that is used for reading data from one or more codewords, pages or blocks in a storage medium (e.g., storage medium 130, Figure 1A);
  • data write module 216-2 that is used for writing data to one or more codewords, pages or blocks in a storage medium (e.g., storage medium 130, Figure 1A);
  • data erase module 218-2 that is used for erasing data from one or more blocks in a storage medium (e.g., storage medium 130, Figure 1A);
  • garbage collection module 220-2 that is used for garbage collection for one or more blocks in a storage medium (e.g., storage medium 130, Figure 1A);
  • metrics module 222-2 that is used for generating and/or obtaining one or more metrics of a storage device (e.g., storage device 120, Figure 1A); • trigger detection module 224-2 that is used for detecting a trigger condition (e.g., in accordance with one or more metrics of a storage device);
  • enabling module 226-2 that is used for enabling an amelioration process associated with a trigger condition (e.g., detected by trigger detection module 224-2);
  • notification module 228-2 that is used for notifying an application, module or process, of the host, of a trigger condition (e.g., detected by trigger detection module 224-2) and/or of an absence of the trigger condition;
  • amelioration module 230-2 that is used for performing an amelioration process to reduce declared capacity of non-volatile memory of a storage device (e.g., storage device 120, Figure 1A), optionally including: o detection module 231-2 that is used for detecting an amelioration trigger for reducing declared capacity of the non-volatile memory of the storage device; o utilization module 232-2 that is used for reducing utilization of the nonvolatile memory of the storage device; and o capacity module 234-2 that is used for reducing declared capacity of the nonvolatile memory of the storage device.
  • o detection module 231-2 that is used for detecting an amelioration trigger for reducing declared capacity of the non-volatile memory of the storage device
  • o utilization module 232-2 that is used for reducing utilization of the nonvolatile memory of the storage device
  • o capacity module 234-2 that is used for reducing declared capacity of the nonvolatile memory of the storage device.
  • Each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above.
  • the above identified modules or programs i.e., sets of instructions
  • memory 206-2 may store a subset of the modules and data structures identified above.
  • memory 206-2 may store additional modules and data structures not described above.
  • the programs, modules, and data structures stored in memory 206-2, or the non-transitory computer readable storage medium of memory 206-2 provide instructions for implementing some of the methods described below.
  • some or all of these modules may be implemented with specialized hardware circuits that subsume part or all of the module functionality.
  • Figure 2A-2 shows management module 121-2 in accordance with some embodiments
  • Figure 2A-2 is intended more as a functional description of the various features which may be present in management module 121-2 than as a structural schematic of the embodiments described herein.
  • the programs, modules, and data structures shown separately could be combined and some programs, modules, and data structures could be separated.
  • Figure 2B-1 is a block diagram illustrating a system management module 151-
  • System management module is in a storage system controller, as shown in Figure IB.
  • System management module 151-1 typically includes one or more processing units (sometimes called CPUs or processors) 152-1 for executing modules, programs and/or instructions stored in memory 246-1 and thereby performing processing operations, memory 246-1 (sometimes called storage system controller memory or controller memory), and one or more
  • the one or more communication buses 248-1 optionally include circuitry (sometimes called a chipset) that interconnects and controls communications between system components.
  • Memory 246-1 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices.
  • Memory 246-1 optionally includes one or more storage devices remotely located from the CPU(s) 152-1.
  • Memory 246-1, or alternatively the non- volatile memory device(s) within memory 246-1 comprises a non-transitory computer readable storage medium.
  • memory 246-1, or the non-transitory computer readable storage medium of memory 246-1 stores the following programs, modules, and data structures, or a subset or superset thereof:
  • system mapping module 250-1 that is used for mapping virtual logical addresses (e.g., used by computer system 142, Figure IB) to intermediate addresses (e.g., which are mapped by storage devices 160 to physical addresses, Figure IB);
  • metrics module 252-1 that is used for generating and/or obtaining one or more metrics of a storage device (e.g., any of storage devices 160-1 to 160-m, Figure IB);
  • trigger detection module 254-1 that is used for detecting a trigger condition (e.g., in accordance with one or more metrics of a storage device); • enabling module 256-1 that is used for enabling an amelioration process associated with a trigger condition (e.g., detected by trigger detection module 254-1);
  • notification module 258-1 that is used for notifying a host to which a storage device is operatively coupled of a trigger condition (e.g., detected by trigger detection module 254-1) and/or of an absence of the trigger condition;
  • amelioration module 260-1 that is used for performing an amelioration process to reduce declared capacity of non-volatile memory of a storage device (e.g., storage device 160, Figure IB), optionally including: o detection module 261-1 that is used for detecting an amelioration trigger for reducing declared capacity of the non-volatile memory of the storage device; o utilization module 262-1 that is used for reducing utilization of the nonvolatile memory of the storage device; and o capacity module 264-1 that is used for reducing declared capacity of the nonvolatile memory of the storage device; and
  • data redundancy module 266-1 that is used for redundantly encoding data (e.g., to implement a particular RAID (redundant array of independent disks) level);
  • communication module 268-1 that is used for facilitating communications with other devices, for example via a storage area network (SAN).
  • SAN storage area network
  • Each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above.
  • the above identified modules or programs i.e., sets of instructions
  • memory 246-1 may store a subset of the modules and data structures identified above.
  • memory 246-1 may store additional modules and data structures not described above.
  • the programs, modules, and data structures stored in memory 246-1, or the non-transitory computer readable storage medium of memory 246-1 provide instructions for implementing some of the methods described below.
  • some or all of these modules may be implemented with specialized hardware circuits that subsume part or all of the module functionality.
  • Figure 2B-1 shows system management module 151-1 in
  • Figure 2B-1 is intended more as a functional description of the various features which may be present in system management module 151-1 than as a structural schematic of the embodiments described herein.
  • the programs, modules, and data structures shown separately could be combined and some programs, modules, and data structures could be separated.
  • Figure 2B-2 is a block diagram illustrating a system management module 151-
  • management module is located in the host, as shown in Figure IB; in some such
  • the storage system is called a host-managed storage system.
  • management module 151-2 typically includes one or more processing units (sometimes called CPUs or processors) 152-2 for executing modules, programs and/or instructions stored in memory 246-2 and thereby performing processing operations, memory 246-2 (sometimes called host memory), and one or more communication buses 248-2 for interconnecting these components.
  • the one or more communication buses 248-2 optionally include circuitry (sometimes called a chipset) that interconnects and controls communications between system components.
  • System management module 151-2 is coupled to storage system controller 150 by the one or more communication buses 248-2.
  • Memory 246-2 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and may include non- volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 246-2 optionally includes one or more storage devices remotely located from the CPU(s) 152-2. Memory 246-2, or alternatively the non-volatile memory device(s) within memory 246-2, comprises a non-transitory computer readable storage medium. In some embodiments, memory 246-2, or the non-transitory computer readable storage medium of memory 246-2 stores the following programs, modules, and data structures, or a subset or superset thereof:
  • system mapping module 250-2 that is used for mapping virtual logical addresses (e.g., used by computer system 142, Figure IB) to intermediate addresses (e.g., which are mapped by storage devices 160 to physical addresses, Figure IB);
  • metrics module 252-2 that is used for generating and/or obtaining one or more metrics of a storage device (e.g., any of storage devices 160-1 to 160-m, Figure IB);
  • trigger detection module 254-2 that is used for detecting a trigger condition (e.g., in accordance with one or more metrics of a storage device);
  • enabling module 256-2 that is used for enabling an amelioration process associated with a trigger condition (e.g., detected by trigger detection module 254-2);
  • notification module 258-2 that is used for notifying a host to which a storage device is operatively coupled of a trigger condition (e.g., detected by trigger detection module 254-2) and/or of an absence of the trigger condition;
  • amelioration module 260-2 that is used for performing an amelioration process to reduce declared capacity of non-volatile memory of a storage device (e.g., storage device 160, Figure IB), optionally including: o detection module 261-2 that is used for detecting an amelioration trigger for reducing declared capacity of the non-volatile memory of the storage device; o utilization module 262-2 that is used for reducing utilization of the nonvolatile memory of the storage device; and o capacity module 264-2 that is used for reducing declared capacity of the nonvolatile memory of the storage device;
  • data redundancy module 266-2 that is used for redundantly encoding data (e.g., to implement a particular RAID (redundant array of independent disks) level) ;
  • communication module 268-2 that is used for facilitating communications with other devices, for example, via a storage area network (SAN).
  • SAN storage area network
  • Each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above.
  • the above identified modules or programs i.e., sets of instructions
  • memory 246-2 may store a subset of the modules and data structures identified above.
  • memory 246-2 may store additional modules and data structures not described above.
  • the programs, modules, and data structures stored in memory 246-2, or the non-transitory computer readable storage medium of memory 246-2 provide instructions for implementing some of the methods described below.
  • some or all of these modules may be implemented with specialized hardware circuits that subsume part or all of the module functionality.
  • Figure 2B-2 shows system management module 151-2 in
  • Figure 2B-2 is intended more as a functional description of the various features which may be present in system management module 151-2 than as a structural schematic of the embodiments described herein.
  • the programs, modules, and data structures shown separately could be combined and some programs, modules, and data structures could be separated.
  • Figure 2C-1 is a block diagram illustrating a cluster management module 181-
  • Cluster management module 181-1 typically includes one or more processing units (sometimes called CPUs or processors) 182-1 for executing modules, programs and/or instructions stored in memory 276-1 and thereby performing processing operations, memory 276-1, and one or more communication buses 278-1 for interconnecting these components.
  • the one or more communication buses 278-1 optionally include circuitry (sometimes called a chipset) that interconnects and controls communications between system components.
  • Memory 276-1 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices.
  • Memory 276-1 optionally includes one or more storage devices remotely located from the CPU(s) 182-1.
  • Memory 276-1, or alternatively the non- volatile memory device(s) within memory 276-1 comprises a non-transitory computer readable storage medium.
  • memory 276-1, or the non-transitory computer readable storage medium of memory 276-1 stores the following programs, modules, and data structures, or a subset or superset thereof:
  • cluster mapping module 280-1 that is used for mapping virtual logical addresses (e.g., used by computer system 172, Figure 1C) to intermediate addresses (e.g., which are mapped by storage subsystems 192 to physical addresses, Figure 1C);
  • metrics module 282-1 that is used for generating and/or obtaining one or more metrics of a storage device (e.g., any of the storage devices 194-1 to 194-n or 194-j to 194-k, Figure 1C);
  • trigger detection module 284-1 that is used for detecting a trigger condition (e.g., in accordance with one or more metrics of a storage device);
  • enabling module 286-1 that is used for enabling an amelioration process associated with a trigger condition (e.g., detected by trigger detection module 284-1);
  • notification module 288-1 that is used for notifying a host to which a storage device is operatively coupled of a trigger condition (e.g., detected by trigger detection module 284-1) and/or of an absence of the trigger condition;
  • amelioration module 290-1 that is used for performing an amelioration process to reduce declared capacity of non-volatile memory of a storage device (e.g., storage device 194, Figure 1C), optionally including: o detection module 291-1 that is used for detecting an amelioration trigger for reducing declared capacity of the non-volatile memory of the storage device; o utilization module 292-1 that is used for reducing utilization of the nonvolatile memory of the storage device; and o capacity module 294-1 that is used for reducing declared capacity of the nonvolatile memory of the storage device;
  • data redundancy module 296-1 that is used for redundantly encoding data
  • communication module 298-1 that is used for facilitating communications with other devices, for example, via a storage area network (SAN).
  • SAN storage area network
  • Each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above.
  • the above identified modules or programs i.e., sets of instructions
  • memory 276-1 may store a subset of the modules and data structures identified above.
  • memory 276-1 may store additional modules and data structures not described above.
  • the programs, modules, and data structures stored in memory 276-1, or the non-transitory computer readable storage medium of memory 276-1 provide instructions for implementing some of the methods described below.
  • some or all of these modules may be implemented with specialized hardware circuits that subsume part or all of the module functionality.
  • Figure 2C-1 shows cluster management module 181-1 in accordance with some embodiments
  • Figure 2C-1 is intended more as a functional description of the various features which may be present in cluster management module 181-1 than as a structural schematic of the embodiments described herein.
  • the programs, modules, and data structures shown separately could be combined and some programs, modules, and data structures could be separated.
  • Figure 2C-2 is a block diagram illustrating a cluster management module 181-
  • Cluster management module 181-2 is located, at least in part, in the host, as shown in Figure 1C; in some such embodiments the storage system uses host-based cluster management.
  • Cluster management module 181-2 typically includes one or more processing units (sometimes called CPUs or processors) 182-2 for executing modules, programs and/or instructions stored in memory 276-2 and thereby performing processing operations, memory 276-2 (sometimes called host memory), and one or more communication buses 278-2 for interconnecting these components.
  • the one or more communication buses 278-2 optionally include circuitry (sometimes called a chipset) that interconnects and controls communications between system components.
  • Cluster management module 181-2 is coupled to cluster controller 180 by the one or more communication buses 278-2.
  • Memory 276-2 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid state memory devices, and may include non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices.
  • Memory 276-2 optionally includes one or more storage devices remotely located from the CPU(s) 182-2.
  • Memory 276-2, or alternatively the non- volatile memory device(s) within memory 276-2 comprises a non-transitory computer readable storage medium.
  • memory 276-2 or the non-transitory computer readable storage medium of memory 276-2 stores the following programs, modules, and data structures, or a subset or superset thereof: • cluster mapping module 280-2 that is used for mapping virtual logical addresses (e.g., used by computer system 172, Figure 1C) to intermediate addresses (e.g., which are mapped by storage subsystems 192 to physical addresses, Figure 1C);
  • cluster mapping module 280-2 that is used for mapping virtual logical addresses (e.g., used by computer system 172, Figure 1C) to intermediate addresses (e.g., which are mapped by storage subsystems 192 to physical addresses, Figure 1C);
  • metrics module 282-2 that is used for generating and/or obtaining one or more metrics of a storage device (e.g., any of the storage devices 194-1 to 194-n or 194-j to 194-k, Figure 1C);
  • trigger detection module 284-2 that is used for detecting a trigger condition (e.g., in accordance with one or more metrics of a storage device);
  • enabling module 286-2 that is used for enabling an amelioration process associated with a trigger condition (e.g., detected by trigger detection module 284-2);
  • notification module 288-2 that is used for notifying a host to which a storage device is operatively coupled of a trigger condition (e.g., detected by trigger detection module 284-2) and/or of an absence of the trigger condition;
  • amelioration module 290-2 that is used for performing an amelioration process to reduce declared capacity of non-volatile memory of a storage device (e.g., storage device 194, Figure 1C), optionally including: o detection module 291-2 that is used for detecting an amelioration trigger for reducing declared capacity of the non-volatile memory of the storage device; o utilization module 292-2 that is used for reducing utilization of the nonvolatile memory of the storage device; and o capacity module 294-2 that is used for reducing declared capacity of the nonvolatile memory of the storage device;
  • data redundancy module 296-2 that is used for redundantly encoding data
  • communication module 298-2 that is used for facilitating communications with other devices, for example, via a storage area network (SAN).
  • SAN storage area network
  • Each of the above identified elements may be stored in one or more of the previously mentioned memory devices, and corresponds to a set of instructions for performing a function described above.
  • the above identified modules or programs i.e., sets of instructions
  • memory 276-2 may store a subset of the modules and data structures identified above.
  • memory 276-2 may store additional modules and data structures not described above.
  • the programs, modules, and data structures stored in memory 276-2, or the non-transitory computer readable storage medium of memory 276-2 provide instructions for implementing some of the methods described below.
  • some or all of these modules may be implemented with specialized hardware circuits that subsume part or all of the module functionality.
  • Figure 2C-2 shows cluster management module 181-2 in accordance with some embodiments
  • Figure 2C-2 is intended more as a functional description of the various features which may be present in cluster management module 181-2 than as a structural schematic of the embodiments described herein.
  • the programs, modules, and data structures shown separately could be combined and some programs, modules, and data structures could be separated.
  • FIG. 2D is a block diagram illustrating amelioration module 230 included in management module 121-1 of Figure 2A-1 and/or management module 121-2 of Figure 2A- 2, in accordance with some embodiments.
  • amelioration module 230 includes utilization module 232 and capacity module 234.
  • utilization module 232 includes the following programs and/or modules, or a subset or superset thereof:
  • trimming module 235 that is used for trimming from a storage device at least a
  • deleting module 236 that is used for deleting from a storage device discardable data that is used by a host
  • moving module 237 that is used for moving a portion of data that is used by a host from a storage device to another storage device.
  • capacity module 234 includes the following programs and/or modules, or a subset or superset thereof: • LB A reduction module 238 (sometimes called a logical address reduction module) that is used for reducing a range of logical addresses, reducing a count of logical addresses, and/or making specific logical addresses unavailable to a host; and
  • advertising module 239 that is used for advertising a reduced declared capacity of non-volatile memory of a storage device or storage subsystem.
  • the amelioration process includes a utilization reduction process (e.g., performed by utilization module 232) and a declared capacity reduction process (e.g., performed by capacity module 234).
  • the amelioration process has a target reduced declared capacity to be achieved by the
  • the amelioration process and utilizes the target reduced declared capacity to determine a target amount of utilization reduction to be achieved by the amelioration process.
  • the target amount of utilization reduction is zero.
  • performance of the utilization reduction process, or one or more portions of the utilization reduction process is unneeded and therefore skipped or forgone.
  • the amelioration process (e.g., periodically, semi- continuously, irregularly, initially and/or finally) recomputes or re-evaluates a number of parameters, such as the target reduced declared capacity and/or the target amount of utilization reduction, as those parameters may change in value due to the amelioration process and/or normal storage operations (e.g., read, write, erase and trim or unmap operations).
  • the utilization reduction is re-prioritized, re-scheduled, or aborted.
  • Figure 2D uses the example of amelioration module 230 included in Figures 2A-1 and 2A-2, the description of Figure 2D similarly applies to other amelioration modules (e.g., amelioration module 260-1 of Figure 2B-1, amelioration module 260-2 of Figure 2B-2, amelioration module 290-1 of Figure 2C-1, and/or amelioration module 290-2 of Figure 2C- 2), and for sake of brevity, the details are not repeated here.
  • amelioration module 260-1 of Figure 2B-1 amelioration module 260-2 of Figure 2B-2
  • amelioration module 290-1 of Figure 2C-1 and/or amelioration module 290-2 of Figure 2C- 2
  • the trim operation indicates that specific portions of the LBA space (320,
  • any data that was previously stored using the specific portion of the LBA space is no longer available to the host (e.g., is discarded).
  • the physical pages used to previously store this data may be reused for other purposes. The reuse may be done coincident in time with the trim operation or at a future time (e.g., during garbage collection). As discussed elsewhere, the reused physical pages may be used with a different encoding format or different redundancy mechanism than the encoding format or different redundancy mechanism used by those same physical pages before the reuse.
  • the trim operation is sometimes also referred to as an unmap operation.
  • the trim operation as used herein, is not necessarily identical to the trim operation of the SATA protocol.
  • the unmap operation as used herein, is not necessarily identical to the unmap operation of the SCSI protocol.
  • Figure 3 is a block diagram of a logical block address (LBA) space 320
  • a logical address is the address at which an item (e.g., a file or other data) resides from the perspective of a host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, and/or computer system 172, Figure 1C).
  • a logical address (e.g., in LBA space 320) differs from a physical address (e.g., in physical space 318) due to the operation of a mapping function or address translator (e.g., a function or module that includes translation table 212-1, Figure 2A-1, or mapping table 402, Figure 4).
  • a logical block address (LBA) is mapped to a physical flash address (e.g., a physical page number (PPN), including a bank, block, and page), as described further with respect to Figure 4.
  • PPN physical page number
  • a logical address space includes allocated logical address space (e.g., allocated LBA space 342) and unallocated logical address space (e.g., unallocated LBA space 340).
  • unallocated logical address space is logical address space at which no data is stored.
  • unallocated logical address space includes logical address space that has never been written to and/or has been discarded (previously written data may be discarded through a trim or unmap operation, and is sometimes called trimmed logical address space).
  • unallocated LBA space 340 includes trimmed LBA space 330.
  • allocated logical address space is logical address space that was previously- written by a host, the previously- written data including data that is no longer used by a host (e.g., not live data 332) and data that is still in use by the host (e.g., live data 334).
  • not live data is data in a portion of the logical address space that is marked as free, available or unused in the metadata of a file system.
  • a file system may choose to convert not live address space into unallocated address space through a trim or unmap operation.
  • allocated LBA space 342 represents an amount of allocated space
  • unallocated LBA space 340 represents an amount of unallocated space.
  • live data 334 and not live data 332 in Figure 3 represent amounts (e.g., counts of LBAs) of live data and not live data, respectively.
  • live data 334 nor not live data 332 is necessarily a single contiguous region of LBA space 320 or allocated LBA space 342, nor do the positions of live data 334 and not live data 332 illustrated in Figure 3 have any correlation to the logical or physical address values of the live data and not live data.
  • live data 334 and/or not live data 332 will be present in multiple regions of LBA space 320, and are thus non-contiguous.
  • a remapping or coalescing process which can also be called defragmentation, can be performed to consolidate some or all live data 334 into a contiguous region of LBA space 320.
  • Allocated logical address space (342) is space that is utilized.
  • the utilization reduction modules and processes discussed herein are modules, applications and processes whose purpose is to reduce the size of the allocated logical address space, and thus reduce utilization of non-volatile memory in a storage device or data storage system.
  • reducing the size of the allocated logical address space requires reducing the amount of live data 334 and/or not live data 332 stored by a storage device, or storage system, thereby converting a portion of the allocated logical address space into unallocated logical address space.
  • portions of not live data 332 are trimmed, and thereby converted into unallocated logical address space through the use of trim or unmap operations.
  • a logical address may be outside of LBA Space (320) and is therefore unavailable.
  • a previously available logical address can be made unavailable by reducing the size of the LBA space (320) such that that address is no longer within LBA space (320) and hence becomes unavailable (e.g. it is an undefined operation or erroneous operation to request a normal storage operation to a logical address that is outside of LBA space (320)).
  • LBA Space (320) can be reduced by a command to the storage device, or a host can limit its usage of logical addresses to a reduced range of logical addresses therefore effectively reducing LBA space (320).
  • the total number of allocated logical addresses (342) is limited.
  • specific logical addresses are considered to be unavailable if usage of them would cause the system to exceed the limited total number. For example, if the total number of allocated logical addresses is limited to five and the currently allocated addresses are 1, 3, 19, 45 and 273838 then any specific logical address other than these five (e.g., 6, 15, 137, etc.) would be considered unavailable.
  • Figure 4 is a block diagram of a mapping table 402 and physical address space
  • mapping table 402 is used to translate a logical block address (LBA) from the perspective of a host (e.g., computer system 110, Figure 1A) to a physical address in a physical address space (e.g., physical address space 410) of non-volatile memory in a storage device (e.g., storage device 120, Figure 1 A).
  • LBA logical block address
  • an LBA is the address of the smallest unit of stored data that is addressable by a host (e.g., 512 B or 4096 B).
  • LBAs are a sequence of integers organized in numerical order in the logical address space.
  • LBAs are integers chosen from a logical address space but need not be contiguous.
  • the amount of addressable space is governed by a limit on the number of logical addresses that can be allocated, but those logical addresses are distributed over a larger address space than the maximum number of logical addresses that can be allocated (e.g., to a host or a set of hosts or clients).
  • mapping table 402 is stored in memory associated with the storage device (e.g., in memory 206-1, as part of translation table 212-1, Figure 2A-1).
  • a physical address is a physical page number (PPN), including a bank number, a block number, and a page number.
  • PPN physical page number
  • LBA 0 is mapped to bank 1 (e.g., Bank 420-1), block 3 (e.g., Block 421-3), page 37 (pages not shown in Figure 4) of physical address space 410.
  • Figure 4 shows that physical address space 410 includes a plurality of non- volatile memory blocks 421, 422 423, 424.
  • each non- volatile memory block in the physical address space of a storage device typically includes a plurality of pages 426, where each page is typically an instance of the smallest individually accessible (e.g., readable or writable) portion of a block.
  • Figure 4 illustrates one example of a logical address to physical address mapping, in other embodiments, different mappings may be used.
  • each of the logical address entries corresponds to multiple (e.g., eight) logical addresses (e.g., 8 LBAs per logical address entry).
  • mapping table 402 need not contain contiguous LBA addresses and may be organized in any manner to facilitate lookup operations, e.g., hash table, binary tree, content addressable memory, and others.
  • a single-level flash memory cell stores one bit ("0" or "1").
  • the storage density of a SLC memory device is one bit of information per memory cell.
  • a multi-level flash memory cell can store two or more bits of information per cell by using different ranges within the total voltage range of the memory cell to represent a multi-bit bit-tuple.
  • the storage density of a MLC memory device is multiple-bits per cell (e.g., two bits per memory cell).
  • Flash memory devices utilize memory cells to store data as electrical values, such as electrical charges or voltages.
  • Each flash memory cell typically includes a single transistor with a floating gate that is used to store a charge, which modifies the threshold voltage of the transistor (i.e., the voltage needed to turn the transistor on).
  • the magnitude of the charge, and the corresponding threshold voltage is used to represent one or more data values.
  • a reading threshold voltage is applied to the control gate of the transistor and the resulting sensed current or voltage is mapped to a data value.
  • cell voltage typically means the threshold voltage of the memory cell, which is the minimum voltage that needs to be applied to the gate of the memory cell's transistor in order for the transistor to conduct current.
  • reading threshold voltages sometimes also called reading signals, reading voltages, and/or read thresholds
  • gate voltages applied to the gates of the flash memory cells to determine whether the memory cells conduct current at that gate voltage.
  • the raw data value for that read operation is a "1," and otherwise the raw data value is a "0.”
  • FIG. 5 A is a simplified, prophetic diagram of voltage distributions 300a found in a single-level flash memory cell (SLC) over time, in accordance with some embodiments.
  • the voltage distributions 300a shown in Figure 5A have been simplified for illustrative purposes.
  • the SLC's voltage range extends approximately from a voltage, Vss, at a source terminal of an NMOS transistor to a voltage, V DD , at a drain terminal of the NMOS transistor.
  • voltage distributions 300a extend between Vss and V DD -
  • Sequential voltage ranges 301 and 302 between source voltage Vss and drain voltage VDD are used to represent corresponding bit values "1" and "0,” respectively.
  • Each voltage range 301 , 302 has a respective center voltage Vi 301b, Vo 302b.
  • the memory cell current sensed in response to an applied reading threshold voltages is indicative of a memory cell voltage different from the respective center voltage Vi 301b or Vo 302b corresponding to the respective bit value written into the memory cell.
  • Errors in cell voltage, and/or the cell voltage sensed when reading the memory cell can occur during write operations, read operations, or due to "drift" of the cell voltage between the time data is written to the memory cell and the time a read operation is performed to read the data stored in the memory cell.
  • Each voltage range 301 , 302 also has a respective voltage distribution 301a, 302a that may occur as a result of any number of a combination of error-inducing factors, examples of which are identified above.
  • a reading threshold voltage VR is applied between adjacent center voltages (e.g., applied proximate to the halfway region between adjacent center voltages Vj 301b and V 0 302b).
  • the reading threshold voltage is located between voltage ranges 301 and 302.
  • reading threshold voltage VR is applied in the region proximate to where the voltage distributions 301a and 302a overlap, which is not necessarily proximate to the halfway region between adjacent center voltages Vi 301b and Vo 302b.
  • flash memory In order to increase storage density in flash memory, flash memory has developed from single-level (SLC) cell flash memory to multi-level cell (MLC) flash memory so that two or more bits can be stored by each memory cell.
  • SLC single-level
  • MLC multi-level cell
  • a MLC flash memory device is used to store multiple bits by using voltage ranges within the total voltage range of the memory cell to represent different bit-tuples.
  • a MLC flash memory device is typically more error-prone than a SLC flash memory device created using the same manufacturing process because the effective voltage difference between the voltages used to store different data values is smaller for a MLC flash memory device.
  • a typical error includes a stored voltage level in a particular MLC being in a voltage range that is adjacent to the voltage range that would otherwise be representative of the correct storage of a particular bit-tuple.
  • the impact of such errors can be reduced by gray-coding the data, such that adjacent voltage ranges represent single-bit changes between bit-tuples.
  • FIG. 5B is a simplified, prophetic diagram of voltage distributions 300b found in a multi-level flash memory cell (MLC) over time, in accordance with some embodiments.
  • the voltage distributions 300b shown in Figure 5B have been simplified for illustrative purposes.
  • the cell voltage of a MLC approximately extends from a voltage, Vss, at the source terminal of a NMOS transistor to a voltage, V DD , at the drain terminal.
  • voltage distributions 300b extend between Vss and V DD -
  • Sequential voltage ranges 311, 312, 313, 314 between the source voltage Vss and drain voltages V DD are used to represent corresponding bit-tuples "11,” “01,” “00,” “10,” respectively.
  • Each voltage range 311, 312, 313, 314 has a respective center voltage 31 lb, 312b, 313b, 314b.
  • Each voltage range 311, 312, 313, 314 also has a respective voltage distribution 31 la, 312a, 313a, 314a that may occur as a result of any number of a
  • the charge on the floating gate of the MLC would be set such that the resultant cell voltage is at the center of one of the ranges 311, 312, 313, 314 in order to write the corresponding bit-tuple to the MLC.
  • the resultant cell voltage would be set to one of Vn 31 lb, Voi 312b, Voo 313b and Vio 314b in order to write a corresponding one of the bit-tuples "11,” "01,” “00” and "10.”
  • the initial cell voltage may differ from the center voltage for the data written to the MLC.
  • Reading threshold voltages V R A, V RB and V R C are positioned between adjacent center voltages (e.g., positioned at or near the halfway point between adjacent center voltages) and, thus, define threshold voltages between the voltage ranges 311, 312, 313, 314.
  • one of the reading threshold voltages V R A, V RB and V R C is applied to determine the cell voltage using a comparison process.
  • the actual cell voltage, and/or the cell voltage received when reading the MLC may be different from the respective center voltage Vn 31 lb, Voi 312b, Voo 313b or Vw 314b corresponding to the data value written into the cell.
  • the actual cell voltage may be in an altogether different voltage range, strongly indicating that the MLC is storing a different bit-tuple than was written to the MLC. More commonly, the actual cell voltage may be close to one of the read comparison voltages, making it difficult to determine with certainty which of two adjacent bit-tuples is stored by the MLC.
  • Errors in cell voltage, and/or the cell voltage received when reading the MLC can occur during write operations, read operations, or due to "drift" of the cell voltage between the time data is written to the MLC and the time a read operation is performed to read the data stored in the MLC.
  • drift the time of the cell voltage between the time data is written to the MLC and the time a read operation is performed to read the data stored in the MLC.
  • cell voltage drift sometimes errors in cell voltage, and/or the cell voltage received when reading the MLC, are collectively called "cell voltage drift.”
  • Gray-coding the bit-tuples includes constraining the assignment of bit-tuples such that a respective bit-tuple of a particular voltage range is different from a respective bit-tuple of an adjacent voltage range by only one bit.
  • the corresponding bit-tuples for adjacent ranges 301 and 302 are respectively “11” and “01”
  • the corresponding bit-tuples for adjacent ranges 302 and 303 are respectively “01” and “00”
  • the corresponding bit-tuples for adjacent ranges 303 and 304 are respectively “00” and “10.”
  • gray-coding if the cell voltage drifts close to a read comparison voltage level, the error is typically limited to a single bit within the 2-bit bit-tuple.
  • q 2 (i.e., 2 bits per cell in a MLC flash memory)
  • q 2 (i.e., 2 bits per cell in a MLC flash memory)
  • a triple-level memory cell has eight possible states per cell, yielding three bits of information per cell.
  • a quad-level memory cell has 16 possible states per cell, yielding four bits of information per cell.
  • a cell might store only 6 states, yielding approximately 2.5 bits of information per cell, meaning that two cells together would provide 36 possible states, more than sufficient to store 5 bits of information per pair of cells.
  • Figure 6 illustrates a flowchart representation of a method 600 of managing a storage system, in accordance with some embodiments. At least in some embodiments, method 600 is performed by a storage device (e.g., storage device 120, Figure 1 A) or one or more components of the storage device (e.g., storage controller 124 and/or storage medium 130, Figure 1A), wherein the storage device is operatively coupled with a host system (e.g., computer system 110, Figure 1A).
  • a storage device e.g., storage device 120, Figure 1 A
  • a host system e.g., computer system 110, Figure 1A
  • method 600 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 122-1 of management module 121-1, shown in Figures 1A and 2A-1.
  • method 600 is performed by a storage system (e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, and/or data storage system 170, Figure 1C) or one or more components of the storage system (e.g., computer system 110 and/or storage device 120, Figure 1A, storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C).
  • a storage system e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, and/or data storage system 170, Figure 1C
  • components of the storage system e.g., computer system 110 and/or storage device 120, Figure 1A, storage system controller 150, Figure IB, and/or cluster controller 180, Figure
  • some of the operations of method 600 are performed at a host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, and/or computer system 172, Figure 1C) and information is transmitted to a storage device (e.g., storage device 120, Figure 1A) and/or one or more subsystems of a storage system (e.g., storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C).
  • a host e.g., computer system 110, Figure 1A, computer system 142, Figure IB, and/or computer system 172, Figure 1C
  • information is transmitted to a storage device (e.g., storage device 120, Figure 1A) and/or one or more subsystems of a storage system (e.g., storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C).
  • a storage device e.g., storage device 120, Figure 1A
  • one or more subsystems of a storage system e.g., storage system
  • method 600 is governed, at least in part, by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a subsystem of a storage system, such as the one or more processing units (CPUs) 152-1 of system management module 151-1, shown in Figures IB and 2B-1 or the one or more processing units (CPUs) 182-1 of cluster management module 181-1, shown in Figures 1C and 2C-1.
  • method 600 is governed, at least in part, by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a host (processors of host not shown in Figures 1A, IB, and 1C).
  • method 600 is performed by a storage device (e.g., storage device 120, Figure 1 A).
  • a storage device e.g., storage device 120, Figure 1 A
  • one or more of the operations described in method 600 are performed by a host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, and/or computer system 172, Figure 1C) and/or one or more subsystems of a storage system distinct from the storage device (e.g., storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C).
  • a storage device e.g., storage device 120, Figure 1A
  • triggers (602) a condition for reducing declared capacity of non-volatile memory of the storage device in accordance with one or more metrics of the storage device e.g., including one or more status metrics corresponding to the storage device's ability to retain data, one or more performance metrics corresponding to performance of the storage device, one or more wear metrics corresponding to wear on the storage device, and/or one or more time metrics.
  • Metrics of the storage device include metrics (e.g., wear metrics such as P/E cycle counts, write operation counts and the like) of the non-volatile storage media (e.g., storage medium 130, Figure 1 A) of the storage device, but are not necessarily limited to such metrics.
  • some metrics e.g., some performance metrics, such as latency metrics, metrics that measure how long it takes or how many operations are required to complete a write or erase operation, etc.
  • some metrics reflect both storage media performance as well as controller and/or other storage device component performance.
  • the metrics of the storage device used to determine the trigger condition include a write amplification metric of the storage device.
  • Another metric of the storage device that is used, in some embodiments, to determine the trigger condition is an over provisioning metric (e.g., percentage of total storage capacity that is in excess of the declared capacity of the storage device, and/or percentage of total storage capacity that is in excess of the declared capacity of the storage device after a projected conversion of a number of memory blocks (or other portions of the storage device) from a current encoding format (e.g., TLC, MLC and/or data redundancy mechanism) to a lower storage density encoding (e.g., MLC, SLC and/or data redundancy mechanism).
  • a current encoding format e.g., TLC, MLC and/or data redundancy mechanism
  • a lower storage density encoding e.g., MLC, SLC and/or data redundancy mechanism
  • a trigger condition is determined if a projected over-provisioning metric, corresponding to a number of blocks (or other portions) of the storage device removed from service (e.g., that have been or will be removed from service) due to wear or due to failure of those blocks (or other portions) to meet a predefined quality of service metric, falls below a predefined threshold, or falls below a threshold determined in accordance with a measured or projected write amplification of the storage device.
  • Write amplification is a phenomenon where the actual amount of physical data written to a storage medium (e.g., storage medium 130 in storage device 120) is a multiple of the logical amount of data written by a host (e.g., computer system 110, sometimes called a host) to the storage medium.
  • a host e.g., computer system 110, sometimes called a host
  • the garbage collection process to perform these operations results in re -writing data one or more times. This multiplying effect increases the number of writes required over the life of a storage medium, which shortens the time it can reliably operate.
  • the formula to calculate the write amplification of a storage system is given by equation:
  • One of the goals of any flash memory based data storage system architecture is to reduce write amplification as much as possible so that available endurance is used to meet storage medium reliability and warranty specifications. Higher system endurance also results in lower cost as the storage system may need less over-provisioning. By reducing write amplification, the endurance of the storage medium is increased and the overall cost of the storage system is decreased. Generally, garbage collection is performed on erase blocks with the fewest number of valid pages for best performance and best write amplification.
  • the trigger condition is detected in accordance with a non-linear and/or linear combination of the one or more metrics.
  • the trigger condition is detected by comparing a wear metric such as P/E cycle counts to a previously determined value, e.g., a threshold value.
  • the trigger condition can also be asserted by other means, e.g., by a human operator or scheduled by a human operator.
  • it may be desirable to initiate the amelioration process because of the expected availability or unavailability of resources.
  • it may be desirable to initiate the amelioration process because performance characteristics of the storage device (including reliability) are altered.
  • the trigger condition is detected in accordance with historical knowledge of the one or more metrics.
  • historical knowledge can be a running average of one or more metrics.
  • historical knowledge can be used to determine (e.g., compute) one or more projected values of one or more metrics at a particular time in the future (e.g., an hour, day, week, or month in the future), and the trigger condition can be detected in accordance with the one or more projected values.
  • the trigger condition is detected by comparing a historical wear metric such as P/E cycle counts to a previously determined value to anticipate wear out of a portion of the storage media.
  • the trigger condition is detected by comparing a historical metric, such as the bit error rate (BER), or the rate of change of the metric, BER (of the storage media, or a portion of the storage media), or a projected value (e.g., a projected BER rate at a particular time in the future, as determined based on a current or historical BER and a rate of change of the BER), against a previously determined value to anticipate performance degradation due to increased computation requirements of error correction.
  • a historical metric such as the bit error rate (BER), or the rate of change of the metric, BER (of the storage media, or a portion of the storage media), or a projected value (e.g., a projected BER rate at a particular time in the future, as determined based on a current or historical BER and a rate of change of the BER)
  • the trigger condition may be dependent on metrics obtained from a plurality of the storage devices.
  • the amelioration process may operate on more than one storage device at a time, either sequentially or in parallel.
  • a storage system may have a fixed maximum rate of capacity reduction independent of how many storage devices are currently being operated on in parallel by the amelioration process (e.g., maximum rate of data movement between the storage devices while reducing utilization).
  • the trigger condition should include considering, separately and in combination, the metrics of the plurality of storage devices when
  • the storage device notifies (604) a host (e.g., computer system 110, Figure
  • the storage device or a host detects the amelioration trigger and, in
  • the amelioration process includes a process to reduce utilization (608), a process to reduce declared capacity (610), and/or a process to advertise (612) a reduced declared capacity.
  • the amelioration process (606) includes a utilization reduction process (608) (e.g., performed by utilization module 232, Figures 2A-1 and 2A-2, utilization module 262, Figures 2B-1 and 2B-
  • Figures 2C-1 and 2C-2) and/or a declared capacity reduction process (610) e.g., performed by capacity module 234, Figures 2A-1 and 2A-2, capacity module 264, Figures 2B-1 and 2B-2, or capacity module 294, Figures 2C-1 and 2C-2).
  • a declared capacity reduction process e.g., performed by capacity module 234, Figures 2A-1 and 2A-2, capacity module 264, Figures 2B-1 and 2B-2, or capacity module 294, Figures 2C-1 and 2C-2).
  • Figures 7A-7D illustrate a flowchart representation of a method 700 of managing a storage system, in accordance with some embodiments.
  • method 700 is performed by a storage device (e.g., storage device 120, Figure 1A, storage device 160, Figure IB, or storage device 194, Figure 1C) or one or more components of the storage device (e.g., storage controller 124, Figure 1 A), wherein the storage device is operatively coupled with a host system (e.g., computer system 1 10, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, or cluster controller 180, Figure 1C).
  • a host system e.g., computer system 1 10, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, or cluster controller 180, Figure 1C.
  • method 700 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 122-1 of management module 121-1, shown in Figures 1A and 2A-1.
  • method 700 is performed by a storage system (e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, and/or data storage system 170, Figure 1C) or one or more components of the storage system (e.g., storage device 120, Figure 1A, storage device 160, Figure IB, or storage device 194, Figure 1C).
  • some of the operations of method 700 are performed at a storage device (e.g., storage device 120, Figure 1A, storage device 160, Figure IB, or storage device 194, Figure 1C) and information is transmitted to a host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, or cluster controller 180, Figure 1C).
  • a storage device e.g., storage device 120, Figure 1A
  • a storage system e.g., data storage system 100, Figure 1A.
  • one or more of the operations described in method 700 are performed by a storage device of another storage system (e.g., storage device 160 of data storage system 140, Figure IB, or storage device 194 of data storage system 170, Figure 1C).
  • the storage device e.g., storage device 120, Figure 1 A
  • the storage device generates (704) one or more metrics of the storage device, the storage device including non- volatile memory.
  • a metrics module e.g., metrics module 222-1, Figure 2A-1) is used to generate one or more metrics of the storage device, the storage device including non-volatile memory, as described above with respect to Figure 2A-1.
  • the storage device comprises (712) one or more flash memory devices.
  • the storage device comprises a storage medium (e.g., storage medium 130, Figure 1A), and the storage medium comprises one or more nonvolatile storage devices, such as flash memory devices.
  • the storage medium e.g., storage medium 130, Figure 1A
  • the storage medium is a single flash memory device, while in other embodiments the storage medium includes a plurality of flash memory devices.
  • the storage medium includes dozens or hundreds of flash memory devices, organized in parallel memory channels, such as 16, 32 or 64 flash memory devices per memory channel, and 8, 16 or 32 parallel memory channels.
  • the non- volatile storage medium (e.g., storage medium 130, Figure 1A) includes NAND-type flash memory or NOR-type flash memory. In other embodiments, the storage medium comprises one or more other types of non-volatile storage devices.
  • generating (714) one or more metrics of the storage device includes generating at least one metric, of the one or more metrics, for each memory portion of a plurality of memory portions of the storage device. In some embodiments, at least one metric is generated for each block of a plurality of blocks of the storage device. In some embodiments, at least one metric is generated for each page of a plurality of pages of the storage device. In some embodiments, at least one metric is generated for each region of a plurality of regions of the storage device. In some embodiments, some metrics are generated on a block basis, some metrics are generated on a page basis, some metrics are generated on a region basis, and/or some metrics are generated on a storage device basis.
  • the one or more metrics of the storage device include
  • storage controller 124 ( Figure 1A) or a component thereof (e.g., metrics module 222-1,
  • Figure 2A-1 generates and/or maintains one or more status metrics for each memory portion of a plurality of memory portions (e.g., in storage medium 130, Figure 1A) of the storage device.
  • the one or more status metrics indicate a respective memory portion's ability to retain data.
  • the one or more status metrics associated with a respective memory portion, of a plurality of memory portions of the storage device are stored in a characterization vector corresponding to the respective memory portion.
  • the one or more status metrics stored in the characterization vector for the respective memory portion include a subset or superset of: (a) a bytes written field indicating a number of bytes of data written to pages in the respective memory portion, (b) a program-erase (P/E) cycle field indicating a current count of the number of P/E cycles performed on the respective memory portion, (c) a bit error rate (BER) field indicating a number of errors detected in a codeword read from pages of the respective memory portion, and (d) other usage information indicating the health, performance, and/or endurance of the respective memory portion, as it relates to the respective memory portion's ability to retain data.
  • P/E program-erase
  • BER bit error rate
  • the one or more status metrics indicate the storage device's ability, as a whole, to retain data. For example, as the storage device ages, the one or more status metrics reflect the storage device's diminished ability to retain data (e.g., data read from the storage device typically have more errors as the storage device ages).
  • the one or more metrics of the storage device include (718) one or more performance metrics corresponding to performance of the storage device.
  • storage controller 124 Figure 1A or a component thereof (e.g., metrics module 222-1, Figure 2A-1) generates and/or maintains one or more performance metrics for each memory portion of a plurality of memory portions (e.g., in storage medium 130, Figure 1A) of the storage device.
  • the one or more performance metrics correspond to performance of a respective memory portion of the plurality of memory portions.
  • the one or more performance metrics associated with a respective memory portion, of a plurality of memory portions of the storage device are stored in a characterization vector corresponding to the respective memory portion.
  • the one or more performance metrics stored in the characterization vector for the respective memory portion include a subset or superset of: (a) a measure of latency, and (b) transaction time.
  • the one or more performance metrics correspond to performance of the storage device as a whole.
  • the one or more performance metrics include a measure of latency for the storage device and/or transaction time for the storage device. [00481]
  • the one or more metrics of the storage device include
  • storage controller 124 ( Figure 1 A) or a component thereof (e.g., metrics module 222-1, Figure 2A-1) generates and/or maintains one or more wear metrics for each memory portion of a plurality of memory portions (e.g., in storage medium 130, Figure 1A) of the storage device.
  • the one or more wear metrics correspond to wear on a respective memory portion of the plurality of memory portions.
  • the one or more wear metrics associated with a respective memory portion, of a plurality of memory portions of the storage device are stored in a characterization vector corresponding to the respective memory portion.
  • the one or more wear metrics stored in the characterization vector for the respective memory portion include a subset or superset of: (a) a count of cumulative writes to the respective memory portion, (b) a count of cumulative reads from the respective memory portion, (c) a count of P/E cycles performed on the respective memory portion, and (d) a BER for the respective memory portion.
  • the one or more wear metrics correspond to wear on the storage device as a whole.
  • the one or more wear metrics include a count of cumulative writes to the storage device (e.g., to the storage device's storage medium or storage media), a count of cumulative reads from the storage device (e.g., from the storage device's storage medium or storage media), a count of P/E cycles performed on the storage device and/or a BER for the storage device.
  • the one or more metrics of the storage device include
  • storage controller 124 ( Figure 1 A) or a component thereof (e.g., metrics module 222-1, Figure 2A-1) generates and/or maintains one or more time metrics.
  • the one or more time metrics include a wall-clock time.
  • the one or more metrics of the storage device include (724) values of the one or more metrics from more than one time.
  • the one or more metrics of the storage device include a count of cumulative writes to the storage device (e.g., to the storage device's storage medium or storage media) at a first time and a count of cumulative writes to the storage device (e.g., to the storage device's storage medium or storage media) at a second time.
  • values of the one or more metrics from more than one time include historical knowledge of the one or more metrics.
  • the one or more metrics from more than one time include a running average of the one or more metrics.
  • historical knowledge can be used to determine (e.g., compute) one or more projected values of one or more metrics at a particular time in the future (e.g., an hour, day, week, or month in the future).
  • historical knowledge of the one or more metrics of the storage device is used to detect a trigger condition, as described below with respect to operation 706.
  • the storage device detects (706) a trigger condition in accordance with the one or more metrics of the storage device.
  • the trigger condition is detected in accordance with a non-linear and/or linear combination of the one or more metrics.
  • the trigger condition is detected in accordance with historical knowledge of the one or more metrics, as described above with respect to operation 602 of Figure 6.
  • a trigger detection module e.g., trigger detection module 224-1, Figure 2A-1 is used to detect a trigger condition in accordance with the one or more metrics of the storage device, as described above with respect to Figure 2A-1.
  • the storage device e.g., storage device 120, Figure 1 A
  • the storage device enables (708) an amelioration process associated with the detected trigger condition, the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • the amelioration process includes a process to reduce utilization by a host, a process to reduce declared capacity of the nonvolatile memory of the storage device, and/or a process to advertise a reduced declared capacity.
  • the amelioration process includes altering an encoding format (e.g., from TLC to SLC and/or changing the redundancy mechanism) of at least a portion of the non-volatile memory of the storage device.
  • altering the encoding format of at least a portion of the non-volatile memory of the storage device includes setting the encoding format of an entirety of the non-volatile memory of the storage device to a low-density physical encoding format, for example SLC.
  • SLC low-density physical encoding format
  • the storage device prior to the amelioration process the storage device includes some blocks (e.g., 98%) encoded as TLC and other blocks (e.g., 2%) encoded as SLC, and after the amelioration process all blocks are encoded using the lower-density physical encoding format, SLC.
  • the latter example may correspond to a storage device that initially stores all client data using TLC and all storage device metadata using SLC.
  • the amelioration process converts all of the client data from TLC to SLC without changing the encoding format of the storage device metadata (SLC).
  • an enabling module e.g., enabling module 226-1, Figure 2A-1 is used to enable an amelioration process associated with the detected trigger condition, the amelioration process to reduce declared capacity of the non-volatile memory of the storage device, as described above with respect to Figure 2A-1.
  • the amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes a process to reduce utilization of the non-volatile memory of the storage device, for example as described above with respect to operation 608 of Figure 6.
  • enabling the amelioration process associated with the detected trigger condition includes (726) notifying a host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C) to which the storage device is operatively coupled of the trigger condition.
  • notifying the host of the trigger condition includes notifying the host with an unsolicited communication.
  • the unsolicited communication includes an interrupt communication.
  • the unsolicited communication includes a remote direct memory access (RDMA).
  • the unsolicited communication includes a TCP connection request or a TCP data transmission.
  • the unsolicited communication includes any other form of unsolicited communication.
  • the host includes (728) a client on behalf of which data is stored in the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; data storage system 170, Figure 1C).
  • the client is or includes an entity on behalf of which data is stored in the storage system.
  • the host is (1) computer system 110 ( Figure 1A) or a client process, module or application executed by computer system 110, (2) computer system 142 ( Figure IB) or a client process, module or application executed by computer system 142, and/or (3) computer system 172 ( Figure 1C) or a client process, module or application executed by computer system 172.
  • the host includes (730) a storage system controller of the storage system (e.g., data storage system 140, Figure IB).
  • the storage system controller controls and/or coordinates operations among one or more storage devices.
  • the host is storage system controller 150 ( Figure IB).
  • the data storage system e.g., data storage system 140, Figure IB
  • the data storage system is called a scale-up system.
  • the host includes (732) a cluster controller of the storage system (e.g., data storage system 170, Figure IC).
  • the cluster controller controls and/or coordinates operations among one or more data storage subsystems, as shown for example in Figure IC, where each of the data storage subsystems may be implemented as a data storage system having one or more storage devices (e.g., data storage system 140, Figure IB).
  • the host is cluster controller 180 ( Figure IC).
  • the data storage system e.g., data storage system 170, Figure IC
  • the data storage system is called a scale -out system or sometimes known as a clustered storage system.
  • enabling the amelioration process associated with the detected trigger condition includes (734): (1) receiving a query from a host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure IC, storage system controller 150, Figure IB, and/or cluster controller 180, Figure IC) to which the storage device is operatively coupled, and (2) in response to receiving the query, reporting the trigger condition.
  • a host e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure IC, storage system controller 150, Figure IB, and/or cluster controller 180, Figure IC
  • the host polls for the trigger condition and the storage device receives the query from the host and in response to receiving the query, reports the trigger condition.
  • enabling the amelioration process associated with the detected trigger condition includes (736): (1) receiving a command from a host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172,
  • a host e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172,
  • Figure IC storage system controller 150, Figure IB, and/or cluster controller 180, Figure IC) to which the storage device is operatively coupled, and (2) in response to receiving the command, sending a response to the command and a notification of the trigger condition.
  • the command includes an I/O (input/output) request.
  • the I/O request includes a read request from the storage device and/or a write request to the storage device.
  • the command includes a request for temperature of the storage device.
  • the command includes a request for some other status of the storage device.
  • the notification of the trigger condition is piggy-backed on a response to the command from the host. For example, in some embodiments, the host issues a read request for data from the storage device, and the storage device (1) receives the read request from the host, and (2) in response to receiving the read request, the storage device sends data corresponding to the read request and a notification of the trigger condition.
  • enabling the amelioration process associated with the detected trigger condition includes (738): (1) receiving a command from a host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure IC, storage system controller 150, Figure IB, and/or cluster controller 180, Figure IC) to which the storage device is operatively coupled, and (2) in response to receiving the command, sending a response to the command and a notification that prompts the host to obtain information with respect to the trigger condition.
  • the command includes an I/O (input/output) request.
  • the I/O request includes a read request from the storage device and/or a write request to the storage device.
  • the command includes a request for temperature of the storage device. In some embodiments, the command includes a request for some other status of the storage device. In some embodiments, the notification that prompts the host to obtain information with respect to the trigger condition is piggy-backed on a response to the command from the host. For example, in some embodiments, the host issues a read request for data from the storage device, and the storage device (1) receives the read request from the host, and (2) in response to receiving the read request, the storage device sends data corresponding to the read request and a notification (e.g., by setting a notification bit) that prompts the host to obtain information with respect to the trigger condition. In some embodiments, the mechanism used for returning such a notification when responding to a command from the host is a SCSI deferred error or deferred error response code.
  • enabling the amelioration process associated with the detected trigger condition includes (740) scheduling the amelioration process to be performed on the storage device.
  • the trigger condition feeds back to the storage device and the storage device enables the amelioration process by scheduling the amelioration process to be performed on the storage device.
  • the amelioration process is performed, at least in part, by an apparatus other than the storage device (e.g., performed at least in part by the host, or by a storage system controller or by a cluster controller of a data storage system that includes at least one storage device distinct from the storage device).
  • enabling the amelioration process associated with the detected trigger condition includes (742) determining one or more parameters for the amelioration process.
  • the one or more parameters for the amelioration process include a level of urgency for the amelioration process, a target reduced declared capacity of the non-volatile memory of the storage device, and/or a target amount of reduction in utilization of the non-volatile memory of the storage device, or any combination or subset thereof.
  • the one or more parameters for the amelioration process include a parameter indicating that the urgency level is high (e.g., the amelioration process needs to begin within the next hour) and a parameter indicating that at least 1 GB of storage capacity needs to be reduced in the storage device.
  • enabling the amelioration process associated with the detected trigger condition further includes (744) reporting at least a subset of the one or more parameters for the amelioration process. For example, in some embodiments, enabling the amelioration process associated with the detected trigger condition further includes reporting a target reduction in storage capacity of the non-volatile memory in the storage device.
  • the storage device (1) re-evaluates the trigger condition in accordance with the one or more metrics of the storage device, and (2) in accordance with a determination that the trigger condition is no longer valid, aborts the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • the one or more metrics of the storage device may change such that the trigger condition is no longer valid (e.g., the amelioration process is no longer needed).
  • normal storage operations will continue to be performed (e.g., read, write, delete, trim, etc.). Normal storage operations include operations like trim that explicitly reduce the storage device utilization, possibly enough to merit aborting the amelioration process. Other storage activity such as garbage collection may also reduce utilization, possibly enough to merit aborting the amelioration process.
  • Other storage activity such as garbage collection may also reduce utilization, possibly enough to merit aborting the amelioration process.
  • the trigger condition is (e.g., periodically, semi-continuously, irregularly, initially, finally, etc.) re-evaluated in accordance with the one or more metrics of the storage device, as the one or more metrics may change in value due to the amelioration process and/or normal storage operations (e.g., read, write, erase and trim or unmap operations).
  • a trigger detection module e.g., trigger detection module 224-1, Figure 2A-1
  • an enabling module e.g., enabling module 226-1, Figure 2A-1 are used to, after enabling the amelioration process, (1) re-evaluate the trigger condition in accordance with the one or more metrics of the storage device, and (2) in accordance with a
  • any operations of method 700 described above are performed by a storage device, the storage device including (1) non-volatile memory (e.g., comprising one or more non-volatile storage devices, such as flash memory devices), (2) one or more processors, and (3) controller memory (e.g., non-volatile memory or volatile memory in or coupled to the controller) storing one or more programs, which when executed by the one or more processors cause the storage device to perform or control performance of any of the methods described herein.
  • non-volatile memory e.g., comprising one or more non-volatile storage devices, such as flash memory devices
  • controller memory e.g., non-volatile memory or volatile memory in or coupled to the controller
  • any operations of method 700 described above are performed by a storage device including means for performing any of the methods described herein.
  • any operations of method 700 described above are performed by a storage system comprising (1) a storage medium (e.g., comprising one or more non- volatile storage devices, such as flash memory devices) (2) one or more processors, and (3) memory (e.g., non-volatile memory or volatile memory in the storage system) storing one or more programs, which when executed by the one or more processors cause the storage system to perform or control performance of any of the methods described herein.
  • a storage medium e.g., comprising one or more non- volatile storage devices, such as flash memory devices
  • processors e.g., non-volatile memory or volatile memory in the storage system
  • memory e.g., non-volatile memory or volatile memory in the storage system
  • FIGS 8A-8D illustrate a flowchart representation of a method 800 of managing a storage system, in accordance with some embodiments.
  • method 800 is performed by a host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, or cluster controller 180, Figure 1C) or one or more components of the host (e.g., a host, e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, or cluster controller 180, Figure 1C) or one or more components of the host (e.g.,
  • a host e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, or cluster controller 180, Figure 1C
  • a host e.g., computer system 110, Figure 1A, computer system 142, Figure
  • method 800 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a host, such as the one or more processing units (CPUs) 152-1 of system management module 151-1, shown in Figures IB and 2B-1, or the one or more processing units (CPUs) 182-1 of cluster management module 181-1, shown in Figures 1C and 2C-1.
  • processors such as the one or more processing units (CPUs) 152-1 of system management module 151-1, shown in Figures IB and 2B-1, or the one or more processing units (CPUs) 182-1 of cluster management module 181-1, shown in Figures 1C and 2C-1.
  • method 800 is performed by a storage system (e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, or data storage system 170, Figure 1C) or one or more components of the storage system (e.g., computer system 110, Figure 1A, storage system controller 150, Figure IB, or cluster controller 180, Figure 1C).
  • a storage system e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, or data storage system 170, Figure 1C
  • components of the storage system e.g., computer system 110, Figure 1A, storage system controller 150, Figure IB, or cluster controller 180, Figure 1C.
  • some of the operations of method 800 are performed at a host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, or computer system 172, Figure 1C) and information is transmitted to a storage device (e.g., storage device 120, Figure 1 A) and/or one or more subsystems of a storage system (e.g., storage system controller 150, Figure IB, or cluster controller 180, Figure 1C).
  • a host e.g., computer system 110, Figure 1A, computer system 142, Figure IB, or computer system 172, Figure 1C
  • information is transmitted to a storage device (e.g., storage device 120, Figure 1 A) and/or one or more subsystems of a storage system (e.g., storage system controller 150, Figure IB, or cluster controller 180, Figure 1C).
  • a storage device e.g., storage device 120, Figure 1 A
  • one or more subsystems of a storage system e.g., storage system controller 150, Figure IB, or
  • method 800 is governed, at least in part, by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a host (processors of host not shown in Figures 1A, IB, and 1C).
  • the host includes (812) a client on behalf of which data is stored in the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; data storage system 170, Figure 1C).
  • the client is or includes an entity on behalf of which data is stored in the storage system.
  • the host is computer system 110 ( Figure 1A) or a client process or application executed by computer system 110, or is a computer system 142 ( Figure IB) or a client process or application executed by computer system 142, or computer system 172 ( Figure 1C) or a client process or application executed by computer system 172.
  • the host includes (814) a storage system controller of the storage system (e.g., data storage system 140, Figure IB).
  • the storage system controller controls and/or coordinates operations among one or more storage devices.
  • the host is storage system controller 150 ( Figure IB).
  • the data storage system e.g., data storage system 140, Figure IB
  • the data storage system is called a scale-up system.
  • the host includes (816) a cluster controller of the storage system (e.g., data storage system 170, Figure 1C).
  • the cluster controller controls and/or coordinates operations among one or more data storage subsystems, as shown for example in Figure 1C, where each of the data storage subsystems may be implemented as a data storage system having one or more storage devices (e.g., data storage system 140, Figure IB).
  • the host is cluster controller 180 ( Figure 1C).
  • the data storage system e.g., data storage system 170, Figure 1C
  • the data storage system is called a scale-out system or a clustered storage system.
  • the host obtains (804) one or more metrics of the storage device (e.g., storage device 120, Figure 1A, a particular storage device 160, Figure IB, or a particular storage device 194, Figure 1C), the storage device including non- volatile memory.
  • the storage device generates and/or maintains the one or more metrics of the storage device, and the host obtains the one or more metrics from the storage device.
  • one or more subsystems of a storage system distinct from the storage device generate and/or maintain the one or more metrics of the storage device, and the host obtains the one or more metrics from the one or more subsystems.
  • the host generates and/or maintains the one or more metrics of the storage device.
  • a metrics module e.g., metrics module 222-2, Figure 2A-2 is used to obtain one or more metrics of the storage device, the storage device including non-volatile memory, as described above with respect to Figure 2A-2.
  • the storage device comprises (818) one or more flash memory devices.
  • the storage device comprises a storage medium (e.g., storage medium 130, Figure 1A), and the storage medium comprises one or more nonvolatile storage devices, such as flash memory devices.
  • the storage medium e.g., storage medium 130, Figure 1A
  • the storage medium is a single flash memory device, while in other embodiments the storage medium includes a plurality of flash memory devices.
  • the storage medium includes dozens or hundreds of flash memory devices, organized in parallel memory channels, such as 16, 32 or 64 flash memory devices per memory channel, and 8, 16 or 32 parallel memory channels.
  • the non- volatile storage medium (e.g., storage medium 130, Figure 1A) includes NAND-type flash memory or NOR-type flash memory. In other embodiments, the storage medium comprises one or more other types of non-volatile storage devices.
  • obtaining (804) one or more metrics of the storage device includes obtaining (820) at least one metric, of the one or more metrics, for each memory portion (e.g., block, page, region, etc.) of a plurality of memory portions of the storage device.
  • at least one metric is generated for each block of a plurality of blocks of the storage device.
  • at least one metric is generated for each page of a plurality of pages of the storage device.
  • at least one metric is generated for each region of a plurality of regions of the storage device.
  • some metrics are generated on a block basis, some metrics are generated on a page basis, some metrics are generated on a region basis, and/or some metrics are generated on a storage device basis.
  • the one or more metrics of the storage device include (822) one or more status metrics corresponding to the storage device's ability to retain data.
  • the host or a component thereof e.g., metrics module 222-2, Figure 2A-2
  • the host or a component thereof generates and/or maintains one or more status metrics for each memory portion of a plurality of memory portions (e.g., in storage medium 130, Figure 1A) of the storage device.
  • the one or more status metrics indicate a respective memory portion's ability to retain data.
  • the one or more status metrics associated with a respective memory portion, of a plurality of memory portions of the storage device are stored in a characterization vector corresponding to the respective memory portion.
  • the one or more status metrics stored in the characterization vector for the respective memory portion include a subset or superset of: (a) a bytes written field indicating a number of bytes of data written to pages in the respective memory portion, (b) a program- erase (P/E) cycle field indicating a current count of the number of P/E cycles performed on the respective memory portion, (c) a bit error rate (BER) field indicating a number of errors included in a codeword read from pages of the respective memory portion, and (d) other usage information indicating the health, performance, and/or endurance of the respective memory portion, as it relates to the respective memory portion's ability to retain data.
  • P/E program- erase
  • BER bit error rate
  • the one or more status metrics indicate the storage device's ability, as a whole, to retain data. For example, as the storage device ages, the one or more status metrics reflect the storage device's diminished ability to retain data (e.g., data read from the storage device includes more errors as the storage device ages).
  • the one or more metrics of the storage device include
  • the host or a component thereof e.g., metrics module 222-2, Figure 2A-2
  • the host or a component thereof generates and/or maintains one or more performance metrics for each memory portion of a plurality of memory portions (e.g., in storage medium 130, Figure 1A) of the storage device.
  • the one or more performance metrics correspond to performance of a respective memory portion of the plurality of memory portions.
  • the one or more performance metrics associated with a respective memory portion, of a plurality of memory portions of the storage device are stored in a
  • the one or more performance metrics stored in the characterization vector for the respective memory portion include a subset or superset of: (a) a measure of latency, and (b) transaction time.
  • the one or more performance metrics correspond to performance of the storage device as a whole.
  • the one or more performance metrics include a measure of latency for the storage device and/or transaction time for the storage device.
  • the one or more metrics of the storage device include
  • one or more wear metrics corresponding to wear on the storage device (826) one or more wear metrics corresponding to wear on the storage device.
  • the host or a component thereof e.g., metrics module 222-2, Figure 2A-2
  • the host or a component thereof generates and/or maintains one or more wear metrics for each memory portion of a plurality of memory portions (e.g., in storage medium 130, Figure 1A) of the storage device.
  • the one or more wear metrics correspond to wear on a respective memory portion of the plurality of memory portions.
  • the one or more wear metrics associated with a respective memory portion, of a plurality of memory portions of the storage device are stored in a characterization vector corresponding to the respective memory portion.
  • the one or more wear metrics stored in the characterization vector for the respective memory portion include a subset or superset of: (a) a count of cumulative writes to the respective memory portion, (b) a count of cumulative reads from the respective memory portion, (c) a count of P/E cycles performed on the respective memory portion, and (d) a BER for the respective memory portion.
  • the one or more wear metrics correspond to wear on the storage device as a whole.
  • the one or more wear metrics include a count of cumulative writes to the storage device, a count of cumulative reads from the storage device, a count of P/E cycles performed on the storage device and/or a BER for the storage device.
  • the one or more metrics of the storage device include
  • the host or a component thereof e.g., metrics module 222-2, Figure 2A-2
  • the one or more time metrics include a wall-clock time.
  • the one or more metrics of the storage device include (830) values of the one or more metrics from more than one time.
  • the one or more metrics of the storage device include a count of cumulative writes to the storage device at a first time and a count of cumulative writes to the storage device at a second time.
  • values of the one or more metrics from more than one time include historical knowledge of the one or more metrics.
  • the one or more metrics from more than one time include a running average of the one or more metrics.
  • historical knowledge can be used to determine (e.g., compute) one or more projected values of one or more metrics at a particular time in the future (e.g., an hour, day, week, or month in the future).
  • historical knowledge of the one or more metrics of the storage device is used to detect a trigger condition, as described below with respect to operation 806.
  • the host detects (806) a trigger condition in accordance with the one or more metrics of the storage device.
  • the trigger condition is detected in accordance with a non-linear and/or linear combination of the one or more metrics.
  • the trigger condition is detected in accordance with historical knowledge of the one or more metrics.
  • a trigger detection module e.g., trigger detection module 224-2, Figure 2A-2 is used to detect a trigger condition in accordance with the one or more metrics of the storage device, as described above with respect to Figure 2A- 2.
  • the host enables (808) an amelioration process associated with the detected trigger condition, the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • the amelioration process includes a process to reduce utilization by a host, a process to reduce declared capacity of the nonvolatile memory of the storage device, and/or a process to advertise a reduced declared capacity.
  • the amelioration process includes altering an encoding format (e.g., from TLC to SLC and/or changing the redundancy mechanism) of at least a portion of the non-volatile memory of the storage device.
  • altering the encoding format of at least a portion of the non-volatile memory of the storage device includes setting the encoding format of an entirety of the non-volatile memory of the storage device to a low-density physical encoding format, for example SLC.
  • SLC low-density physical encoding format
  • the storage device prior to the amelioration process the storage device includes some blocks (e.g., 98%) encoded as TLC and other blocks (e.g., 2%) encoded as SLC, and after the amelioration process all blocks are encoded using the lower-density physical encoding format, SLC.
  • the latter example may correspond to a storage device that initially stores all client data using TLC and all storage device metadata using SLC.
  • the amelioration process converts all of the client data from TLC to SLC without changing the encoding format of the storage device metadata (SLC).
  • an enabling module e.g., enabling module 226-2, Figure 2A-2 is used to enable an amelioration process associated with the detected trigger condition, the amelioration process to reduce declared capacity of the non-volatile memory of the storage device, as described above with respect to Figure 2A-2.
  • the amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes a process to reduce utilization of the non-volatile memory of the storage device, for example as described above with respect to operation 608 of Figure 6.
  • enabling (808) the amelioration process associated with the detected trigger condition includes scheduling (832) the amelioration process to be performed on the storage device.
  • the trigger condition feeds back to the host and the host enables the amelioration process by scheduling the amelioration process to be performed on the storage device.
  • enabling (808) the amelioration process associated with the detected trigger condition includes determining (834) one or more parameters for the amelioration process.
  • the one or more parameters for the amelioration process include a level of urgency for the amelioration process, a target reduced declared capacity of the non-volatile memory of the storage device, and/or a target amount of reduction in utilization of the non-volatile memory of the storage device, or any combination or subset thereof.
  • the one or more parameters for the amelioration process include a parameter indicating that the urgency level is high (e.g., the amelioration process needs to begin within the next hour) and a parameter indicating that at least 1 GB of storage capacity needs to be reduced in the storage device.
  • enabling (808) the amelioration process associated with the detected trigger condition further includes conveying (836) at least a subset of the one or more parameters for the amelioration process to the storage device.
  • enabling the amelioration process associated with the detected trigger condition further includes conveying to the storage device a target amount of storage capacity of the non-volatile memory that needs to be reduced in the storage device.
  • enabling the amelioration process associated with the detected trigger condition further includes conveying to the storage device which portions of the storage medium (e.g., storage medium 130, Figure 1A) to reformat by altering an encoding format.
  • the host re-evaluates the trigger condition in accordance with the one or more metrics of the storage device, and (2) in accordance with a determination that the trigger condition is no longer valid, aborts the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • the one or more metrics of the storage device may change such that the trigger condition is no longer valid (e.g., the amelioration process is no longer needed).
  • normal storage operations will continue to be performed (e.g., read, write, delete, trim, etc.).
  • Normal storage operations include operations like trim that explicitly reduce the storage device utilization, possibly enough to merit aborting the amelioration process.
  • Other storage activity such as garbage collection may also reduce utilization, possibly enough to merit aborting the amelioration process.
  • the trigger condition is (e.g., periodically, semi-continuously, irregularly, initially, finally, etc.) re-evaluated in accordance with the one or more metrics of the storage device, as the one or more metrics may change in value due to the amelioration process and/or normal storage operations (e.g., read, write, erase and trim or unmap operations).
  • a trigger detection module e.g., trigger detection module 224-2, Figure 2A-2
  • an enabling module e.g., enabling module 226-2, Figure 2A-2
  • any of the methods described above are performed by a storage system, the storage system including (1) one or more storage devices (e.g., comprising one or more non-volatile storage devices, such as flash memory devices), (2) a host to which the one or more storage devices are operatively coupled, (3) one or more processors, and (4) controller memory storing one or more programs, which when executed by the one or more processors cause the host to perform or control performance of any of the methods described herein.
  • one or more storage devices e.g., comprising one or more non-volatile storage devices, such as flash memory devices
  • any of the methods described above are performed by a host system, coupled to one or more storage devices, the host system including means for performing any of the methods described herein.
  • any operations of method 800 described above are performed by a host system comprising (1) an interface for operatively coupling to a storage system, (2) one or more processors, and (3) controller memory storing one or more programs, which when executed by the one or more processors cause the host system to perform or control performance of any of the methods described herein.
  • Figures 9A-9D illustrate a flowchart representation of a method 900 of managing a storage system, in accordance with some embodiments.
  • method 900 is performed by a storage system (e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, or data storage system 170, Figure 1C) or one or more components of the storage system (e.g., computer system 110, Figure 1A, storage system controller 150, Figure IB, or cluster controller 180, Figure 1C).
  • a storage system e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, or data storage system 170, Figure 1C
  • components of the storage system e.g., computer system 110, Figure 1A, storage system controller 150, Figure IB, or cluster controller 180, Figure 1C.
  • method 900 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a storage system, such as the one or more processing units (CPUs) 152-1 of system
  • FIG. 1A For ease of explanation, the following describes method 900 as performed by a storage system (e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, or data storage system 170, Figure 1C).
  • a storage system e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, or data storage system 170, Figure 1C.
  • one or more of the operations described in method 900 are performed by one or more subsystems of the storage system distinct from the storage device (e.g., storage system controller 150, Figure IB or cluster controller 180, Figure 1C).
  • a storage system obtains, (902) for each storage device (e.g., storage device 120, Figure 1A, or any of storage devices 160-1 to 160-m of Figure IB, or any of storage devices 194-1 to 194-n or 194-j to 194-k of Figure 1C) of a plurality of storage devices of the storage system, one or more metrics of the storage device, the storage device including non- volatile memory.
  • Figure 1 A only shows one storage device 120, in some embodiments, data storage system 100 of Figure 1A includes a plurality of storage devices, of which storage device 120 is one example.
  • the storage device generates and/or maintains the one or more metrics of the storage device, and the storage system obtains the one or more metrics from the storage device.
  • one or more subsystems of a storage system distinct from the storage device e.g., storage system controller 150, Figure IB, or cluster controller 180, Figure 1C
  • the storage system obtains the one or more metrics from the one or more subsystems.
  • a metrics module (e.g., metrics module 222, Figures 2A-1 and 2A-2; metrics module 252-1, Figure 2B-1; or metrics module 282-1, Figure 2C-1) is used to obtain, for each storage device of a plurality of storage devices of the storage system, one or more metrics of the storage device, the storage device including non-volatile memory, as described above with respect to Figures 2A-1, 2A-2, 2B-1, and 2C-1.
  • obtaining (902) one or more metrics of a respective storage device (e.g., storage device 120 of Figure 1A, a respective storage device 160 of Figure IB, or a respective storage device 194 of Figure 1C) of the plurality of storage devices of the storage system includes obtaining (912) at least one metric, of the one or more metrics, for each memory portion of a plurality of memory portions of the respective storage device.
  • at least one metric is generated for each block of a plurality of blocks of the respective storage device.
  • at least one metric is generated for each page of a plurality of pages of the respective storage device.
  • At least one metric is generated for each region of a plurality of regions of the respective storage device. In some embodiments, some metrics are generated on a block basis, some metrics are generated on a page basis, some metrics are generated on a region basis, and/or some metrics are generated on a storage device basis.
  • the one or more metrics of the respective storage device include (914) one or more status metrics corresponding to the respective storage device's ability to retain data.
  • the storage system or a component thereof e.g., metrics module 222-1, Figure 2A-1, of the respective storage device; metrics module 252-1, Figure 2B-1, of storage system controller 150; or metrics module 282-1, Figure 2C-1, of cluster controller 180
  • the storage system or a component thereof generates and/or maintains one or more status metrics for each memory portion of a plurality of memory portions (e.g., in storage medium 130, Figure 1 A) of the respective storage device.
  • the one or more status metrics indicate a respective memory portion's ability to retain data.
  • the one or more status metrics associated with a respective memory portion, of a plurality of memory portions of the respective storage device are stored in a characterization vector corresponding to the respective memory portion.
  • the one or more status metrics stored in the characterization vector for the respective memory portion include a subset or superset of: (a) a bytes written field indicating a number of bytes of data written to pages in the respective memory portion, (b) a program-erase (P/E) cycle field indicating a current count of the number of P/E cycles performed on the respective memory portion, (c) a bit error rate (BER) field indicating a number of errors included in a codeword read from pages of the respective memory portion, and (d) other usage information indicating the health, performance, and/or endurance of the respective memory portion, as it relates to the respective memory portion's ability to retain data.
  • P/E program-erase
  • BER bit error rate
  • the one or more status metrics indicate the respective storage device's ability, as a whole, to retain data. For example, as the respective storage device ages, the one or more status metrics reflect the respective storage device's diminished ability to retain data (e.g., data read from the respective storage device includes more errors as the respective storage device ages).
  • the one or more metrics of the respective storage device include (916) one or more performance metrics corresponding to performance of the respective storage device.
  • the storage system or a component thereof e.g., metrics module 222-1, Figure 2A-1, of the respective storage device; metrics module 252-1, Figure 2B-1, of storage system controller 150; or metrics module 282-1, Figure 2C-1, of cluster controller 180
  • the storage system or a component thereof generates and/or maintains one or more performance metrics for each memory portion of a plurality of memory portions (e.g., in storage medium 130, Figure 1 A) of the respective storage device.
  • the one or more performance metrics correspond to performance of a respective memory portion of the plurality of memory portions.
  • the one or more performance metrics associated with a respective memory portion, of a plurality of memory portions of the respective storage device are stored in a characterization vector corresponding to the respective memory portion.
  • the one or more performance metrics stored in the characterization vector for the respective memory portion include a subset or superset of: (a) a measure of latency, and (b) transaction time.
  • the one or more performance metrics correspond to performance of the respective storage device as a whole.
  • the one or more performance metrics include a measure of latency for the respective storage device and/or transaction time for the respective storage device.
  • the one or more metrics of the respective storage device include (918) one or more wear metrics corresponding to wear on the respective storage device.
  • the storage system or a component thereof e.g., metrics module 222-1, Figure 2A-1, of the respective storage device; metrics module 252-1, Figure 2B-1, of storage system controller 150; or metrics module 282-1, Figure 2C-1, of cluster controller 180
  • the storage system or a component thereof generates and/or maintains one or more wear metrics for each memory portion of a plurality of memory portions (e.g., in storage medium 130, Figure 1A) of the respective storage device.
  • the one or more wear metrics correspond to wear on a respective memory portion of the plurality of memory portions.
  • the one or more wear metrics associated with a respective memory portion, of a plurality of memory portions of the respective storage device are stored in a
  • the one or more wear metrics stored in the characterization vector for the respective memory portion include a subset or superset of: (a) a count of cumulative writes to the respective memory portion, (b) a count of cumulative reads from the respective memory portion, (c) a count of P/E cycles performed on the respective memory portion, and (d) a BER for the respective memory portion.
  • the one or more wear metrics correspond to wear on the respective storage device as a whole.
  • the one or more wear metrics include a count of cumulative writes to the respective storage device, a count of cumulative reads from the respective storage device, a count of P/E cycles performed on the respective storage device and/or a BER for the respective storage device.
  • the one or more metrics of the respective storage device include (920) one or more time metrics.
  • the storage system or a component thereof e.g., metrics module 222-1, Figure 2A-1, of the respective storage device; metrics module 252-1, Figure 2B-1, of storage system controller 150; or metrics module 282-1, Figure 2C-1, of cluster controller 180
  • the one or more time metrics include a wall-clock time.
  • the one or more metrics of the respective storage device include (922) values of the one or more metrics from more than one time.
  • the one or more metrics of the respective storage device include a count of cumulative writes to the respective storage device at a first time and a count of cumulative writes to the respective storage device at a second time.
  • values of the one or more metrics from more than one time include historical knowledge of the one or more metrics.
  • the one or more metrics from more than one time include a running average of the one or more metrics.
  • historical knowledge can be used to determine (e.g., compute) one or more projected values of one or more metrics at a particular time in the future (e.g., an hour, day, week, or month in the future).
  • historical knowledge of the one or more metrics of the respective storage device is used to detect a trigger condition, as described below with respect to operation 904.
  • the storage system detects (904) a trigger condition in accordance with the one or more metrics of one or more respective storage devices of the plurality of storage devices of the storage system.
  • the trigger condition is detected in accordance with a non-linear and/or linear combination of the one or more metrics.
  • the trigger condition is detected in accordance with historical knowledge of the one or more metrics.
  • a trigger detection module (e.g., trigger detection module 224, Figures 2A-1 and 2A-2; trigger detection module 254-1, Figure 2B-1; or trigger detection module 284-1, Figure 2C-1) is used to detect a trigger condition in accordance with the one or more metrics of the respective storage device of the plurality of storage devices of the storage system, as described above with respect to Figures 2A-1, 2A-2, 2B-1, and 2C-1. Examples of trigger conditions are discussed above with reference to Figure 6.
  • one or more metrics from the plurality of storage devices of the storage system are combined to detect the trigger condition. For example, it may be advantageous to examine the rate of change of historical wear metrics such as P/E cycle counts in the plurality of storage devices in order to allow the amelioration process sufficient time to complete before any one device reaches a wear limit.
  • the respective storage device comprises (906) one or more flash memory devices.
  • the respective storage device comprises a storage medium (e.g., storage medium 130, Figure 1A), and the storage medium comprises one or more non- volatile storage devices, such as flash memory devices.
  • the storage medium (e.g., storage medium 130, Figure 1A) is a single flash memory device, while in other embodiments the storage medium includes a plurality of flash memory devices.
  • the storage medium includes dozens or hundreds of flash memory devices, organized in parallel memory channels, such as 16, 32 or 64 flash memory devices per memory channel, and 8, 16 or 32 parallel memory channels.
  • the non- volatile storage medium (e.g., storage medium 130, Figure 1 A) includes NAND-type flash memory or NOR-type flash memory.
  • the storage medium comprises one or more other types of non- volatile storage devices.
  • the storage system enables (908) an amelioration process associated with the detected trigger condition, the amelioration process to reduce declared capacity of the nonvolatile memory of the respective storage device.
  • the amelioration process includes altering an encoding format (e.g., from TLC to SLC and/or changing the redundancy mechanism) of at least a portion of the non-volatile memory of the respective storage device.
  • altering the encoding format of at least a portion of the non-volatile memory of the storage device includes setting the encoding format of an entirety of the non- volatile memory of the storage device to a low-density physical encoding format, for example SLC.
  • the storage device prior to the amelioration process the storage device includes some blocks (e.g., 98%) encoded as TLC and other blocks (e.g., 2%) encoded as SLC, and after the amelioration process all blocks are encoded using the lower-density physical encoding format, SLC.
  • the latter example may correspond to a storage device that initially stores all client data using TLC and all storage device metadata using SLC.
  • the amelioration process converts all of the client data from TLC to SLC without changing the encoding format of the storage device metadata (SLC).
  • the amelioration process includes a process to reduce utilization by a host, a process to reduce declared capacity of the non-volatile memory of the respective storage device, and/or a process to advertise a reduced declared capacity.
  • an enabling module e.g., enabling module 226, Figures 2A-1 and 2A-2; enabling module 256-1, Figure 2B-1; or enabling module 286-1, Figure 2C-1 is used to enable an amelioration process associated with the detected trigger condition, the amelioration process to reduce declared capacity of the non-volatile memory of the respective storage device, as described above with respect to Figures 2A-1, 2A-2, 2B-1, and 2C-1.
  • enabling the amelioration process associated with the detected trigger condition includes (924) notifying a host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C) to which the respective storage device is operatively coupled of the trigger condition.
  • notifying the host of the trigger condition includes notifying the host with an unsolicited communication.
  • the unsolicited communication includes an interrupt communication.
  • the host includes (926) a client on behalf of which data is stored in the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; data storage system 170, Figure 1C).
  • the client is or includes an entity on behalf of which data is stored in the storage system.
  • the host is (1) computer system 110 ( Figure 1A) or a client process, module or application executed by computer system 110, (2) computer system 142 ( Figure IB) or a client process or application executed by computer system 142, and/or (3) computer system 172 ( Figure 1C) or (4) a client process, module or application executed by computer system 172.
  • the host includes (928) a storage system controller of the storage system (e.g., data storage system 140, Figure IB).
  • the storage system controller controls and/or coordinates operations among one or more storage devices.
  • the host is storage system controller 150 ( Figure IB).
  • the data storage system e.g., data storage system 140, Figure IB
  • the data storage system is called a scale-up system.
  • the host includes (930) a cluster controller of the storage system (e.g., data storage system 170, Figure 1C).
  • the cluster controller controls and/or coordinates operations among one or more data storage subsystems, as shown for example in Figure 1C, where each of the data storage subsystems may be implemented as a data storage system having one or more storage devices (e.g., data storage system 140, Figure IB).
  • the host is cluster controller 180 ( Figure IC).
  • the data storage system e.g., data storage system 170, Figure IC
  • the data storage system is called a scale -out system.
  • enabling the amelioration process associated with the detected trigger condition includes (932): (1) receiving a query from a host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure IC, storage system controller 150, Figure IB, and/or cluster controller 180, Figure IC) to which the respective storage device is operatively coupled, and (2) in response to receiving the query, reporting the trigger condition.
  • a host e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure IC, storage system controller 150, Figure IB, and/or cluster controller 180, Figure IC
  • computer system 142 Figure IB
  • storage system controller 150 Figure IB
  • computer system 172 ( Figure IC) polls for the trigger condition and cluster controller 150 ( Figure IC) receives the query from computer system 172 and in response to receiving the query, reports the trigger condition.
  • the host polls for the trigger condition and the respective storage device receives the query from the host and in response to receiving the query, reports the trigger condition.
  • enabling the amelioration process associated with the detected trigger condition includes (934): (1) receiving a command from a host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure IC, storage system controller 150, Figure IB, and/or cluster controller 180, Figure IC) to which the respective storage device is operatively coupled, and (2) in response to receiving the command, sending a response to the command and a notification of the trigger condition.
  • the command includes an I/O (input/output) request.
  • the I/O request includes a read request from the respective storage device and/or a write request to the respective storage device.
  • the command includes a request for temperature of the respective storage device. In some embodiments, the command includes a request for some other status of the respective storage device. In some embodiments, the notification of the trigger condition is piggy-backed on a response to the command from the host. For example, in some embodiments, the host issues a read request for data from the respective storage device, and the respective storage device (1) receives the read request from the host, and (2) in response to receiving the read request, the respective storage device sends data corresponding to the read request and a notification of the trigger condition.
  • the host e.g., computer system 142, Figure IB; or computer system 172, Figure IC
  • the storage system or one or more components of the storage system (1) receives the read request from the host, and (2) in response to receiving the read request, the storage system or one or more components of the storage system sends data corresponding to the read request and a notification of the trigger condition.
  • enabling the amelioration process associated with the detected trigger condition includes (936): (1) receiving a command from a host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure IC, storage system controller 150, Figure IB, and/or cluster controller 180, Figure IC) to which the respective storage device is operatively coupled, and (2) in response to receiving the command, sending a response to the command and a notification that prompts the host to obtain information with respect to the trigger condition.
  • the command includes an I/O (input/output) request.
  • the I/O request includes a read request from the respective storage device and/or a write request to the respective storage device.
  • the command includes a request for temperature of the respective storage device. In some embodiments, the command includes a request for some other status of the respective storage device. In some embodiments, the notification that prompts the host to obtain information with respect to the trigger condition is piggy-backed on a response to the command from the host. For example, in some embodiments, the host issues a read request for data from the respective storage device, and the respective storage device (1) receives the read request from the host, and (2) in response to receiving the read request, the respective storage device sends data corresponding to the read request and a notification (e.g., by setting a notification bit) that prompts the host to obtain information with respect to the trigger condition.
  • a notification e.g., by setting a notification bit
  • the host e.g., computer system 142, Figure IB; or computer system 172, Figure IC
  • the storage system or one or more components of the storage system (1) receives the read request from the host, and (2) in response to receiving the read request, the respective storage device sends data corresponding to the read request and a notification (e.g., by setting a notification bit) that prompts the host to obtain information with respect to the trigger condition.
  • the mechanism used for returning such a notification when responding to a command from the host is a SCSI deferred error or deferred error response code.
  • enabling the amelioration process associated with the detected trigger condition includes (938) scheduling the amelioration process to be performed on the respective storage device.
  • the trigger condition feeds back to storage system controller 150 ( Figure IB) and storage system controller 150 enables the amelioration process by scheduling the amelioration process to be performed on the respective storage device.
  • the trigger condition feeds back to cluster controller 180 ( Figure 1C) and cluster controller 180 enables the amelioration process by scheduling the amelioration process to be performed on the respective storage device.
  • enabling the amelioration process associated with the detected trigger condition includes (940) determining one or more parameters for the amelioration process.
  • the one or more parameters for the amelioration process include a level of urgency for the amelioration process and a targeted amount of storage capacity of the non- volatile memory that needs to be reduced in the respective storage device.
  • the one or more parameters for the amelioration process include a parameter indicating that the urgency level is high (e.g., the amelioration process needs to begin within the next hour) and a parameter indicating that 1 GB of storage capacity needs to be reduced in the respective storage device.
  • enabling the amelioration process associated with the detected trigger condition further includes (942) reporting at least a subset of the one or more parameters for the amelioration process. For example, in some embodiments, enabling the amelioration process associated with the detected trigger condition further includes reporting a targeted amount of storage capacity of the non-volatile memory that needs to be reduced in the respective storage device.
  • the obtaining, the enabling, or both the obtaining and the enabling are performed (944) by one or more subsystems of the storage system distinct from the plurality of storage devices.
  • the obtaining, the enabling, or both the obtaining and the enabling are performed by a storage system controller (e.g., storage system controller 150, Figure IB) of the storage system (e.g., data storage system 140, Figure IB).
  • the obtaining, the enabling, or both the obtaining and the enabling are performed by a cluster controller (e.g., cluster controller 180, Figure 1C) of the storage system (e.g., data storage system 170, Figure 1C).
  • method 900 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 152-1 of system management module 151-1, shown in Figures IB and 2B-1 or the one or more processing units (CPUs) 182-1 of cluster management module 181-1, shown in Figures 1C and 2C-1.
  • processors of a device such as the one or more processing units (CPUs) 152-1 of system management module 151-1, shown in Figures IB and 2B-1 or the one or more processing units (CPUs) 182-1 of cluster management module 181-1, shown in Figures 1C and 2C-1.
  • the storage system re-evaluates the trigger condition in accordance with the one or more metrics of the respective storage device, and (2) in accordance with a determination that the trigger condition is no longer valid, aborts the amelioration process to reduce declared capacity of the non-volatile memory of the respective storage device. For example, in some
  • the one or more metrics of the respective storage device may change such that the trigger condition is no longer present (e.g., the amelioration process is no longer needed).
  • the amelioration process is no longer needed.
  • normal storage operations read, write, trim, etc.
  • the metrics may change so that the trigger condition is no longer present.
  • a trigger detection module e.g., trigger detection module 224, Figures 2A-1 and 2A-2; trigger detection module 254-1, Figure 2B-1; or trigger detection module 284-1, Figure 2C-1
  • an enabling module e.g., enabling module 226, Figures 2A-1 and 2A-2; enabling module 256-1, Figure 2B-1; or enabling module 286-1, Figure 2C-1
  • enabling module 226, Figures 2A-1 and 2A-2 enabling module 256-1, Figure 2B-1; or enabling module 286-1, Figure 2C-1
  • enabling module e.g., enabling module 226, Figures 2A-1 and 2A-2; enabling module 256-1, Figure 2B-1; or enabling module 286-1, Figure 2C-1
  • any of the method described above are performed by a storage system, the storage system including (1) non- volatile memory, (2) one or more processors, and (3) controller memory storing one or more programs, which when executed by the one or more processors cause the storage system to perform or control performance of any of the methods described herein.
  • any of the methods described above are performed by a storage system including means for performing any of the methods described herein.
  • any of the methods described above are performed by a storage system, the storage system including (1) a plurality of storage devices, (2) one or more subsystems having one or more processors, and (3) memory storing one or more programs, which when executed by the one or more processors cause the one or more subsystems to perform or control performance of any of the methods described herein.
  • Figures 1 OA- IOC illustrate a flowchart representation of a method 1000 of managing a storage system, in accordance with some embodiments. At least in some embodiments, method 1000 is performed by a storage device (e.g., storage device 120, Figure
  • method 1000 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 122-1 of management module 121-1, shown in Figures 1A and 2A-1.
  • processors such as the one or more processing units (CPUs) 122-1 of management module 121-1, shown in Figures 1A and 2A-1.
  • method 1000 is performed by a storage system (e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, and/or data storage system 170, Figure 1C) or one or more components of the storage system (e.g., storage device 120, Figure 1A, storage device 160, Figure IB, or storage device 194, Figure 1C). In some embodiments, some of the operations of method 1000 are performed at a storage device
  • storage device 120 (e.g., storage device 120, Figure 1A, storage device 160, Figure IB, or storage device 194)
  • FIG. 1C Figure 1C
  • a host e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller
  • method 1000 is performed by a storage device (e.g., storage device 120, Figure 1A) of a storage system (e.g., data storage system 100, Figure 1A).
  • a storage device e.g., storage device 120, Figure 1A
  • a storage system e.g., data storage system 100, Figure 1A
  • one or more of the operations described in method 1000 are performed by a storage device of another storage system (e.g., storage device 160 of data storage system 140, Figure IB, or storage device 194 of data storage system 170, Figure 1C).
  • the storage device including nonvolatile memory (1002), the storage device (e.g., storage device 120, Figure 1A) detects (1004) a trigger condition for reducing declared capacity of the non- volatile memory of the storage device.
  • the trigger condition is detected in accordance with a non-linear and/or linear combination of one or more metrics of the storage device.
  • the trigger condition is detected in accordance with historical knowledge of the one or more metrics, as described above with respect to operation 602 of Figure 6.
  • the trigger condition is detected when the storage device transitions from normal operation to read-only mode.
  • a trigger detection module (e.g., trigger detection module 224-1, Figure 2A-1) is used to detect a trigger condition for reducing declared capacity of the non-volatile memory of the storage device, as described above with respect to Figure 2A-1.
  • the amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes a process to reduce utilization of the non-volatile memory of the storage device, for example as described above with respect to operation 608 of Figure 6.
  • the storage device comprises (1006) one or more flash memory devices.
  • the storage device comprises a storage medium (e.g., storage medium 130, Figure 1A), and the storage medium comprises one or more nonvolatile storage devices, such as flash memory devices.
  • the storage medium e.g., storage medium 130, Figure 1A
  • the storage medium is a single flash memory device, while in other embodiments the storage medium includes a plurality of flash memory devices.
  • the storage medium includes dozens or hundreds of flash memory devices, organized in parallel memory channels, such as 16, 32 or 64 flash memory devices per memory channel, and 8, 16 or 32 parallel memory channels.
  • the non- volatile storage medium (e.g., storage medium 130, Figure 1A) includes NAND-type flash memory or NOR-type flash memory. In other embodiments, the storage medium comprises one or more other types of non-volatile storage devices.
  • the trigger condition is (1008) detected by the storage device in accordance with one or more metrics of the storage device. In some embodiments, the one or more metrics of the storage device include one or more status metrics
  • the storage device including nonvolatile memory (1002), the storage device (e.g., storage device 120, Figure 1A) notifies (1010) a host to which the storage device is operatively coupled of the trigger condition for reducing declared capacity of the non-volatile memory of the storage device, the trigger condition for enabling performance of an amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • a notification module e.g., notification module 228-1, Figure 2A-1 is used to notify a host to which the storage device is operatively coupled of the trigger condition for reducing declared capacity of the non- volatile memory of the storage device, as described above with respect to Figure 2A-1.
  • the host includes (1014) a client on behalf of which data is stored in the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; data storage system 170, Figure 1C).
  • the client is or includes an entity on behalf of which data is stored in the storage system.
  • the host is (1) computer system 110 ( Figure 1A) or a client process, module or application executed by computer system 110, (2) computer system 142 ( Figure IB) or a client process, module or application executed by computer system 142, and/or (3) computer system 172 ( Figure 1C) or a client process, module or application executed by computer system 172.
  • the host includes (1016) a storage system controller of the storage system (e.g., data storage system 140, Figure IB).
  • the storage system controller controls and/or coordinates operations among one or more storage devices.
  • the host is storage system controller 150
  • the data storage system (e.g., data storage system 140, Figure IB) is called a scale-up system.
  • the host includes (1018) a cluster controller of the storage system (e.g., data storage system 170, Figure 1C).
  • the cluster controller controls and/or coordinates operations among one or more data storage subsystems, as shown for example in Figure 1C, where each of the data storage subsystems may be implemented as a data storage system having one or more storage devices (e.g., data storage system 140, Figure IB).
  • the host is cluster controller 180 ( Figure 1C).
  • the data storage system (e.g., data storage system 170, Figure 1C) is called a scale-out system, sometimes known as a clustered storage system.
  • notifying (1010) the host of the trigger condition for reducing declared capacity of the non-volatile memory of the storage device includes notifying (1020) the host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C) with an unsolicited communication.
  • the unsolicited communication includes an interrupt communication.
  • the unsolicited communication includes a remote direct memory access (RDMA).
  • RDMA remote direct memory access
  • the unsolicited communication includes a TCP connection request.
  • the unsolicited communication includes any other form of unsolicited communication.
  • notifying (1010) the host of the trigger condition for reducing declared capacity of the non- volatile memory of the storage device includes (1022): (1) receiving a query from the host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C), and (2) in response to receiving the query, reporting the trigger condition.
  • the host e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C
  • the host polls for the trigger condition and the storage device receives the query from the host and in response to receiving the query, reports the trigger condition.
  • notifying (1010) the host of the trigger condition for reducing declared capacity of the non- volatile memory of the storage device includes (1024): (1) receiving a command from the host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C), and (2) in response to receiving the command, sending a response to the command and a notification of the trigger condition.
  • the command includes an I/O (input/output) request.
  • the I/O request includes a read request from the storage device and/or a write request to the storage device.
  • the command includes a request for temperature of the storage device. In some embodiments, the command includes a request for some other status of the storage device. In some embodiments, the notification of the trigger condition is piggy-backed on a response to the command from the host. For example, in some embodiments, the host issues a read request for data from the storage device, and the storage device (1) receives the read request from the host, and (2) in response to receiving the read request, the storage device sends data corresponding to the read request and a notification of the trigger condition.
  • notifying (1010) the host of the trigger condition for reducing declared capacity of the non- volatile memory of the storage device includes (1026): (1) receiving a command from the host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C), and (2) in response to receiving the command, sending a response to the command and a notification that prompts the host to obtain information with respect to the trigger condition.
  • the command includes an I/O (input/output) request.
  • the I/O request includes a read request from the storage device and/or a write request to the storage device.
  • the command includes a request for temperature of the storage device. In some embodiments, the command includes a request for some other status of the storage device. In some embodiments, the notification that prompts the host to obtain information with respect to the trigger condition is piggy-backed on a response to the command from the host. For example, in some embodiments, the host issues a read request for data from the storage device, and the storage device (1) receives the read request from the host, and (2) in response to receiving the read request, the storage device sends data corresponding to the read request and a notification (e.g., by setting a notification bit) that prompts the host to obtain information with respect to the trigger condition. In some embodiments, the mechanism used for returning a notification when responding to a command from the host is a SCSI deferred error or deferred error response code.
  • notifying (1010) the host of the trigger condition for reducing declared capacity of the non-volatile memory of the storage device further includes notifying (1028) the host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C) that the storage device is in read-only mode.
  • notifying the host that the storage device is in read-only mode includes notifying the host that the storage device is not accepting any more write commands.
  • notifying the host that the storage device is in read-only mode includes sending one or more rejections of write commands.
  • the storage device after notifying the host of the trigger condition for reducing declared capacity of the non-volatile memory of the storage device (1012), the storage device (1) re-evaluates the trigger condition in accordance with the one or more metrics of the storage device, and (2) in accordance with a determination that the trigger condition is no longer valid, notifies the host of an absence of the trigger condition for reducing declared capacity of the non- volatile memory of the storage device.
  • the one or more metrics of the storage device may change such that the trigger condition is no longer valid (e.g., the amelioration process is no longer needed).
  • Normal storage operations include operations like trim that explicitly reduce the storage device utilization, possibly enough to merit aborting the amelioration process.
  • Other storage activity such as garbage collection may also reduce utilization, possibly enough to merit aborting the amelioration process.
  • the trigger condition (e.g., periodically, semi- continuously, initially, finally, occasionally or irregularly) is recomputed or re-evaluated in accordance with the one or more metrics of the storage device, as the one or more metrics may change in value due to the amelioration process and/or normal storage operations (e.g., read, write, erase and trim or unmap operations).
  • a trigger detection module e.g., trigger detection module 224-1, Figure 2A-1
  • a notification module e.g., notification module 228-1, Figure 2A-1
  • a trigger detection module e.g., trigger detection module 224-1, Figure 2A-1
  • a notification module e.g., notification module 228-1, Figure 2A-1
  • a trigger detection module e.g., trigger detection module 224-1, Figure 2A-1
  • a notification module e.g., notification module 228-1, Figure 2A-1
  • any operations of method 1000 described above are performed by a storage device, the storage device including (1) non-volatile memory (e.g., comprising one or more non- volatile storage devices, such as flash memory devices) (2) one or more processors, and (3) controller memory (e.g., non-volatile memory or volatile memory in or coupled to the controller) storing one or more programs, which when executed by the one or more processors cause the storage device to perform or control performance of any of the methods described herein.
  • non-volatile memory e.g., comprising one or more non- volatile storage devices, such as flash memory devices
  • controller memory e.g., non-volatile memory or volatile memory in or coupled to the controller
  • any operations of method 1000 described above are performed by a storage device including means for performing any of the methods described herein.
  • any operations of method 1000 described above are performed by a storage system comprising (1) a storage medium (e.g., comprising one or more non- volatile storage devices, such as flash memory devices) (2) one or more processors, and (3) memory (e.g., non-volatile memory or volatile memory in the storage system) storing one or more programs, which when executed by the one or more processors cause the storage system to perform or control performance of any of the methods described herein.
  • a storage medium e.g., comprising one or more non- volatile storage devices, such as flash memory devices
  • processors e.g., non-volatile memory or volatile memory in the storage system
  • memory e.g., non-volatile memory or volatile memory in the storage system
  • Figures 1 lA-11C illustrate a flowchart representation of a method 1100 of managing a storage system, in accordance with some embodiments. At least in some embodiments, method 1100 is performed by a storage system (e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, or data storage system 170, Figure 1C) or one or more components of the storage system (e.g., computer system 1 10, Figure 1A, storage system controller 150, Figure IB, or cluster controller 180, Figure 1C).
  • a storage system e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, or data storage system 170, Figure 1C
  • a storage system e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, or data storage system 170, Figure 1C
  • components of the storage system e.g., computer system 1 10, Figure 1A, storage system controller 150, Figure IB, or cluster controller 180, Figure 1C.
  • method 1100 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a storage system, such as the one or more processing units (CPUs) 152-1 of system
  • FIG. 1A For ease of explanation, the following describes method 1100 as performed by a storage system (e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, or data storage system 170, Figure 1C).
  • a storage system e.g., data storage system 100, Figure 1A, data storage system 140, Figure IB, or data storage system 170, Figure 1C.
  • one or more of the operations described in method 1100 are performed by one or more subsystems of the storage system distinct from the storage device (e.g., storage system controller 150, Figure IB or cluster controller 180, Figure 1C).
  • a storage system obtains (1102), for each storage device (e.g., storage device 120, Figure 1A, or any of storage devices 160-1 to 160-m of Figure IB, or any of storage devices 194-1 to 194-n or 194-j to 194-k of Figure 1C) of a plurality of storage devices of the storage system, one or more metrics of the storage device, the storage device including non- volatile memory.
  • each storage device e.g., storage device 120, Figure 1A, or any of storage devices 160-1 to 160-m of Figure IB, or any of storage devices 194-1 to 194-n or 194-j to 194-k of Figure 1C
  • Figure 1 A only shows one storage device 120
  • data storage system 100 of Figure 1A includes a plurality of storage devices, of which storage device 120 is one example.
  • the storage device generates and/or maintains the one or more metrics of the storage device, and the storage system obtains the one or more metrics from the storage device.
  • one or more subsystems of a storage system distinct from the storage device e.g., storage system controller 150, Figure IB, or cluster controller 180, Figure 1C
  • the storage system obtains the one or more metrics from the one or more subsystems.
  • a metrics module (e.g., metrics module 222, Figures 2A-1 and 2A-2; metrics module 252-1, Figure 2B-1; or metrics module 282-1, Figure 2C-1) is used to obtain, for each storage device of a plurality of storage devices of the storage system, one or more metrics of the storage device, the storage device including non-volatile memory, as described above with respect to Figures 2A-1, 2A-2, 2B-1, and 2C-1.
  • the storage system detects (1104) a trigger condition for reducing declared capacity of the non-volatile memory of a respective storage device of the plurality of storage devices of the storage system, the trigger condition detected in accordance with the one or more metrics of two or more of the storage devices of the plurality of storage devices in the storage system.
  • the one or more metrics of the respective storage device include one or more status metrics corresponding to the respective storage device's ability to retain data, one or more performance metrics corresponding to performance of the respective storage device, one or more wear metrics corresponding to wear on the respective storage device, and/or one or more time metrics, as described above with respect to operation 602 of Figure 6.
  • the trigger condition is detected in accordance with a non-linear and/or linear combination of the one or more metrics of the respective storage device. In some embodiments, the trigger condition is detected in accordance with historical knowledge of the one or more metrics of the respective storage device, as described above with respect to operation 602 of Figure 6. In some embodiments, the trigger condition is detected when the respective storage device transitions from normal operation to read-only mode. In some embodiments, one or more metrics from the plurality of storage devices of the storage system are combined to detect the trigger condition. For example, it may be advantageous to examine the rate of change of historical wear metrics such as P/E cycle counts in the plurality of storage devices in order to allow the amelioration process sufficient time to complete before any one device reaches a wear limit.
  • historical wear metrics such as P/E cycle counts
  • a trigger detection module (e.g., trigger detection module 224, Figures 2A-1 and 2A-2; trigger detection module 254-1, Figure 2B-1; or trigger detection module 284-1, Figure 2C-1) is used to detect a trigger condition for reducing declared capacity of the non- volatile memory of a respective storage device of the plurality of storage devices of the storage system, the trigger condition detected in accordance with the one or more metrics of the respective storage device, as described above with respect to Figures 2A-1, 2A-2, 2B-1, and 2C-1.
  • the respective storage device comprises (1106) one or more flash memory devices.
  • the respective storage device comprises a storage medium (e.g., storage medium 130, Figure 1A), and the storage medium comprises one or more non- volatile storage devices, such as flash memory devices.
  • the storage medium comprises one or more non- volatile storage devices, such as flash memory devices.
  • the storage medium (e.g., storage medium 130, Figure 1A) is a single flash memory device, while in other embodiments the storage medium includes a plurality of flash memory devices.
  • the storage medium includes dozens or hundreds of flash memory devices, organized in parallel memory channels, such as 16, 32 or 64 flash memory devices per memory channel, and 8, 16 or 32 parallel memory channels.
  • the non- volatile storage medium (e.g., storage medium 130, Figure 1 A) includes NAND-type flash memory or NOR-type flash memory.
  • the storage medium comprises one or more other types of non- volatile storage devices.
  • the storage system notifies (1108) a host to which the respective storage device is operatively coupled of the trigger condition for reducing declared capacity of the non-volatile memory of the respective storage device, the trigger condition for enabling performance of an amelioration process to reduce declared capacity of the non-volatile memory of the respective storage device.
  • a notification module (e.g., notification module 228, Figures 2A-1 and 2A-2, notification module 258-1, Figure 2B-1, or notification module 288-1, Figure 2C-1) is used to notify a host to which the respective storage device is operatively coupled of the trigger condition for reducing declared capacity of the non-volatile memory of the respective storage device, the trigger condition associated with an amelioration process to reduce declared capacity of the non- volatile memory of the respective storage device, as described above with respect to Figures 2A-1, 2A-2, 2B-1, and 2C-1.
  • the notification of the trigger condition causes performance of the amelioration process to be enabled.
  • the amelioration process reduces declared capacity of two or more of the storage devices in the storage system.
  • the host includes (1112) a client on behalf of which data is stored in the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; data storage system 170, Figure 1C).
  • the client is or includes an entity on behalf of which data is stored in the storage system.
  • the host is (1) computer system 110 ( Figure 1A) or a client process, module or application executed by computer system 110, (2) computer system 142 ( Figure IB) or a client process, module or application executed by computer system 142, and/or (3) computer system 172 ( Figure 1C) or a client process, module or application executed by computer system 172.
  • the host includes (1114) a storage system controller of the storage system (e.g., data storage system 140, Figure IB).
  • the storage system controller controls and/or coordinates operations among one or more storage devices.
  • the host is storage system controller 150 ( Figure IB).
  • the data storage system e.g., data storage system 140, Figure IB
  • the data storage system is called a scale-up system.
  • the host includes (1116) a cluster controller of the storage system (e.g., data storage system 170, Figure 1C).
  • the cluster controller controls and/or coordinates operations among one or more data storage subsystems, as shown for example in Figure 1C, where each of the data storage subsystems may be implemented as a data storage system having one or more storage devices (e.g., data storage system 140, Figure IB).
  • the host is cluster controller 180 ( Figure 1C).
  • the data storage system e.g., data storage system 170, Figure 1C
  • the data storage system is called a scale-out system or a clustered storage system.
  • notifying (1108) the host of the trigger condition for reducing declared capacity of the non-volatile memory of the respective storage device includes notifying (1118) the host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C) with an unsolicited communication.
  • the unsolicited communication includes an interrupt communication.
  • the unsolicited communication includes a remote direct memory access (RDMA).
  • RDMA remote direct memory access
  • the unsolicited communication includes a TCP connection request.
  • the unsolicited communication includes any other form of unsolicited communication.
  • notifying (1108) the host of the trigger condition for reducing declared capacity of the non-volatile memory of the respective storage device includes (1120): (1) receiving a query from the host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C), and (2) in response to receiving the query, reporting the trigger condition.
  • the host polls for the trigger condition and the storage system receives the query from the host and in response to receiving the query, reports the trigger condition.
  • notifying (1108) the host of the trigger condition for reducing declared capacity of the non-volatile memory of the respective storage device includes (1122): (1) receiving a command from the host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C), and (2) in response to receiving the command, sending a response to the command and a notification of the trigger condition.
  • the command includes an I/O (input/output) request.
  • the I/O request includes a read request from the respective storage device and/or a write request to the respective storage device.
  • the command includes a request for temperature of the respective storage device. In some embodiments, the command includes a request for some other status of the respective storage device. In some embodiments, the notification of the trigger condition is piggy-backed on a response to the command from the host. For example, in some embodiments, the host issues a read request for data from the respective storage device, and the storage system (1) receives the read request from the host, and (2) in response to receiving the read request, the storage system sends data corresponding to the read request and a notification of the trigger condition.
  • notifying (1108) the host of the trigger condition for reducing declared capacity of the non-volatile memory of the respective storage device includes (1124): (1) receiving a command from the host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C), and (2) in response to receiving the command, sending a response to the command and a notification that prompts the host to obtain information with respect to the trigger condition.
  • the command includes an I/O (input/output) request.
  • the I/O request includes a read request from the respective storage device and/or a write request to the respective storage device.
  • the command includes a request for temperature of the respective storage device.
  • the command includes a request for some other status of the respective storage device.
  • the notification that prompts the host to obtain information with respect to the trigger condition is piggy-backed on a response to the command from the host.
  • the host issues a read request for data from the respective storage device, and the storage system (1) receives the read request from the host, and (2) in response to receiving the read request, the storage system sends data corresponding to the read request and a notification (e.g., by setting a notification bit) that prompts the host to obtain information with respect to the trigger condition.
  • the mechanism used for returning a notification when responding to a command from the host is a SCSI deferred error or deferred error response code.
  • notifying (1108) the host of the trigger condition for reducing declared capacity of the non-volatile memory of the respective storage device includes notifying (1126) the host (e.g., computer system 110, Figure 1A, computer system 142, Figure IB, computer system 172, Figure 1C, storage system controller 150, Figure IB, and/or cluster controller 180, Figure 1C) that the respective storage device is in read-only mode.
  • notifying the host that the respective storage device is in readonly mode includes notifying the host that the respective storage device is not accepting any more write commands.
  • notifying the host that the respective storage device is in read-only mode includes sending one or more rejections of write commands.
  • the obtaining, the notifying, or both the obtaining and the notifying are performed (1128) by one or more subsystems of the storage system distinct from the plurality of storage devices.
  • the obtaining, the notifying, or both the obtaining and the notifying are performed by a storage system controller (e.g., storage system controller 150, Figure IB) of the storage system (e.g., data storage system 140, Figure IB).
  • the obtaining, the notifying, or both the obtaining and the notifying are performed by a cluster controller (e.g., cluster controller 180, Figure 1C) of the storage system (e.g., data storage system 170, Figure 1C).
  • method 1100 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 152-1 of system management module 151-1, shown in Figures IB and 2B-1 or the one or more processing units (CPUs) 182-1 of cluster management module 181-1, shown in Figures 1C and 2C-1.
  • processors of a device such as the one or more processing units (CPUs) 152-1 of system management module 151-1, shown in Figures IB and 2B-1 or the one or more processing units (CPUs) 182-1 of cluster management module 181-1, shown in Figures 1C and 2C-1.
  • the amelioration process to reduce declared capacity of the non- volatile memory of the respective storage device includes (1130) a process to reduce utilization of the non-volatile memory of the respective storage device, for example as described above with respect to operation 608 of Figure 6.
  • the storage system (1) re-evaluates the trigger condition in accordance with the one or more metrics of the two or more storage devices of the plurality of storage devices in the storage system, and (2) in accordance with a determination that the trigger condition is no longer valid, notifies the host of an absence of the trigger condition for reducing declared capacity of the non-volatile memory of the respective storage device.
  • the one or more metrics of the respective storage device may change such that the trigger condition is no longer valid (e.g., the amelioration process is no longer needed).
  • Normal storage operations include operations like trim that explicitly reduce the respective storage device utilization, possibly enough to merit aborting the amelioration process.
  • Other storage activity such as garbage collection may also reduce utilization, possibly enough to merit aborting the amelioration process.
  • the trigger condition (e.g., periodically, semi- continuously, initially, finally, occasionally or irregularly) is recomputed or re-evaluated in accordance with the one or more metrics of the respective storage device, as the one or more metrics may change in value due to the amelioration process and/or normal storage operations (e.g., read, write, erase and trim or unmap operations).
  • a trigger detection module e.g., trigger detection module 224, Figures 2A-1 and 2A-2; trigger detection module 254-1, Figure 2B-1; or trigger detection module 284-1, Figure 2C-1
  • a notification module e.g., notification module 228, Figures 2A-1 and 2A-2, notification module 258-1, Figure 2B-1, or notification module 288-1, Figure 2C-1 are used to, after notifying the host of the trigger condition for reducing declared capacity of the non- volatile memory of the respective storage device: (1) re-evaluate the trigger condition in accordance with the one or more metrics of the respective storage device, and (2) in accordance with a determination that the trigger condition is no longer valid, notify the host of an absence of the trigger condition for reducing declared capacity of the non- volatile memory of the respective storage device, as described above with respect to Figures 2A-1, 2A-2, 2B-1, and 2C-1.
  • any operations of method 1100 described above are performed by a storage system, the storage system including (1) non-volatile memory (e.g., comprising one or more non-volatile storage devices, such as flash memory devices), (2) one or more processors, and (3) controller memory (e.g., non-volatile memory or volatile memory in or coupled to a controller of the storage system) storing one or more programs, which when executed by the one or more processors cause the storage system to perform or control performance of any of the methods described herein.
  • non-volatile memory e.g., comprising one or more non-volatile storage devices, such as flash memory devices
  • controller memory e.g., non-volatile memory or volatile memory in or coupled to a controller of the storage system
  • any operations of method 1100 described above are performed by a storage system including means for performing any of the methods described herein.
  • any operations of method 1100 described above are performed by a storage system including (1) a plurality of storage devices, (2) one or more subsystems having one or more processors, and (3) memory (e.g., non-volatile memory or volatile memory in the storage system) storing one or more programs, which when executed by the one or more processors cause the one or more subsystems to perform or control performance of any of the methods described herein.
  • a storage system including (1) a plurality of storage devices, (2) one or more subsystems having one or more processors, and (3) memory (e.g., non-volatile memory or volatile memory in the storage system) storing one or more programs, which when executed by the one or more processors cause the one or more subsystems to perform or control performance of any of the methods described herein.
  • Figures 12A-12C illustrate a flowchart representation of a method 1200 of managing a storage system, in accordance with some embodiments.
  • the method 1200 includes detecting (1202) an amelioration trigger for reducing declared capacity of non-volatile memory of a storage device (e.g., storage device 120 of Figure 1A, or any of storage devices 160-1 to 160-m of Figure IB, or any of storage devices 194-1 to 194-n or 194-j to 194-k of Figure 1C) of the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; or data storage system 170, Figure 1C), and in accordance with the detected amelioration trigger, performing (1204) an amelioration process to reduce declared capacity of the non-volatile memory of the storage device, including reducing a range of logical addresses of a logical address space available to a host (e.g., computer system 110 of Figure 1A, computer system 142 of Figure IB, or computer
  • a host e
  • reducing declared capacity of non- volatile memory of the storage device includes reducing storage capacity of the non-volatile memory of the storage device.
  • the declared capacity of non-volatile memory of the storage device is sometimes called the advertised capacity, and is typically used by the operating system and/or a file system of the host as the maximum capacity that the host's operating system or file system is permitted to allocate.
  • the amelioration trigger is detected (1202), for example by detection module 231 ( Figure 2A-1 or 2A-2), 261 ( Figure 2B-1 or 2B-2) or 291 ( Figure 2C-1 or 2C-2).
  • detecting the amelioration trigger includes receiving or generating the amelioration trigger.
  • an amelioration module e.g., amelioration module
  • FIG. 230 Figures 2A-1 and 2A-2, amelioration module 260, Figures 2B-1 and 2B-2, and/or amelioration module 290, Figures 2C-1 and 2C-2) is used to perform (1204) the amelioration process, in accordance with the amelioration trigger, to reduce declared capacity of the nonvolatile memory of the storage device, including reducing a range of logical addresses of a logical address space available to a host.
  • the host e.g., computer system 110 of Figure 1A, computer system 142 of Figure IB, or computer system 172 of Figure 1C
  • the host includes (1206) a client on behalf of which data is stored in the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; data storage system 170, Figure 1C).
  • the client is or includes an entity on behalf of which data is stored in the storage system.
  • the host includes (1208) a storage system controller
  • the storage system controller controls and/or coordinates operations among one or more storage devices.
  • the data storage system e.g., data storage system 140, Figure IB
  • the data storage system is called a scale-up system.
  • the host includes (1210) a cluster controller (e.g., cluster controller 180, Figure 1C) of the storage system (e.g., data storage system 170, Figure 1C).
  • the cluster controller controls and/or coordinates operations among one or more data storage subsystems, as shown for example in Figure 1C, where each of the data storage subsystems may be implemented as a data storage system having one or more storage devices (e.g., data storage system 140, Figure IB), of these embodiments, the data storage system (e.g., data storage system 170, Figure 1C) is called a scale-out system or a clustered storage system.
  • the detecting, the performing, or both the detecting and the performing are performed (1212) by the storage device (e.g., storage device 120, Figure 1A, storage device 160, Figure IB, or storage device 194, Figure 1C) or one or more components of the storage device (e.g., storage controller 124, Figure 1A).
  • method 1200, or at least the detecting operation 1202 and/or performing operation 1204 of method 1200 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 122-1 of management module 121- 1 , shown in Figures 1 A and 2 A- 1.
  • CPUs processing units
  • the detecting, the performing, or both the detecting and the performing are performed (1214) by one or more subsystems of the storage system distinct from the storage device.
  • the detecting, the performing, or both the detecting and the performing are performed by a storage system controller (e.g., storage system controller 150, Figure IB) of the storage system (e.g., data storage system 140, Figure IB).
  • method 1200 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 152-1 of management module 151-1, shown in Figures IB and 2B-l .
  • processors such as the one or more processing units (CPUs) 152-1 of management module 151-1, shown in Figures IB and 2B-l .
  • the detecting, the performing, or both the detecting and the performing are performed (1216), at least in part, by the host.
  • method 1200, or at least the detecting operation 1202 and/or performing operation 1204 of method 1200 is governed at least in part by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a host (processors not shown in Figures 1A, IB and 1C), such as the one or more processing units (CPUs) of management module 121-2 ( Figures 1A and 2A-2); or management module 151-2 ( Figures IB and 2B-2); or management module 181-2 ( Figures 1C and 2C-2).
  • reducing (1204) the range of logical addresses of the logical address space available to the host includes reducing (1218) the range of logical addresses of the logical address space available to the host in accordance with one or more parameters for the amelioration process.
  • the one or more parameters for the amelioration process include a level of urgency for the amelioration process, a target reduced declared capacity of the non-volatile memory of the storage device, and/or a target amount of reduction in utilization of the non-volatile memory of the storage device, or any combination or subset thereof.
  • reducing the range of logical addresses of the logical address space available to the host in accordance with one or more parameters for the amelioration process includes reducing enough logical addresses of the logical address space available to the host to meet a target amount of declared capacity reduction specified by the one or more parameters.
  • the one or more parameters of the amelioration process indicate or correspond to a target amount of reduction in storage capacity available to the host.
  • the one or more parameters take into account, or enable the amelioration process to take into account, a projected reduction in the declared capacity of the non-volatile memory of the storage device that will be needed in a predefined time period (e.g., an hour, a day, a week, or some other predefined time period).
  • reducing (1204) the range of logical addresses of the logical address space available to the host includes removing (1220) a contiguous portion of the logical address space available to the host (e.g., a beginning range, an end range, or a range not at the beginning or end of the logical address space).
  • a contiguous portion of the logical address space available to the host e.g., a beginning range, an end range, or a range not at the beginning or end of the logical address space.
  • LB A space 320 is an example of a logical address space available to a host.
  • removing a contiguous portion of the logical address space available to the host could be accomplished by removing a determined number of logical addresses from either the high end or low end of the LBA Space 320, thereby making either the determined number of the highest logical addresses (e.g., highest LBA's) or lowest logical addresses (e.g., lowest LBA's) unavailable to the host.
  • the highest logical addresses e.g., highest LBA's
  • lowest logical addresses e.g., lowest LBA's
  • reducing the range of logical addresses of the logical address space available to the host includes altering (1222) one or more logical address entries of a mapping table, the mapping table used to translate logical addresses in the logical address space (e.g., LBA space 320) to physical addresses in a physical address space (e.g., physical address space 318) of the storage device.
  • This is sometimes implemented using remapping, such as remapping one set of logical addresses (e.g., logical block addresses) to another set of logical addresses.
  • altering (1222) one or more logical address entries of the mapping table includes moving (1224) the one or more logical address entries without moving data stored at the one or more physical addresses associated with the one or more logical address entries. It is noted that moving logical address entries of the mapping table can, at least in some circumstances, be accomplished without physically moving data from one location to another within the physical address space of the non- volatile memory of the storage device. Typically, the moving operation involves transferring the physical address from the source table entry to the destination table entry and converting the source table entry to an unallocated state. In some embodiments, converting the source table entry to an unallocated state is done by removing the source table entry from the mapping table.
  • the method includes selecting (1226) the one or more logical address entries to be altered so as to minimize performance degradation. For example, to achieve a particular amount of declared capacity reduction, in some circumstances there may be multiple candidates, or sets of candidates, of logical address entries of the mapping table that could be altered to help accomplish the reduction in declared storage capacity of the nonvolatile memory of the storage device. Further, some of the sets of candidates may be accessed or overwritten less frequently than other sets of candidates. By selecting (1226) the one or more logical address entries from among the sets of candidates that are accessed or overwritten the least frequently, the process minimizes performance degradation. In another example, the logical address entries to be altered are selected in accordance with a wear- leveling methodology, so as to promote uniform wearing, or to avoid uneven wearing of the non-volatile memory media.
  • the method includes selecting (1228) the one or more logical address entries to be altered so as to minimize overhead from garbage collection. While moving logical address entries of the mapping table may, at least in some circumstances, be accomplished without physically moving data from one location to another within the physical address space of the non-volatile memory of the storage device, in practice the underlying cause of the detected amelioration trigger may dictate the physical movement of some data within the physical address space of the non-volatile memory of the storage device. In accordance with some embodiments, the device or system that does the selecting (1228) uses a process or analytical method to minimize the amount of data movement, that will result from altering the logical address entries.
  • the method includes selecting (1230) the one or more logical address entries to be altered so as to minimize a number of logical address entries to be altered.
  • the device or system that does the selecting (1230) may use a process or analytical method similar to that used in some "disk defragmentation" processes to identify the smallest number of logical address entries to alter.
  • performing (1204) the amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes reducing utilization of the non-volatile memory of the storage device, for example as described above with respect to operation 608 of Figure 6.
  • reducing (1204) the range of logical addresses of the logical address space available to the host includes copying data stored at a first set of physical addresses associated with a first set of logical address entries in a mapping table to a second set of physical addresses associated with a second set of logical address entries in the mapping table, and updating the first set of logical address entries in the mapping table, the mapping table used to translate logical addresses in the logical address space to physical addresses in a physical address space of the storage device.
  • the first set of logical address entries includes a single logical address entry of the mapping table.
  • the first set of logical address entries includes a contiguous set of logical address entries of the mapping table.
  • the first set of logical address entries includes a non-contiguous set of logical address entries of the mapping table.
  • the second set of logical address entries includes a single logical address entry of the mapping table.
  • the second set of logical address entries includes a contiguous set of logical address entries of the mapping table.
  • the second set of logical address entries includes a non-contiguous set of logical address entries of the mapping table.
  • data stored at a physical address associated with LBA 5 is copied to a physical address associated with LBA 100 (e.g., in mapping table 402, Figure 4), and the logical address entry for LBA 5 is updated.
  • data stored at physical addresses associated with LBA 150 through LBA 157 is copied to physical addresses associated with LBA 10 through LBA 17 (e.g., in mapping table 402, Figure 4), and logical address entries for LBA 150 through LBA 157 are updated.
  • data stored at physical addresses associated with LBA 138, LBA 252, and LBA 388 is copied to physical addresses associated with LBA 56, LBA 21, and LBA 7, respectively, and logical address entries for LBA 138, LBA 252, and LBA 388 are updated.
  • LBA 138, LBA 252, and LBA 388 are updated.
  • data stored at physical addresses associated with LBA 138, LBA 252, and LBA 388 is copied to physical addresses associated with LBA 56 through LBA 58, and logical address entries for LBA 138, LBA 252, and LBA 388 are updated.
  • updating the first set of logical address entries in the mapping table includes invalidating one or more logical address entries of the first set of logical address entries. In some embodiments, updating the first set of logical address entries in the mapping table includes invalidating one logical address entry of the first set of logical address entries. In some embodiments, updating the first set of logical address entries in the mapping table includes invalidating some (but not all) of the logical address entries of the first set of logical address entries. In some embodiments, updating the first set of logical address entries in the mapping table includes invalidating all of the logical address entries of the first set of logical address entries.
  • performing the amelioration process (1204) to reduce declared capacity of the non-volatile memory of the storage device further includes advertising (1232) a reduced declared capacity of the non-volatile memory of the storage device.
  • the storage device, or a corresponding storage controller, cluster controller, management module or data storage system sends a message to the host advertising the reduced declared capacity of the non-volatile memory of the storage device.
  • advertising the reduced declared capacity is accomplished by sending an interrupt or other in-band or out-of-band message to the host.
  • advertising the reduced declared capacity is accomplished by receiving a query from a host to which the storage device is operatively coupled, and in response to receiving the query, reporting the reduced declared capacity of the non-volatile memory of the storage device.
  • the host is configured to periodically query the storage system, storage controller, management module, cluster controller or storage device, for example for a system or device health status.
  • advertising the reduced declared capacity is accomplished by receiving a command (e.g., a storage read or write command) from a host to which the storage device is operatively coupled, and in response to receiving the command, sending a response to the command that includes a notification of the reduced declared capacity of the non-volatile memory of the storage device.
  • a command e.g., a storage read or write command
  • advertising the reduced declared capacity is accomplished by receiving a command (e.g., a storage read or write command) from a host to which the storage device is operatively coupled, and in response to receiving the command, sending both a response to the command and a notification that prompts the host to obtain information, including the reduced declared capacity of the non-volatile memory of the storage device, from the storage device or from the data storage system that includes the storage device.
  • a command e.g., a storage read or write command
  • the mechanism used for returning a notification when responding to a command from the host is a SCSI deferred error or deferred error response code.
  • the method after beginning performance of the amelioration process (1204) to reduce declared capacity of the non- volatile memory of the storage device, the method includes detecting an indication (1234) to abort the reduction in declared capacity of the non-volatile memory of the storage device; and in response to detecting the indication to abort the reduction in declared capacity of the non-volatile memory of the storage device, aborting (1236) performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • detecting the indication to abort is herein defined to mean either receiving a signal to abort the reduction in declared capacity (e.g., receiving the signal from a controller of the storage device or a storage system controller of a storage system that includes the storage device) or evaluating one or more metrics of the storage device and based on the evaluation, determining to abort the reduction in declared capacity.
  • normal storage operations will continue to be performed (e.g., read, write, delete, trim, etc.). Normal storage operations include operations like trim that explicitly reduce the storage device utilization, possibly enough to merit aborting the amelioration process. Other storage activity such as garbage collection may also reduce utilization, possibly enough to merit aborting the amelioration process.
  • the amelioration process (e.g., periodically, semi- continuously, initially, finally, occasionally or irregularly) recomputes or re-evaluates a number of parameters, such as the target reduced declared capacity and/or the target amount of utilization reduction), as those parameters may change in value due to the amelioration process and/or normal storage operations (e.g., read, write, erase and trim or unmap operations).
  • a number of parameters such as the target reduced declared capacity and/or the target amount of utilization reduction
  • normal storage operations e.g., read, write, erase and trim or unmap operations.
  • one or more portions of the amelioration process such as the utilization reduction process, is re -prioritized, re-scheduled, or aborted.
  • the storage device includes (1238) one or more flash memory devices.
  • the storage device comprises a storage medium (e.g., storage medium 130, Figure 1A), and the storage medium comprises one or more nonvolatile storage devices, such as flash memory devices.
  • the storage medium e.g., storage medium 130, Figure 1A
  • the storage medium is a single flash memory device, while in other embodiments the storage medium includes a plurality of flash memory devices.
  • the storage medium includes dozens or hundreds of flash memory devices, organized in parallel memory channels, such as 16, 32 or 64 flash memory devices per memory channel, and 8, 16 or 32 parallel memory channels.
  • the non- volatile storage medium (e.g., storage medium 130, Figure 1A) includes NAND-type flash memory or NOR-type flash memory. In other embodiments, the storage medium comprises one or more other types of non-volatile storage devices.
  • Figures 13A-13D illustrate a flowchart representation of a method 1300 of managing a storage system, in accordance with some embodiments. At least in some embodiments, method 1300 is performed by a storage device (e.g., storage device 120, Figure 1 A) or one or more components of the storage device (e.g., storage controller 124, Figure 1 A), where the storage device is operatively coupled with a host system (e.g., computer system 110, Figure 1A).
  • a host system e.g., computer system 110, Figure 1A
  • method 1300 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 122-1 of management module 121-1, shown in Figures 1A and 2A-1.
  • method 1300 is performed by a storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; or data storage system 170, Figure 1C) or one or more components of the storage system (e.g., computer system 110 and/or storage device 120, Figure 1A; storage system controller 150, Figure IB; or cluster controller 180, Figure 1C).
  • a storage system e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; or data storage system 170, Figure 1C
  • components of the storage system e.g., computer system 110 and/or storage device 120, Figure 1A; storage system controller 150, Figure IB; or cluster controller 180, Figure 1C.
  • method 1300 is performed at a host (e.g., computer system 110, Figure 1A; computer system 142, Figure IB; or computer system 172, Figure 1C) and information is transmitted to a storage device (e.g., storage device 120, Figure 1A) and/or one or more subsystems of a storage system (e.g., storage system controller 150, Figure IB; and/or cluster controller 180, Figure 1C).
  • a storage device e.g., storage device 120, Figure 1A
  • one or more subsystems of a storage system e.g., storage system controller 150, Figure IB; and/or cluster controller 180, Figure 1C.
  • method 1300 is governed, at least in part, by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a host (not shown in Figures 1A, IB, and 1C).
  • the following describes method 1300 as performed by a storage device (e.g., storage device 120, Figure 1A).
  • Method 1300 includes detecting (1302) an amelioration trigger for reducing declared capacity of non- volatile memory of a storage device (e.g., storage device 120, Figure 1A; any of storage devices 160-1 to 160-m, Figure IB; or any of storage devices 194- 1 to 194-n or 194-j to 194-k, Figure 1C) of the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; or data storage system 170, Figure 1C).
  • detecting the amelioration trigger includes receiving, or generating the amelioration trigger.
  • a detection module (e.g., detection module 231, Figures 2A-1 and 2A-2; detection module 261, Figures 2B-1 and 2B-2; or detection module 291, Figures 2C-1 and 2C-2) is used to detect the amelioration trigger for reducing declared capacity of non- volatile memory of the storage device.
  • method 1300 includes performing (1304) an amelioration process to reduce declared capacity of the non- volatile memory of the storage device, by making specific logical addresses of a logical address space unavailable to a host.
  • an amelioration module (e.g., amelioration module 230, Figures 2A-1 and 2A-2; amelioration module 260, Figures 2B-1 and 2B-2; or amelioration module 290, Figures 2C-1 and 2C-2) is used to perform (1304) the amelioration process, in accordance with the amelioration trigger, to reduce declared capacity of the nonvolatile memory of the storage device, including making specific logical addresses of a logical address space unavailable to a host (e.g., with LBA reduction module 238, Figure 2D).
  • performing (1304) the amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes reducing utilization of the non-volatile memory of the storage device, for example as described above with respect to operation 608 of Figure 6.
  • the host includes (1306) a client on behalf of which data is stored in the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; or data storage system 170, Figure 1C).
  • the client is or includes an entity on behalf of which data is stored in the storage system.
  • the host is computer system 110 ( Figure 1A) or a client process, module or application executed by computer system 110, computer system 142 ( Figure IB) or a client process, module or application executed by computer system 142, and/or computer system 172 ( Figure 1C) or a client process, module or application executed by computer system 172.
  • the host includes (1308) a storage system controller of the storage system (e.g., data storage system 140, Figure IB).
  • the storage system controller controls and/or coordinates operations among one or more storage devices.
  • the host is storage system controller 150 ( Figure IB).
  • the data storage system e.g., data storage system 140, Figure IB
  • the data storage system is called a scale-up system.
  • the host includes (1310) a cluster controller (e.g., cluster controller 180, Figure 1C) of the storage system (e.g., data storage system 170, Figure 1C).
  • the cluster controller controls and/or coordinates operations among one or more data storage subsystems, as shown for example in Figure 1C, where each of the data storage subsystems may be implemented as a data storage system having one or more storage devices (e.g., data storage system 140, Figure IB).
  • the host is cluster controller 180 ( Figure 1C).
  • the data storage system e.g., data storage system 170, Figure 1C
  • the data storage system is called a scale-out system or a clustered storage system.
  • the detecting, the performing, or both the detecting and the performing are performed (1312) by the storage device (e.g., storage device 120, Figure 1A; a respective storage device 160, Figure IB; or a respective storage device 194, Figure 1C) or one or more components of the storage device (e.g., storage controller 124, Figure 1A).
  • the storage device e.g., storage device 120, Figure 1A; a respective storage device 160, Figure IB; or a respective storage device 194, Figure 1C
  • method 1300, or at least the detecting operation 1302 and/or performing operation 1304 of method 1300 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 122-1 of
  • the detecting, the performing, or both the detecting and the performing are performed (1314) by one or more subsystems of the storage system distinct from the storage device.
  • the detecting, the performing, or both the detecting and the performing are performed by a storage system controller (e.g., storage system controller 150, Figure IB) of the storage system (e.g., data storage system 140, Figure IB).
  • method 1300 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 152-1 of management module 151-1, shown in Figures IB and 2B-l .
  • processors such as the one or more processing units (CPUs) 152-1 of management module 151-1, shown in Figures IB and 2B-l .
  • the detecting, the performing, or both the detecting and the performing are performed (1316), at least in part, by the host.
  • method 1300, or at least the detecting operation 1302 and/or performing operation 1304 of method 1300 is governed at least in part by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a host (processors not shown in Figures 1A, IB, and 1C), such as the one or more processing units (CPUs) of management module 121-2 ( Figures 1A and 2A-2), management module 151-2 ( Figures IB and 2B-2), or management module 181-2 ( Figures 1C and 2C-2).
  • CPUs processing units
  • making specific logical addresses of the logical address space unavailable to the host includes making (1318) specific logical addresses of the logical address space unavailable to the host in accordance with one or more parameters for the amelioration process.
  • the one or more parameters for the amelioration process include a level of urgency for the amelioration process, a target reduced declared capacity of the non-volatile memory of the storage device, and/or a target amount of reduction in utilization of the non-volatile memory of the storage device, or any combination or subset thereof.
  • making specific logical addresses of the logical address space unavailable to the host in accordance with one or more parameters for the amelioration process includes making enough logical addresses of the logical address space unavailable to the host to meet a target amount of declared capacity reduction specified by the one or more parameters.
  • the one or more parameters of the amelioration process indicate or correspond to a target amount of reduction in storage capacity available to the host.
  • the one or more parameters take into account, or enable the amelioration process to take into account, a projected reduction in the declared capacity of the non-volatile memory of the storage device that will be needed in a predefined time period (e.g., an hour, a day, a week, or some other predefined time period).
  • making specific logical addresses of the logical address space unavailable to the host includes enumerating (1320) one or more portions of the logical address space that are to be unavailable to the host.
  • the enumerated one or more portions of the logical address space are not to be used in storage operations (e.g., read, write, trim, etc.).
  • the one or more enumerated portions of the logical address space that are (1322) unavailable to the host are determined in accordance with an algorithmic definition of which logical addresses of the logical address space are unavailable.
  • the algorithmic definition of which logical addresses of the logical address space are unavailable is a modulo or range scheme (e.g., every 10th logical address is determined to be unavailable or all logical addresses in the range 50 to 100 are determined to be unavailable).
  • the one or more enumerated portions of the logical address space that are (1324) unavailable to the host are determined in accordance with a determination of which logical addresses of the logical address space are unused. In some embodiments, one or more unused logical addresses are selected and marked as unavailable.
  • making specific logical addresses of the logical address space unavailable to the host includes (1326): specifying a first list of logical addresses of the logical address space that are in use (e.g., live data 334); and specifying a second list of logical addresses of the logical address space that are available for use (e.g., portions of not live data 332 and/or portions of trimmed LBA space 330), where logical addresses of the logical address space not specified on the first list and not specified on the second list are logical addresses of the logical address space unavailable to the host.
  • a file system generally maintains, in its metadata, a list of the logical block addresses that are in use by the files of the file system. Additionally, the file system generally maintains a list of free blocks (i.e., not live data 332 and trimmed LBA space 330). A subset of the list of free blocks would be suitable candidates for being made unavailable and could be removed from the list of free blocks.
  • the first list and/or the second list is maintained (1328) at the host.
  • the first list and/or the second list is maintained at computer system 110 ( Figure 1A), computer system 142 ( Figure IB), computer system 172 ( Figure 1C), storage system controller 150 (Figure IB), or cluster controller 180 ( Figure 1C) depending on the implementation.
  • the first list and/or the second list is maintained (1330) at the storage device (e.g., storage device 120, Figure 1A; any of storage devices 160-1 to 160-m, Figure IB; or any of storage devices 194-1 to 194-n or 194-j to 194-k, Figure 1C).
  • the storage device e.g., storage device 120, Figure 1A; any of storage devices 160-1 to 160-m, Figure IB; or any of storage devices 194-1 to 194-n or 194-j to 194-k, Figure 1C).
  • the first list and/or the second list is maintained (1332) external to the storage device.
  • the first list and/or the second list is maintained in memory associated with storage controller 124 ( Figure 1A), storage system controller 150 (Figure IB), cluster controller 180 ( Figure 1C), or another location communicatively coupled with data storage system 100 ( Figure 1A), data storage system 140 ( Figure IB), or data storage system 170 ( Figure 1C).
  • a list of the specific logical addresses of the logical address space unavailable to the host is maintained (1334) at the host.
  • list of the specific logical addresses of the logical address space unavailable to the host is maintained at computer system 110 ( Figure 1A), computer system 142 ( Figure IB), computer system 172 ( Figure 1C), storage system controller 150 (Figure IB), or cluster controller 180 ( Figure 1C) depending on the implementation.
  • a list of the specific logical addresses of the logical address space unavailable to the host is maintained (1336) at the storage device (e.g., storage device 120, Figure 1A; any of storage devices 160-1 to 160-m, Figure IB; or any of storage devices 194-1 to 194-n or 194-j to 194-k, Figure 1C).
  • the storage device e.g., storage device 120, Figure 1A; any of storage devices 160-1 to 160-m, Figure IB; or any of storage devices 194-1 to 194-n or 194-j to 194-k, Figure 1C).
  • a list of the specific logical addresses of the logical address space unavailable to the host is maintained (1338) external to the storage device.
  • the list of the specific logical addresses of the logical address space unavailable to the host is maintained in memory associated with storage controller 124 ( Figure 1 A), storage system controller 150 (Figure IB), cluster controller 180 ( Figure 1C), or another location communicatively coupled with data storage system 100 ( Figure 1A), data storage system 140 ( Figure IB), or data storage system 170 ( Figure 1C).
  • the host selects (1340) the specific logical addresses of the logical address space to make unavailable to the host. For example, in a file system, the list of free blocks are preferred candidates for the specific logical addresses to be made unavailable to the host.
  • the specific logical addresses of the logical address space unavailable to the host are selected (1342) to minimize performance degradation. For example, in a file system, members of the list of free blocks are preferentially selected if the logical addresses associated with them are contained in the trimmed LB A space 330 ( Figure 3).
  • the specific logical addresses of the logical address space unavailable to the host are selected (1344) to minimize overhead from garbage collection. For example, in a file system, physical pages associated with the logical addresses that are members of the list of free blocks that are in the not live data (332) portion of the logical address space are examined to preferentially select logical addresses to become unavailable to reduce garbage collection overhead.
  • making (1304) specific logical addresses of the logical address space unavailable to the host includes copying data stored at a first set of physical addresses associated with a first set of logical address entries in a mapping table to a second set of physical addresses associated with a second set of logical address entries in the mapping table, and updating the first set of logical address entries in the mapping table, the mapping table used to translate logical addresses in the logical address space to physical addresses in a physical address space of the storage device.
  • the first set of logical address entries includes a single logical address entry of the mapping table.
  • the first set of logical address entries includes a contiguous set of logical address entries of the mapping table.
  • the first set of logical address entries includes a non-contiguous set of logical address entries of the mapping table.
  • the second set of logical address entries includes a single logical address entry of the mapping table.
  • the second set of logical address entries includes a contiguous set of logical address entries of the mapping table.
  • the second set of logical address entries includes a non-contiguous set of logical address entries of the mapping table.
  • data stored at a physical address associated with LBA 5 is copied to a physical address associated with LBA 100 (e.g., in mapping table 402, Figure 4), and the logical address entry for LBA 5 is updated.
  • data stored at physical addresses associated with LBA 150 through LBA 157 is copied to physical addresses associated with LBA 10 through LBA 17 (e.g., in mapping table 402, Figure 4), and logical address entries for LBA 150 through LBA 157 are updated.
  • data stored at physical addresses associated with LBA 138, LBA 252, and LBA 388 is copied to physical addresses associated with LBA 56, LBA 21, and LBA 7, respectively, and logical address entries for LBA 138, LBA 252, and LBA 388 are updated.
  • LBA 138, LBA 252, and LBA 388 are updated.
  • data stored at physical addresses associated with LBA 138, LBA 252, and LBA 388 is copied to physical addresses associated with LBA 56 through LBA 58, and logical address entries for LBA 138, LBA 252, and LBA 388 are updated.
  • updating the first set of logical address entries in the mapping table includes invalidating one or more logical address entries of the first set of logical address entries. In some embodiments, updating the first set of logical address entries in the mapping table includes invalidating one logical address entry of the first set of logical address entries. In some embodiments, updating the first set of logical address entries in the mapping table includes invalidating some (but not all) of the logical address entries of the first set of logical address entries. In some embodiments, updating the first set of logical address entries in the mapping table includes invalidating all of the logical address entries of the first set of logical address entries.
  • performing (1304) the amelioration process to reduce declared capacity of the non-volatile memory of the storage device further includes advertising (1346) a reduced declared capacity of the non-volatile memory of the storage device.
  • an advertising module e.g., advertising module 239, Figure 2D
  • the storage device, or a corresponding storage controller, cluster controller, management module or data storage system sends a message to the host advertising the reduced declared capacity of the non-volatile memory of the storage device.
  • advertising the reduced declared capacity is accomplished by sending an interrupt or other message to the host.
  • advertising the reduced declared capacity is accomplished by receiving a query from a host to which the storage device is operatively coupled, and in response to receiving the query, reporting the reduced declared capacity of the non-volatile memory of the storage device.
  • the host is configured to periodically query the storage system, storage controller, management module, cluster controller, or storage device, for example, for a system or device health status.
  • advertising the reduced declared capacity is accomplished by receiving a command (e.g., a storage read or write command) from a host to which the storage device is operatively coupled, and in response to receiving the command, sending a response to the command that includes a notification of the reduced declared capacity of the non-volatile memory of the storage device.
  • a command e.g., a storage read or write command
  • advertising the reduced declared capacity is accomplished by receiving a command (e.g., a storage read or write command) from a host to which the storage device is operatively coupled, and in response to receiving the command, sending both a response to the command and a notification that prompts the host to obtain information, including the reduced declared capacity of the non-volatile memory of the storage device, from the storage device or from the data storage system that includes the storage device.
  • a command e.g., a storage read or write command
  • the mechanism used for returning a notification when responding to a command from the host is a SCSI deferred error or deferred error response code.
  • method 1300 includes (1348): after beginning performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device, detecting an indication to abort the reduction in declared capacity of the non- volatile memory of the storage device; and, in response to detecting the indication to abort the reduction in declared capacity of the non- volatile memory of the storage device, aborting performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • detecting the indication to abort is herein defined to mean either receiving a signal to abort the reduction in declared capacity (e.g., receiving the signal from a controller of the storage device or a storage system controller of a storage system that includes the storage device) or evaluating one or more metrics of the storage device and based on the evaluation, determining to abort the reduction in declared capacity.
  • normal storage operations will continue to be performed (e.g., read, write, delete, trim, etc.). Normal storage operations include operations like trim that explicitly reduce the storage device utilization, possibly enough to merit aborting the amelioration process. Other storage activity such as garbage collection may also reduce utilization, possibly enough to merit aborting the amelioration process.
  • the amelioration process (e.g., periodically, semi- continuously, initially, finally, occasionally or irregularly) recomputes or re-evaluates a number of parameters, such as the target reduced declared capacity and/or the target amount of utilization reduction), as those parameters may change in value due to the amelioration process and/or normal storage operations (e.g., read, write, erase and trim or unmap operations).
  • a number of parameters such as the target reduced declared capacity and/or the target amount of utilization reduction
  • normal storage operations e.g., read, write, erase and trim or unmap operations.
  • one or more portions of the amelioration process such as the utilization reduction process, is re-prioritized, re-scheduled, or aborted.
  • the storage device includes (1350) one or more flash memory devices.
  • the storage device comprises a storage medium (e.g., storage medium 130, Figure 1A), and the storage medium comprises one or more nonvolatile storage devices, such as flash memory devices.
  • the storage medium e.g., storage medium 130, Figure 1A
  • the storage medium is a single flash memory device, while in other embodiments the storage medium includes a plurality of flash memory devices.
  • the storage medium includes dozens or hundreds of flash memory devices, organized in parallel memory channels, such as 16, 32 or 64 flash memory devices per memory channel, and 8, 16 or 32 parallel memory channels.
  • the non- volatile storage medium (e.g., storage medium 130, Figure 1A) includes NAND-type flash memory or NOR-type flash memory. In other embodiments, the storage medium comprises one or more other types of non-volatile storage devices.
  • FIGS 14A-14C illustrate a flowchart representation of a method 1400 of managing a storage system, in accordance with some embodiments. At least in some embodiments, method 1400 performed by a storage device (e.g., storage device 120, Figure 1 A) or one or more components of the storage device (e.g., storage controller 124 and/or storage medium 130, Figure 1A), where the storage device is operatively coupled with a host system (e.g., computer system 110, Figure 1A).
  • a storage device e.g., storage device 120, Figure 1 A
  • a host system e.g., computer system 110, Figure 1A
  • method 1400 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 122-1 of management module 121-1, shown in Figures 1A and 2A- 1.
  • method 1400 is performed by a storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; and/or data storage system 170, Figure 1C) or one or more components of the storage system (e.g., computer system 110 and/or storage device 120, Figure 1A; storage system controller 150, Figure IB; and/or cluster controller 180, Figure 1C).
  • a storage system e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; and/or data storage system 170, Figure 1C
  • components of the storage system e.g., computer system 110 and/or storage device 120, Figure 1A; storage system controller 150, Figure IB; and/or cluster controller 180
  • method 1400 is performed at a host (e.g., computer system 110, Figure 1A; computer system 142, Figure IB; and/or computer system 172, Figure 1C) and information is transmitted to a storage device (e.g., storage device 120, Figure 1 A) and/or one or more subsystems of a storage system (e.g., storage system controller 150, Figure IB; and/or cluster controller 180, Figure 1C).
  • a storage device e.g., storage device 120, Figure 1 A
  • one or more subsystems of a storage system e.g., storage system controller 150, Figure IB; and/or cluster controller 180, Figure 1C.
  • method 1400 is governed, at least in part, by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a host (not shown in Figures 1A, IB, and 1C).
  • a storage device e.g., storage device 120, Figure 1A.
  • Method 1400 includes detecting (1402) an amelioration trigger for reducing declared capacity of non- volatile memory of a storage device (e.g., storage device 120, Figure 1A; any of storage devices 160-1 to 160-m, Figure IB; or any of storage devices 194- 1 to 194-n or 194-j to 194-k, Figure 1C) of the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; or data storage system 170, Figure 1C).
  • detecting the amelioration trigger includes receiving, or generating the amelioration trigger.
  • a detection module (e.g., detection module 231, Figures 2A-1 and 2A-2; detection module 261, Figures 2B-1 and 2B-2; or detection module 291, Figures 2C-1 and 2C-2) is used to detect the amelioration trigger for reducing declared capacity of non- volatile memory of the storage device.
  • method 1400 includes performing (1404) an amelioration process to reduce declared capacity of the non- volatile memory of the storage device, by reducing a count of logical addresses of a logical address space available to a host.
  • an amelioration module (e.g., amelioration module 230, Figures 2A-1 and 2A-2; amelioration module 260, Figures 2B-1 and 2B-2; or amelioration module 290, Figures 2C-1 and 2C-2) is used to perform (1404) the amelioration process, in accordance with the amelioration trigger, to reduce declared capacity of the nonvolatile memory of the storage device, including reducing a count of logical addresses of a logical address space available to a host (e.g., with LBA reduction module 238, Figure 2D).
  • amelioration module e.g., amelioration module 230, Figures 2A-1 and 2A-2; amelioration module 260, Figures 2B-1 and 2B-2; or amelioration module 290, Figures 2C-1 and 2C-2
  • performing (1404) the amelioration process to reduce declared capacity of the non-volatile memory of the storage device includes reducing utilization of the non-volatile memory of the storage device, for example as described above with respect to operation 608 of Figure 6.
  • the host includes (1406) a client on behalf of which data is stored in the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; or data storage system 170, Figure 1C).
  • the client is or includes an entity on behalf of which data is stored in the storage system.
  • the host is computer system 110 ( Figure 1A) or a client process or application executed by computer system 110, computer system 142 ( Figure IB) or a client process or application executed by computer system 142, and/or computer system 172 ( Figure 1C) or a client process or application executed by computer system 172.
  • the host includes (1408) a storage system controller of the storage system (e.g., data storage system 140, Figure IB).
  • the storage system controller controls and/or coordinates operations among one or more storage devices.
  • the host is storage system controller 150 ( Figure IB).
  • the data storage system e.g., data storage system 140, Figure IB
  • the data storage system is called a scale-up system.
  • the host includes (1410) a cluster controller (e.g., cluster controller 180, Figure 1C) of the storage system (e.g., data storage system 170, Figure 1C).
  • the cluster controller controls and/or coordinates operations among one or more data storage subsystems, as shown for example in Figure 1C, where each of the data storage subsystems may be implemented as a data storage system having one or more storage devices (e.g., data storage system 140, Figure IB).
  • the host is cluster controller 180 ( Figure 1C).
  • the data storage system e.g., data storage system 170, Figure 1C
  • the data storage system is called a scale-out system or sometimes known as a clustered storage system.
  • the detecting, the performing, or both the detecting and the performing are performed (1412) by the storage device (e.g., storage device 120, Figure 1A; a respective storage device 160, Figure IB; or a respective storage device 194, Figure 1C) or one or more components of the storage device (e.g., storage controller 124, Figure 1 A).
  • method 1400, or at least the detecting operation 1402 and/or performing operation 1404 of method 1400 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 122-1 of
  • the detecting, the performing, or both the detecting and the performing are performed (1414) by one or more subsystems of the storage system distinct from the storage device.
  • the detecting, the performing, or both the detecting and the performing are performed by a storage system controller (e.g., storage system controller 150, Figure IB) of the storage system (e.g., data storage system 140, Figure IB).
  • method 1400 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 152-1 of management module 151-1, shown in Figures IB and 2B-l .
  • processors such as the one or more processing units (CPUs) 152-1 of management module 151-1, shown in Figures IB and 2B-l .
  • the detecting, the performing, or both the detecting and the performing are performed (1416), at least in part, by the host.
  • method 1400, or at least the detecting operation 1402 and/or performing operation 1404 of method 1400 is governed at least in part by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a host (processors not shown in Figures 1A, IB, and 1C), such as the one or more processing units (CPUs) of management module 121-2 ( Figures 1A and 2A-2), management module 151-2 ( Figures IB and 2B-2), or management module 181-2 ( Figures 1C and 2C-2).
  • reducing the count of logical addresses of the logical address space available to the host includes reducing (1418) the count of logical addresses of the logical address space available to the host in accordance with one or more parameters for the amelioration process.
  • the one or more parameters for the amelioration process include a level of urgency for the amelioration process, a target reduced declared capacity of the non-volatile memory of the storage device, and/or a target amount of reduction in utilization of the non-volatile memory of the storage device, or any combination or subset thereof.
  • reducing the count of logical addresses of the logical address space available to the host in accordance with one or more parameters for the amelioration process includes reducing the count of logical addresses of the logical address space enough to meet a target amount of declared capacity reduction specified by the one or more parameters.
  • the one or more parameters of the amelioration process indicate or correspond to a target amount of reduction in storage capacity available to the host.
  • the one or more parameters take into account, or enable the amelioration process to take into account, a projected reduction in the declared capacity of the non- volatile memory of the storage device that will be needed in a predefined time period (e.g., an hour, a day, a week, or some other predefined time period).
  • reducing the count of logical addresses of the logical address space available to the host includes maintaining (1420) a count of logical addresses of the logical address space that are currently in use. In some embodiments, the count is maintained by the host. In some embodiments, the count is maintained by the storage device, and in some embodiments the count is maintained by the storage system.
  • the host enforces (1422) that a count of logical addresses in use by the host of the logical address space does not exceed the count of logical addresses of the logical address space available to the host.
  • the storage device enforces (1424) that a count of logical addresses in use by the host of the logical address space does not exceed the count of logical addresses of the logical address space available to the host.
  • method 1400 further includes returning (1426) an error to the host, in accordance with a determination that a write command from the host would cause the count of logical addresses in use by the host to exceed the count of logical addresses available to the host. For example, a storage device with a declared capacity of six logical addresses and currently storing data at logical addresses 1, 3, 5, 8, 15 and 31415923 would return an error upon a request to write to logical address 2.
  • performing (1404) the amelioration process to reduce declared capacity of the non-volatile memory of the storage device further includes advertising (1428) a reduced declared capacity of the non-volatile memory of the storage device.
  • an advertising module e.g., advertising module 239, Figure 2D
  • the storage device, or a corresponding storage controller, cluster controller, management module or data storage system sends a message to the host advertising the reduced declared capacity of the non-volatile memory of the storage device.
  • advertising the reduced declared capacity is accomplished by sending an interrupt or other message to the host.
  • advertising the reduced declared capacity is accomplished by receiving a query from a host to which the storage device is operatively coupled, and in response to receiving the query, reporting the reduced declared capacity of the non-volatile memory of the storage device.
  • the host is configured to periodically query the storage system, storage controller, management module, cluster controller, or storage device, for example, for a system or device health status.
  • advertising the reduced declared capacity is accomplished by receiving a command (e.g., a storage read or write command) from a host to which the storage device is operatively coupled, and in response to receiving the command, sending a response to the command that includes a notification of the reduced declared capacity of the non-volatile memory of the storage device.
  • a command e.g., a storage read or write command
  • advertising the reduced declared capacity is accomplished by receiving a command (e.g., a storage read or write command) from a host to which the storage device is operatively coupled, and in response to receiving the command, sending both a response to the command and a notification that prompts the host to obtain information, including the reduced declared capacity of the non-volatile memory of the storage device, from the storage device or from the data storage system that includes the storage device.
  • a command e.g., a storage read or write command
  • the mechanism used for returning a notification when responding to a command from the host is a SCSI deferred error or deferred error response code.
  • method 1400 includes (1430): after beginning performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device, detecting an indication to abort the reduction in declared capacity of the non- volatile memory of the storage device; and, in response to detecting the indication to abort the reduction in declared capacity of the non- volatile memory of the storage device, aborting performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • detecting the indication to abort is herein defined to mean either receiving a signal to abort the reduction in declared capacity (e.g., receiving the signal from a controller of the storage device or a storage system controller of a storage system that includes the storage device) or evaluating one or more metrics of the storage device and based on the evaluation, determining to abort the reduction in declared capacity.
  • normal storage operations will continue to be performed (e.g., read, write, delete, trim, etc.). Normal storage operations include operations like trim that explicitly reduce the storage device utilization, possibly enough to merit aborting the amelioration process. Other storage activity such as garbage collection may also reduce utilization, possibly enough to merit aborting the amelioration process.
  • the amelioration process (e.g., periodically, semi- continuously, initially, finally, or irregularly) recomputes or re-evaluates a number of parameters, such as the target reduced declared capacity and/or the target amount of utilization reduction), as those parameters may change in value due to the amelioration process and/or normal storage operations (e.g., read, write, erase and trim or unmap operations).
  • a number of parameters such as the target reduced declared capacity and/or the target amount of utilization reduction
  • normal storage operations e.g., read, write, erase and trim or unmap operations.
  • one or more portions of the amelioration process such as the utilization reduction process, is re-prioritized, re-scheduled, or aborted.
  • the storage device includes (1432) one or more flash memory devices.
  • the storage device comprises a storage medium (e.g., storage medium 130, Figure 1A), and the storage medium comprises one or more nonvolatile storage devices, such as flash memory devices.
  • the storage medium e.g., storage medium 130, Figure 1A
  • the storage medium is a single flash memory device, while in other embodiments the storage medium includes a plurality of flash memory devices.
  • the storage medium includes dozens or hundreds of flash memory devices, organized in parallel memory channels, such as 16, 32 or 64 flash memory devices per memory channel, and 8, 16 or 32 parallel memory channels.
  • the non- volatile storage medium (e.g., storage medium 130, Figure 1A) includes NAND-type flash memory or NOR-type flash memory. In other embodiments, the storage medium comprises one or more other types of non-volatile storage devices.
  • Figures 15A-15D illustrate a flowchart representation of a method 1500 of managing a storage system, in accordance with some embodiments.
  • the method 1500 includes detecting (1502) an amelioration trigger for reducing declared capacity of non-volatile memory of a storage device (e.g., storage device 120 of Figure 1A, or any of storage devices 160-1 to 160-m of Figure IB, or any of storage devices 194-1 to 194-n or 194-j to 194-k of Figure 1C) of the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; or data storage system 170, Figure 1C), and in accordance with the detected amelioration trigger, performing (1504) an amelioration process to reduce declared capacity of the non-volatile memory of the storage device, including: altering an encoding format of at least a portion of the non- volatile memory of the storage device, and reducing declared capacity of the non-volatile memory of the storage device.
  • an amelioration trigger for reducing declared capacity of
  • the encoding format of data comprises the physical encoding of data in the storage media combined with the logical encoding of data which uses one or more of the redundancy mechanisms as described above.
  • the physical encoding format of a portion of non-volatile memory is a physical encoding format of the memory cells comprising the portion of nonvolatile memory, and indicates the number of bits of information that can be stored per memory cell.
  • examples of physical encoding formats include a single-level flash memory cell (SLC), a multi-level flash memory cell (MLC), a triple-level memory cell (TLC), and a quad-level memory cell (QLC), capable of storing up to one, two, three, and four bits per memory cell, respectively.
  • the physical encoding format can further include other physical encoding formats for a non-integer number of bits of information per memory cell. Altering of the encoding format may involve altering the physical encoding format or the redundancy mechanism(s) or both the physical encoding format and the redundancy mechanism(s). In some embodiments, altering of the encoding format is a multi-step sequence. In a first step of this sequence an indication of the new encoding format is made or recorded. This indication will inform subsequent operations to use the indicated (e.g., lower density) encoding format. In a subsequent step, e.g., garbage collection, data will be recorded with the new encoding format.
  • reducing declared capacity of non-volatile memory of the storage device includes reducing storage capacity, of the non-volatile memory of the storage device, available to the host.
  • the declared capacity of non-volatile memory of the storage device is sometimes called the advertised capacity, and is typically used by the operating system and/or a file system of the host as the maximum capacity that the host's operating system or file system will attempt to allocate.
  • the amelioration trigger is detected (1502) by an amelioration module (e.g., amelioration module 230 of Figure 2A-1, 2A-2 or 2D, 260 of Figure 2B-1 or 2B-2, or 290 of Figure 2C-1 or 2C-2), or a component thereof (e.g., detection module 231 of Figure 2A-1, 2A-2 or 2D, 261 of Figure 2B-1 or 2B-2, or 291 of Figure 2C-1 or 2C-2).
  • detecting the amelioration trigger includes receiving or generating the amelioration trigger.
  • an amelioration module e.g., amelioration module
  • the amelioration module includes a module (e.g., utilization module 232, Figure 2D) for altering an encoding format of at least a portion of the non-volatile memory of the storage device, and a capacity module (e.g., capacity module 234) for reducing declared capacity of the non- volatile memory of the storage device.
  • a module e.g., utilization module 232, Figure 2D
  • a capacity module e.g., capacity module 234 for reducing declared capacity of the non- volatile memory of the storage device.
  • the detecting, the performing, or both the detecting and the performing are performed (1506) by the storage device (e.g., storage device 120, Figure 1A, storage device 160, Figure IB, or storage device 194, Figure 1C) or one or more components of the storage device (e.g., storage controller 124, Figure 1A).
  • method 1500, or at least the detecting operation 1502 and/or performing operation 1504 of method 1500 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 122-1 of management module 121- 1 , shown in Figures 1 A and 2 A- 1.
  • CPUs processing units
  • the detecting, the performing, or both the detecting and the performing are performed (1508) by one or more subsystems of the storage system distinct from the storage device.
  • the detecting, the performing, or both the detecting and the performing are performed by a storage system controller (e.g., storage system controller 150, Figure IB) of the storage system (e.g., data storage system 140, Figure IB).
  • method 1500 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 152-1 of management module 151-1, shown in Figures IB and 2B-l .
  • processors such as the one or more processing units (CPUs) 152-1 of management module 151-1, shown in Figures IB and 2B-l .
  • the detecting, the performing, or both the detecting and the performing are performed (1510), at least in part, by the host.
  • method 1500, or at least the detecting operation 1502 and/or performing operation 1504 of method 1500 is governed at least in part by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a host (processors not shown in Figures 1A, IB and 1C), such as the one or more processing units (CPUs) of management module 121-2 ( Figures 1A and 2A-2); or management module 151-2 ( Figures IB and 2B-2); or management module 181-2 ( Figures 1C and 2C-2).
  • the host e.g., computer system 110 of Figure 1A, computer system 142 of Figure IB, or computer system 172 of Figure 1C
  • the host includes (1512) a client on behalf of which data is stored in the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; data storage system 170, Figure 1C).
  • the client is or includes an entity on behalf of which data is stored in the storage system.
  • the host includes (1514) a storage system controller
  • the storage system controller controls and/or coordinates operations among one or more storage devices.
  • the data storage system e.g., data storage system 140, Figure IB
  • the data storage system is called a scale-up system.
  • the host includes (1516) a cluster controller (e.g., cluster controller 180, Figure 1C) of the storage system (e.g., data storage system 170, Figure 1C).
  • the cluster controller controls and/or coordinates operations among one or more data storage subsystems, as shown for example in Figure 1C, where each of the data storage subsystems may be implemented as a data storage system having one or more storage devices (e.g., data storage system 140, Figure IB).
  • the data storage system e.g., data storage system 170, Figure 1C
  • the data storage system is called a scale-out system or a clustered storage system.
  • altering (1504) the encoding format of at least a portion of the non- volatile memory of the storage device includes altering (1518) the encoding format of at least a portion of the non- volatile memory of the storage device in accordance with one or more parameters for the amelioration process.
  • storage controller 124 ( Figure 1 A) or a component thereof (e.g., amelioration module 230, Figure 2D) alters and/or coordinates the alteration of the encoding format of at least a portion of the non-volatile memory of the storage device (e.g., storage medium 130 of storage device 120, Figure 1 A).
  • the one or more parameters for the amelioration process include a level of urgency for the amelioration process, a target reduced declared capacity of the non-volatile memory of the storage device, and/or a target amount of reduction in utilization of the non-volatile memory of the storage device, or any combination or subset thereof.
  • altering the encoding format of at least a portion of the nonvolatile memory of the storage device in accordance with one or more parameters for the amelioration process includes altering the encoding format only enough to meet a target amount of declared capacity reduction specified by the one or more parameters. Stated another way, the one or more parameters of the amelioration process indicate or correspond to a target amount of reduction in storage capacity available to the host.
  • the encoding format of a portion of the non- volatile memory is altered from TLC to MLC if a target amount of declared capacity reduction can be met by doing so, rather than altering the encoding format from TLC to SLC.
  • the one or more parameters for the amelioration process take into account, or enable the amelioration process to take into account, a projected reduction in the declared capacity of the non- volatile memory of the storage device that will be needed in a predefined time period (e.g., an hour, a day, a week, or any other predefined period).
  • the parameters of the amelioration process indicate a specific reduction in declared capacity and a reduction in utilization.
  • a host might use the trim operation to reduce utilization and then direct the storage device to reduce its declared capacity.
  • the storage device alters the encoding format of a selected portion of the non- volatile storage media (as noted above altering may itself be a multi-step process).
  • the amelioration process steps of reducing utilization, reducing declared capacity and altering the encoding format may occur in any order and may be partially or completely overlapping (e.g., performed in parallel).
  • the amelioration process may change the declared capacity of the storage device prior to the completion of the reduction in utilization, wherein the storage device operates with reduced over-provisioning until the utilization reduction is completed.
  • altering (1504) the encoding format of at least a portion of the non- volatile memory of the storage device includes altering (1520) the encoding format from a higher-density physical encoding format to a lower-density physical encoding format.
  • the encoding format of selectable portion 131 of storage medium 130 is altered from MLC to SLC (e.g., a higher-density physical encoding format to a lower-density physical encoding format) such that the memory cells comprising selectable portion 131 store fewer bits per cell than under the previous encoding format.
  • altering (1504) the encoding format of at least a portion of the non- volatile memory of the storage device includes altering (1522) a number of states per memory cell from a higher number of states to a lower number of states.
  • states of a memory cell represent sequential voltage ranges within the total voltage range of a memory cell that represent different bit-tuples (e.g., in Figure 5B, sequential voltage ranges 31 1 , 312, 313, 314 represent corresponding bit-tuples “1 1 ,” “01 ,” “00,” and "10.”).
  • the number of bits of information in a memory cell of a nonvolatile memory is a function of the possible states that can be stored per memory cell, which is determined by the physical encoding format of the memory cell. Specifically, the number of bits of information that be stored per cell is equal to log 2 N, where N is the corresponding maximum number of possible states per cell. Therefore, in some embodiments, altering a number of states per memory cell includes altering the number of detectable voltage levels for a memory cell, thereby altering the amount of information stored in a memory cell.
  • altering (1504) the encoding format of at least a portion of the non- volatile memory of the storage device includes altering (1524) the encoding format from a Triple-Level Cell (TLC) format to a Multi-Level Cell (MLC) format.
  • TLC Triple-Level Cell
  • MLC Multi-Level Cell
  • altering (1504) the encoding format of at least a portion of the non- volatile memory of the storage device includes altering (1526) the encoding format from a Triple-Level Cell (TLC) format to a Single-Level Cell (SLC) format.
  • altering (1504) the encoding format of at least a portion of the non- volatile memory of the storage device includes altering (1528) the encoding format from a Multi-Level Cell (MLC) format to a Single-Level Cell (SLC) format.
  • MLC Multi-Level Cell
  • SLC Single-Level Cell
  • performing (1504) the amelioration process to reduce declared capacity of the non- volatile memory of the storage device includes reducing (1530) utilization of the non-volatile memory of the storage device.
  • the amelioration process (606) includes a utilization reduction process (608) (e.g., performed by utilization module 232, Figures 2A-1 and 2A-2, utilization module 262, Figures 2B-1 and 2B-2, or utilization module 292, Figures 2C-1 and 2C-2).
  • Examples of ways in which utilization of the non- volatile memory of the storage device may be reduced include trimming or unmapping not live data, deleting discardable data (e.g., temporary data files) used by a host, and/or moving data from the storage device to one or more other storage devices.
  • altering (1504) the encoding format of at least a portion of the non-volatile memory of the storage device includes altering (1532) the encoding format of a memory portion of a plurality of memory portions of the non-volatile memory of the storage device.
  • the encoding format of memory cells comprising bank 1 e.g., Bank 420-1 of the plurality of banks, Banks 420-1 through 420-q
  • the encoding format of memory cells comprising bank 1 is altered from MLC to SLC, resulting in the memory cells of bank 1 storing fewer bits per cell than the memory cells of other banks of the non-volatile memory.
  • altering (1504) the encoding format of at least a portion of the non-volatile memory of the storage device includes altering (1534) the encoding format of all client data of the non-volatile memory of the storage device.
  • the encoding format of all memory cells comprising storage medium 130 ( Figure 1A) is altered from MLC to SLC, resulting in those memory cells of storage medium 130 storing one bit of information per cell, rather than two bits.
  • a storage device may store all client data in a high-density encoding format (e.g., TLC or MLC) but store storage device metadata (e.g., mapping table 402) in a different encoding format (e.g., SLC).
  • the amelioration process may result in all of the storage media associated with the client data being converted to a lower-density encoding format while the storage device metadata remains in its original encoding format.
  • the retention of the client data is optional.
  • the amelioration process may be performed much faster if the client data is not retained, similar to a SCSI low-level format operation.
  • performing the amelioration process (1504) to reduce declared capacity of the non-volatile memory of the storage device further includes advertising (1536) a reduced declared capacity of the non- volatile memory of the storage device.
  • storage controller 124 ( Figure 1 A) or a component thereof (e.g., advertising module 239 of capacity module 234, Figure 2D) advertises/or coordinates the advertisement of a reduced declared capacity of non- volatile memory of a storage device (e.g., storage medium 130 of storage device 120, Figure 1A) or storage subsystem.
  • the storage device, or a corresponding storage controller, cluster controller, management module or data storage system sends a message to the host advertising the reduced declared capacity of the non-volatile memory of the storage device.
  • advertising the reduced declared capacity is accomplished by sending an interrupt or other message to the host.
  • advertising the reduced declared capacity is accomplished by receiving a query from a host to which the storage device is operatively coupled, and in response to receiving the query, reporting the reduced declared capacity of the non-volatile memory of the storage device.
  • the host is configured to periodically query the storage system, storage controller, management module, cluster controller or storage device, for example for a system or device health status.
  • advertising the reduced declared capacity is accomplished by receiving a command (e.g., a storage read or write command) from a host to which the storage device is operatively coupled, and in response to receiving the command, sending a response to the command that includes a notification of the reduced declared capacity of the non-volatile memory of the storage device.
  • a command e.g., a storage read or write command
  • advertising the reduced declared capacity is accomplished by receiving a command (e.g., a storage read or write command) from a host to which the storage device is operatively coupled, and in response to receiving the command, sending both a response to the command and a notification that prompts the host to obtain information, including the reduced declared capacity of the non-volatile memory of the storage device, from the storage device or from the data storage system that includes the storage device.
  • a command e.g., a storage read or write command
  • the mechanism used for returning a notification when responding to a command from the host is a SCSI deferred error or deferred error response code.
  • the method after beginning performance of the amelioration process (1504) to reduce declared capacity of the non-volatile memory of the storage device, the method includes detecting an indication (1538) to abort the reduction in declared capacity of the non-volatile memory of the storage device; and in response to detecting the indication to abort the reduction in declared capacity of the non-volatile memory of the storage device, aborting (1540) performance of the amelioration process to reduce declared capacity of the non-volatile memory of the storage device.
  • detecting the indication to abort is herein defined to mean either receiving a signal to abort the reduction in declared capacity (e.g., receiving the signal from a controller of the storage device or a storage system controller of a storage system that includes the storage device) or evaluating one or more metrics of the storage device and based on the evaluation, determining to abort the reduction in declared capacity.
  • normal storage operations will continue to be performed (e.g., read, write, delete, trim, etc.). Normal storage operations include operations like trim that explicitly reduce the storage device utilization, possibly enough to merit aborting the amelioration process. Other storage activity such as garbage collection may also reduce utilization, possibly enough to merit aborting the amelioration process.
  • the amelioration process (e.g., periodically, semi- continuously, initially, finally, occasionally or irregularly) recomputes or re-evaluates a number of parameters, such as the target reduced declared capacity and/or the target amount of utilization reduction), as those parameters may change in value due to the amelioration process and/or normal storage operations (e.g., read, write, erase and trim or unmap operations).
  • a number of parameters such as the target reduced declared capacity and/or the target amount of utilization reduction
  • normal storage operations e.g., read, write, erase and trim or unmap operations.
  • one or more portions of the amelioration process such as the utilization reduction process, is re -prioritized, re-scheduled, or aborted.
  • the storage device includes (1542) one or more flash memory devices.
  • the storage device comprises a storage medium (e.g., storage medium 130, Figure 1A), and the storage medium comprises one or more nonvolatile storage devices, such as flash memory devices.
  • the storage medium e.g., storage medium 130, Figure 1A
  • the storage medium is a single flash memory device, while in other embodiments the storage medium includes a plurality of flash memory devices.
  • the storage medium includes dozens or hundreds of flash memory devices, organized in parallel memory channels, such as 16, 32 or 64 flash memory devices per memory channel, and 8, 16 or 32 parallel memory channels.
  • the non- volatile storage medium includes NAND-type flash memory or NOR-type flash memory.
  • the storage medium comprises one or more other types of non-volatile storage devices.
  • Figures 16A-16C illustrate a flowchart representation of a method 1600 of managing a storage system, in accordance with some embodiments.
  • the method 1600 includes detecting (1602) an amelioration trigger for reducing declared capacity of non-volatile memory of a storage device (e.g., storage device 120 of Figure 1A, or any of storage devices 160-1 to 160-m of Figure IB, or any of storage devices 194-1 to 194-n or 194-j to 194-k of Figure 1C) of the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; or data storage system 170, Figure 1C), and in accordance with the detected amelioration trigger, performing (1604) an amelioration process to reduce declared capacity of the non-volatile memory of the storage device, including: deleting from the storage device discardable data that is used by a host (e.g., computer system 110 of Figure 1A, computer system 142 of Figure IB, or computer system 172 of Figure 1C
  • a host
  • reducing declared capacity of non- volatile memory of the storage device includes reducing storage capacity, of the non-volatile memory of the storage device, available to the host.
  • the declared capacity of non-volatile memory of the storage device is sometimes called the advertised capacity, and is typically used by the operating system and/or a file system of the host as the maximum capacity that the host's operating system or file system will attempt to allocate.
  • the amelioration trigger is detected (1602) by an amelioration module (e.g., amelioration module 230 of Figure 2A-1, 2A-2 or 2D, 260 of Figure 2B-1 or 2B-2, or 290 of Figure 2C-1 or 2C-2), or a component thereof (e.g., detection module 231 of Figure 2A-1, 2A-2 or 2D, 261 of Figure 2B-1 or 2B-2, or 291 of Figure 2C-1 or 2C-2).
  • detecting the amelioration trigger includes receiving or generating the amelioration trigger.
  • an amelioration module e.g., amelioration module
  • the amelioration module includes a utilization module (e.g., specifically, deleting module 236 of utilization module 232, Figure 2D) for deleting from the storage device discardable data that is used by a host, and a capacity module (e.g., capacity module 234) for reducing declared capacity of the nonvolatile memory of the storage device.
  • a utilization module e.g., specifically, deleting module 236 of utilization module 232, Figure 2D
  • a capacity module e.g., capacity module 234 for reducing declared capacity of the nonvolatile memory of the storage device.
  • the host e.g., computer system 110 of Figure 1A, computer system 142 of Figure IB, or computer system 172 of Figure 1C
  • the host includes (1606) a client on behalf of which data is stored in the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; data storage system 170, Figure 1C).
  • the client is or includes an entity on behalf of which data is stored in the storage system.
  • the host includes (1608) a storage system controller
  • the storage system controller controls and/or coordinates operations among one or more storage devices.
  • the data storage system e.g., data storage system 140, Figure IB
  • the data storage system is called a scale-up system.
  • the host includes (1610) a cluster controller (e.g., cluster controller 180, Figure 1C) of the storage system (e.g., data storage system 170, Figure 1C).
  • the cluster controller controls and/or coordinates operations among one or more data storage subsystems, as shown for example in Figure 1C, where each of the data storage subsystems may be implemented as a data storage system having one or more storage devices (e.g., data storage system 140, Figure IB).
  • the data storage system e.g., data storage system 170, Figure 1C
  • the data storage system is called a scale-out system or a clustered storage system.
  • the detecting, the performing, or both the detecting and the performing are performed (1612) by the storage device (e.g., storage device 120, Figure 1A, storage device 160, Figure IB, or storage device 194, Figure 1C) or one or more components of the storage device (e.g., storage controller 124, Figure 1A).
  • method 1600, or at least the detecting operation 1602 and/or performing operation 1604 of method 1600 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 122-1 of management module 121- 1 , shown in Figures 1 A and 2 A- 1.
  • CPUs processing units
  • the detecting, the performing, or both the detecting and the performing are performed (1614) by one or more subsystems of the storage system distinct from the storage device.
  • the detecting, the performing, or both the detecting and the performing are performed by a storage system controller (e.g., storage system controller 150, Figure IB) of the storage system (e.g., data storage system 140, Figure IB).
  • method 1600 or at least the detecting operation 1602 and/or performing operation 1604 of method 1600, is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 152-1 of management module 151-1, shown in Figures IB and 2B-l .
  • processors such as the one or more processing units (CPUs) 152-1 of management module 151-1, shown in Figures IB and 2B-l .
  • the detecting, the performing, or both the detecting and the performing are performed (1616), at least in part, by the host.
  • method 1600, or at least the detecting operation 1602 and/or performing operation 1604 of method 1600 is governed at least in part by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a host (processors not shown in Figures 1A, IB and 1C), such as the one or more processing units (CPUs) of management module 121-2 ( Figures 1A and 2A-2); or management module 151-2 ( Figures IB and 2B-2); or management module 181-2 ( Figures 1C and 2C-2).
  • deleting (1604) discardable data that is used by the host includes deleting (1618) discardable data that is used by the host in accordance with one or more parameters for the amelioration process.
  • storage controller 124 ( Figure 1 A) or a component thereof e.g., deleting module 236 of utilization module 232, Figure 2D
  • the one or more parameters for the amelioration process include a level of urgency for the amelioration process, a target reduced declared capacity of the non-volatile memory of the storage device, and/or a target amount of reduction in utilization of the nonvolatile memory of the storage device, or any combination or subset thereof.
  • deleting discardable data e.g., temporary data
  • the one or more parameters of the amelioration process indicate or correspond to a target amount of reduction in storage capacity available to the host.
  • the one or more parameters for the amelioration process take into account, or enable the amelioration process to take into account, a projected reduction in the declared capacity of the non- volatile memory of the storage device that will be needed in a predefined time period (e.g., an hour, a day, a week, or any other predefined time period).
  • a predefined time period e.g., an hour, a day, a week, or any other predefined time period.
  • deleting (1604) discardable data that is used by the host includes deleting (1620) temporary data.
  • temporary data is stored in a storage device (e.g., storage medium 130 of storage device 120, Figure 1A), and includes, for example, temporary data of an Operating System installed on the host (e.g., computer system 110, Figure 1A), or temporary data generated by one or more applications, system processes, and/or programs used by the host.
  • an Operating System installed on the host (e.g., computer system 110, Figure 1A)
  • temporary data generated by one or more applications, system processes, and/or programs used by the host For example, in the Unix or Linux operating systems, files stored in the directory "/tmp" are known to be discardable if they have not been used within a specified period of time (typically an hour or a day, etc.).
  • deleting (1604) discardable data that is used by the host includes deleting (1622) data that can be rebuilt when needed.
  • Data that can be rebuilt includes, for example, certain indexes, caches, and other file types that are generated with reference to data stored in the memory of a host (e.g., computer system 110, Figure 1A) and/or storage device (e.g., storage device 120 of Figure 1A, or any of storage devices 160-1 to 160-m of Figure IB, or any of storage devices 194-1 to 194-n or 194-j to 194-k of Figure 1C) of a storage system and can be regenerated if needed.
  • a host e.g., computer system 110, Figure 1A
  • storage device e.g., storage device 120 of Figure 1A, or any of storage devices 160-1 to 160-m of Figure IB, or any of storage devices 194-1 to 194-n or 194-j to 194-k of Figure 1C
  • deleting (1604) discardable data that is used by the host includes deleting (1624) snapshots of data.
  • Snapshots of data include, for example, back-up files/images (e.g., for recovering data), virtual machine snapshots (e.g., for capturing the state, data, and hardware configuration of a virtual machine), and other types of files used for capturing the state of data stored in the memory of a host (e.g., computer system 110, Figure 1A) and/or storage device (e.g., storage device 120 of Figure 1A, or any of storage devices 160-1 to 160-m of Figure IB, or any of storage devices 194-1 to 194-n or 194-j to 194-k of Figure 1C) of a storage system.
  • a host e.g., computer system 110, Figure 1A
  • storage device e.g., storage device 120 of Figure 1A, or any of storage devices 160-1 to 160-m of Figure IB, or any of storage devices 194-1 to 194-n or 194-
  • deleting (1604) discardable data that is used by the host includes deleting (1626) data that is pre -marked by the host as data that is discardable.
  • such data includes preloaded data that the user did not request, one example of which is factory preloaded data stored in storage device 120 by the manufacturer (e.g., programs, applications, back-up files for restoring the OS, etc.).
  • preloaded data that the user did not request is data preloaded by a website or online application in anticipation of future requests by a user.
  • deleting (1604) discardable data that is used by the host includes invalidating (1628) one or more logical address entries of a mapping table (e.g., by using a trim operation), where the logical address entries are associated with the discardable data, and the mapping table is used to translate logical addresses in a logical address space to physical addresses in a physical address space of the storage device.
  • deleting discardable data includes invalidating logical address entry LBA 0 of mapping table 402, where LBA 0 is mapped to bank 1 (e.g., Bank 420-1), block 3 (e.g., Block 421-3), page 37 (pages not shown in Figure 4) of physical address space 410.
  • reducing (1604) declared capacity of the non-volatile memory of the storage device includes making (1630) a number of logical addresses, less than or equal to a number of logical addresses corresponding to the invalidated logical address entries, unavailable to the host.
  • storage controller 124 Figure 1 A or a component thereof (e.g., LBA reduction module 238 of capacity module 234, Figure 2D) reduces and/or coordinates reduction of logical addresses of the logical block address space (e.g., LBA space 320, Figure 3).
  • each of the logical address entries corresponds to multiple (e.g., eight) logical addresses (e.g., 8 LBAs per logical address entry).
  • performing the amelioration process (1604) to reduce declared capacity of the non-volatile memory of the storage device further includes advertising (1632) a reduced declared capacity of the non-volatile memory of the storage device.
  • advertising module 239 of capacity module 234, Figure 2D advertises/or coordinates the advertisement of a reduced declared capacity of non- volatile memory of a storage device
  • the storage device or a corresponding storage controller, cluster controller, management module or data storage system sends a message to the host advertising the reduced declared capacity of the non-volatile memory of the storage device.
  • advertising the reduced declared capacity is accomplished by sending an interrupt or other in-band or out-of-band message to the host.
  • advertising the reduced declared capacity is accomplished by receiving a query from a host to which the storage device is operatively coupled, and in response to receiving the query, reporting the reduced declared capacity of the non-volatile memory of the storage device.
  • the host is configured to periodically query the storage system, storage controller, management module, cluster controller or storage device, for example for a system or device health status.
  • advertising the reduced declared capacity is accomplished by receiving a command (e.g., a storage read or write command) from a host to which the storage device is operatively coupled, and in response to receiving the command, sending a response to the command that includes a notification of the reduced declared capacity of the non-volatile memory of the storage device.
  • a command e.g., a storage read or write command
  • advertising the reduced declared capacity is accomplished by receiving a command (e.g., a storage read or write command) from a host to which the storage device is operatively coupled, and in response to receiving the command, sending both a response to the command and a notification that prompts the host to obtain information, including the reduced declared capacity of the non-volatile memory of the storage device, from the storage device or from the data storage system that includes the storage device.
  • a command e.g., a storage read or write command
  • the mechanism used for returning a notification when responding to a command from the host is a SCSI deferred error or deferred error response code.
  • the method after beginning performance of the amelioration process (1604) to reduce declared capacity of the non- volatile memory of the storage device, the method includes detecting an indication (1634) to abort the reduction in declared capacity of the non-volatile memory of the storage device; and in response to detecting the indication to abort the reduction in declared capacity of the non-volatile memory of the storage device, aborting (1636) performance of the amelioration process to reduce declared capacity of the non- volatile memory of the storage device.
  • Detecting the indication to abort is herein defined to mean either receiving a signal to abort the reduction in declared capacity (e.g., receiving the signal from a controller of the storage device or a storage system controller of a storage system that includes the storage device) or evaluating one or more metrics of the storage device and based on the evaluation, determining to abort the reduction in declared capacity.
  • Normal storage operations will continue to be performed (e.g., read, write, trim, etc.).
  • Normal storage operations include operations like trim that explicitly reduce the storage device utilization, possibly enough to merit aborting the amelioration process.
  • Other storage activity such as garbage collection may also reduce utilization, possibly enough to merit aborting the amelioration process.
  • the amelioration process (e.g., periodically, semi- continuously, initially, finally, occasionally or irregularly) recomputes or re-evaluates a number of parameters, such as the target reduced declared capacity and/or the target amount of utilization reduction), as those parameters may change in value due to the amelioration process and/or normal storage operations (e.g., read, write, erase and trim or unmap operations).
  • a number of parameters such as the target reduced declared capacity and/or the target amount of utilization reduction
  • normal storage operations e.g., read, write, erase and trim or unmap operations.
  • one or more portions of the amelioration process such as the utilization reduction process, is re -prioritized, re-scheduled, or aborted.
  • the storage device includes (1638) one or more flash memory devices.
  • the storage device comprises a storage medium (e.g., storage medium 130, Figure 1A), and the storage medium comprises one or more nonvolatile storage devices, such as flash memory devices.
  • the storage medium e.g., storage medium 130, Figure 1A
  • the storage medium is a single flash memory device, while in other embodiments the storage medium includes a plurality of flash memory devices.
  • the storage medium includes dozens or hundreds of flash memory devices, organized in parallel memory channels, such as 16, 32 or 64 flash memory devices per memory channel, and 8, 16 or 32 parallel memory channels.
  • the non- volatile storage medium (e.g., storage medium 130, Figure 1A) includes NAND-type flash memory or NOR-type flash memory. In other embodiments, the storage medium comprises one or more other types of non-volatile storage devices.
  • Figures 17A-17C illustrate a flowchart representation of a method 1700 of managing a storage system, in accordance with some embodiments.
  • the method 1700 includes detecting (1702) an amelioration trigger for reducing declared capacity of non-volatile memory of a first storage device (e.g., storage device 120 of Figure 1A, or any of storage devices 160-1 to 160-m of Figure IB, or any of storage devices 194-1 to 194-n or 194-j to 194-k of Figure 1C) of the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; or data storage system 170, Figure 1C), and in accordance with the detected amelioration trigger, performing (1704) an amelioration process to reduce declared capacity of the non-volatile memory of the first storage device, including: moving a portion of data (e.g., live data 334, Figure 3) that is used by a host (e.g., computer system 110 of Figure 1A, computer system
  • a host e
  • detecting an amelioration trigger for reducing declared capacity of non-volatile memory of a first storage device of the storage system includes generating the amelioration trigger for reducing declared capacity of non- volatile memory of the first storage device of the storage system.
  • reducing declared capacity of non-volatile memory of the first storage device includes reducing storage capacity, of the non-volatile memory of the first storage device, available to the host.
  • the declared capacity of non- volatile memory of the first storage device is sometimes called the advertised capacity, and is typically used by the operating system and/or a file system of the host as the maximum capacity that the host's operating system or file system will attempt to allocate.
  • the amelioration trigger is detected (1702) by an amelioration module (e.g., amelioration module 230 of Figure 2A-1, 2A-2 or 2D, 260 of Figure 2B-1 or 2B-2, or 290 of Figure 2C-1 or 2C-2), or a component thereof (e.g., detection module 231 of Figure 2A-1, 2A-2 or 2D, 261 of Figure 2B-1 or 2B-2, or 291 of Figure 2C-1 or 2C-2).
  • detecting the amelioration trigger includes receiving or generating the amelioration trigger.
  • an amelioration module e.g., amelioration module
  • the amelioration module includes a utilization module (e.g., specifically, moving module 237 of utilization module 232, Figure 2D) for moving a portion of data that is used by a host from the first storage device to another storage device of the storage system, and a capacity module (e.g., capacity module 234) for reducing declared capacity of the non-volatile memory of the first storage device.
  • a utilization module e.g., specifically, moving module 237 of utilization module 232, Figure 2D
  • a capacity module e.g., capacity module 234 for reducing declared capacity of the non-volatile memory of the first storage device.
  • the host e.g., computer system 110 of Figure 1A, computer system 142 of Figure IB, or computer system 172 of Figure 1C
  • the host includes (1706) a client on behalf of which data is stored in the storage system (e.g., data storage system 100, Figure 1A; data storage system 140, Figure IB; data storage system 170, Figure 1C).
  • the client is or includes an entity on behalf of which data is stored in the storage system.
  • the host includes (1708) a storage system controller
  • the storage system controller controls and/or coordinates operations among one or more storage devices.
  • the data storage system e.g., data storage system 140, Figure IB
  • the data storage system is called a scale-up system.
  • the host includes (1710) a cluster controller (e.g., cluster controller 180, Figure 1C) of the storage system (e.g., data storage system 170, Figure 1C).
  • the cluster controller controls and/or coordinates operations among one or more data storage subsystems, as shown for example in Figure 1C, where each of the data storage subsystems may be implemented as a data storage system having one or more storage devices (e.g., data storage system 140, Figure IB).
  • the data storage system e.g., data storage system 170, Figure 1C
  • the data storage system is called a scale-out system or a clustered storage system.
  • the detecting, the performing, or both the detecting and the performing are performed (1712), at least in part, by the host.
  • method 1700, or at least the detecting operation 1702 and/or performing operation 1704 of method 1700 is governed at least in part by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a host (processors not shown in Figures 1A, IB and 1C), such as the one or more processing units (CPUs) of management module 121-2 ( Figures 1A and 2A-2); or management module 151-2 ( Figures IB and 2B-2); or management module 181-2 ( Figures 1C and 2C-2).
  • the detecting, the performing, or both the detecting and the performing are performed (1714) by one or more subsystems of the storage system distinct from the first storage device.
  • the detecting, the performing, or both the detecting and the performing are performed by a storage system controller (e.g., storage system controller 150, Figure IB) of the storage system (e.g., data storage system 140, Figure IB).
  • method 1700 is governed by instructions that are stored in a non-transitory computer readable storage medium and that are executed by one or more processors of a device, such as the one or more processing units (CPUs) 152-1 of management module 151-1, shown in Figures IB and 2B-1.
  • processors such as the one or more processing units (CPUs) 152-1 of management module 151-1, shown in Figures IB and 2B-1.
  • moving (1704) the portion of data that is used by the host includes moving (1716) the portion of data that is used by the host in accordance with one or more parameters for the amelioration process.
  • storage controller 124 Figure 1 A or a component thereof (e.g., moving module 237 of utilization module 232, Figure 2D) moves and/or coordinates the moving of data that is used by the host from a first storage device (e.g., storage medium 130 of storage device 120, Figure 1A), or a physical location within the first storage device where the data is stored, to another storage device.
  • the one or more parameters for the amelioration process include a level of urgency for the amelioration process, a target reduced declared capacity of the non-volatile memory of the first storage device, and/or a target amount of reduction in utilization of the non-volatile memory of the first storage device, or any combination or subset thereof.
  • moving the portion of data that is used by the host in accordance with one or more parameters for the amelioration process includes moving enough data to meet a target amount of declared capacity reduction specified by the one or more parameters. Stated another way, the one or more parameters of the amelioration process indicate or correspond to a target amount of reduction in storage capacity available to the host.
  • the one or more parameters for the amelioration process take into account, or enable the amelioration process to take into account, a projected reduction in the declared capacity of the non-volatile memory of the first storage device that will be needed in a predefined time period (e.g., an hour, a day, a week, or any other predefined time period).
  • a predefined time period e.g., an hour, a day, a week, or any other predefined time period.
  • moving (1704) the portion of data that is used by the host includes (1722): moving data associated with one or more virtual logical addresses, including updating a virtual address mapping module.
  • system mapping module 250-1 is an example of a virtual address mapping module that maps virtual logical addresses to intermediate addresses, each of which identifies a storage device and a location within the storage device at which data, corresponding to a virtual logical address, is stored.
  • cluster mapping module 280-1 is an example of a virtual address mapping module that maps virtual logical addresses to intermediate addresses, each of which identifies a storage device and a location within the storage device at which data, corresponding to a virtual logical address, is stored.
  • moving the portion of the data includes selecting
  • the method includes selecting which logical address entries to move so as to minimize performance degradation. For example, to achieve a particular amount of declared capacity reduction, in some circumstances there may be multiple candidates, or sets of candidates, of logical addresses for which a mapping in a virtual address mapping module (e.g., system mapping module 250, Figure 2B-1 or Figure 2B-2, or cluster mapping module 280, Figure 2C-1 or Figure 2C-2) could be altered (e.g., by moving associated data, and subsequently updating the virtual address mapping module) to help accomplish the reduction in declared storage capacity of the non-volatile memory of the first storage device.
  • a virtual address mapping module e.g., system mapping module 250, Figure 2B-1 or Figure 2B-2, or cluster mapping module 280, Figure 2C-1 or Figure 2C-2
  • some of the sets of candidates may be accessed or overwritten less frequently than other sets of candidates.
  • the process minimizes performance degradation.
  • the logical addresses are selected in accordance with a wear-leveling methodology, so as to promote uniform wearing, or to avoid uneven wearing of the non- volatile memory media.
  • the one or more logical addresses are selected (1720) so as to minimize overhead from garbage collection. While updating the virtual address mapping module may, at least in some circumstances, be accomplished without physically moving data from one location to another within the physical address space of the non-volatile memory of the first storage device, in practice the underlying cause of the detected amelioration trigger may dictate the physical movement of some data from the first storage device to another storage device.
  • the device or system that does the selecting (1720) uses a process or analytical method to minimize the amount of data movement that will result from altering the virtual address mapping module.
  • reducing (1704) declared capacity of the non- volatile memory of the first storage device includes trimming (1724) logical addresses associated with the data moved from the first storage device. As a result, the "old copy" of the moved data is no longer accessible.
  • storage controller 124 ( Figure 1A) or a component thereof (e.g., LBA reduction module 238 of capacity module 234, Figure 2D) reduces and/or coordinates reduction of logical addresses of the logical block address space (e.g., LBA space 320, Figure 3).
  • the logical addresses are those logical addresses associated with the first storage device prior to performing the amelioration process.
  • each of the logical address entries corresponds to multiple (e.g., eight) logical addresses (e.g., 8 LBAs per logical address entry).
  • performing (1704) the amelioration process to reduce declared capacity of the non-volatile memory of the first storage device further includes invalidating (1726) one or more logical address entries, associated with the portion of data, of a mapping table, the mapping table used to translate logical addresses in the logical address space to physical addresses in a physical address space of the first storage device.
  • performing the amelioration process includes invalidating logical address entry LBA 0 of mapping table 402, where LBA 0 is mapped to bank 1 (e.g., Bank 420-1), block 3 (e.g., Block 421-3), page 37 (pages not shown in Figure 4) of physical address space 410.
  • Corresponding memory blocks, to which the invalidated data and a number of logical addresses correspond, are therefore prepared for being made unavailable to the host, for deletion, and/or for garbage collection (e.g., valid pages of a block are re-written to a new block, and the old block containing invalid data is erased such that the old block is available for new data to be written).
  • performing the amelioration process includes setting one or more indicators for one or more physical addresses corresponding to the one or more invalidated logical address entries to indicate that data stored at the one or more physical addresses can be discarded.
  • a record or other data corresponding to one or more of the physical addresses corresponding to the one or more invalidated logical address entries is moved to a list of memory portions that are eligible or ready for deletion.
  • performing the amelioration process (1704) to reduce declared capacity of the non-volatile memory of the first storage device further includes advertising (1728) a reduced declared capacity of the non- volatile memory of the first storage device.
  • storage controller 124 Figure 1 A or a component thereof (e.g., advertising module 239 of capacity module 234, Figure 2D) advertises/or coordinates the advertisement of a reduced declared capacity of non- volatile memory of a first storage device (e.g., storage medium 130 of storage device 120, Figure 1A) or storage subsystem.
  • the first storage device or a corresponding storage controller, cluster controller, management module or data storage system sends a message to the host advertising the reduced declared capacity of the non-volatile memory of the first storage device.
  • advertising the reduced declared capacity is accomplished by sending an interrupt or other in-band or out-of-band message to the host.
  • advertising the reduced declared capacity is accomplished by receiving a query from a host to which the first storage device is operatively coupled, and in response to receiving the query, reporting the reduced declared capacity of the non-volatile memory of the first storage device.
  • the host is configured to periodically query the storage system, storage controller, management module, cluster controller or storage device, for example for a system or device health status.
  • advertising the reduced declared capacity is accomplished by receiving a command (e.g., a storage read or write command) from a host to which the first storage device is operatively coupled, and in response to receiving the command, sending a response to the command that includes a notification of the reduced declared capacity of the non- volatile memory of the first storage device.
  • a command e.g., a storage read or write command
  • advertising the reduced declared capacity is accomplished by receiving a command (e.g., a storage read or write command) from a host to which the first storage device is operatively coupled, and in response to receiving the command, sending a response to the command that includes a notification of the reduced declared capacity of the non- volatile memory of the first storage device.
  • a command e.g., a storage read or write command
  • advertising the reduced declared capacity is accomplished by receiving a command (e.g., a storage read or write command) from a host to which the first storage device is operatively coupled, and in response to receiving the command, sending both a response to the command and a notification that prompts the host to obtain information, including the reduced declared capacity of the non-volatile memory of the first storage device, from the first storage device or from the data storage system that includes the first storage device.
  • a command e.g., a storage read or write command
  • the mechanism used for returning a notification when responding to a command from the host is a SCSI deferred error or deferred error response code.
  • the method after beginning performance of the amelioration process (1704) to reduce declared capacity of the non- volatile memory of the first storage device, the method includes detecting an indication (1730) to abort the reduction in declared capacity of the non- volatile memory of the first storage device; and in response to detecting the indication to abort the reduction in declared capacity of the non- volatile memory of the first storage device, aborting (1732) performance of the amelioration process to reduce declared capacity of the non-volatile memory of the first storage device.
  • Detecting the indication to abort is herein defined to mean either receiving a signal to abort the reduction in declared capacity (e.g., receiving the signal from a controller of the first storage device or a storage system controller of a storage system that includes the first storage device) or evaluating one or more metrics of the first storage device and based on the evaluation, determining to abort the reduction in declared capacity.
  • normal storage operations will continue to be performed (e.g., read, write, delete, trim, etc.). Normal storage operations include operations like trim that explicitly reduce the first storage device utilization, possibly enough to merit aborting the amelioration process. Other storage activity such as garbage collection may also reduce utilization, possibly enough to merit aborting the amelioration process.
  • the amelioration process (e.g., periodically, semi- continuously, initially, finally, occasionally or irregularly) recomputes or re-evaluates a number of parameters, such as the target reduced declared capacity and/or the target amount of utilization reduction), as those parameters may change in value due to the amelioration process and/or normal storage operations (e.g., read, write, erase and trim or unmap operations).
  • a number of parameters such as the target reduced declared capacity and/or the target amount of utilization reduction
  • normal storage operations e.g., read, write, erase and trim or unmap operations.
  • one or more portions of the amelioration process such as the utilization reduction process, is re -prioritized, re-scheduled, or aborted.
  • the first storage device includes (1734) one or more flash memory devices.
  • the first storage device comprises a storage medium (e.g., storage medium 130, Figure 1 A), and the storage medium comprises one or more non- volatile storage devices, such as flash memory devices.
  • the storage medium e.g., storage medium 130, Figure 1A
  • the storage medium is a single flash memory device, while in other embodiments the storage medium includes a plurality of flash memory devices.
  • the storage medium includes dozens or hundreds of flash memory devices, organized in parallel memory channels, such as 16, 32 or 64 flash memory devices per memory channel, and 8, 16 or 32 parallel memory channels.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • Techniques For Improving Reliability Of Storages (AREA)

Abstract

L'invention concerne des systèmes, des procédés et/ou des dispositifs qui sont utilisés pour réduire la capacité déclarée d'une mémoire non volatile d'un dispositif de stockage dans un système de stockage. Selon un aspect, le procédé consiste à détecter un élément déclencheur d'amélioration pour réduire la capacité déclarée d'une mémoire non volatile d'un dispositif de stockage du système de stockage, et conformément à l'élément déclencheur d'amélioration détecté, à réaliser un processus d'amélioration pour réduire la capacité déclarée de la mémoire non volatile du dispositif de stockage, comprenant la réduction d'une plage d'adresses logiques d'un espace d'adresses logiques accessible à un hôte. Dans certains modes de réalisation, le dispositif de stockage comprend un ou plusieurs dispositifs de mémoire flash. Dans certains modes de réalisation, la détection, la réalisation, ou les deux sont réalisées par le dispositif de stockage, ou par un ou plusieurs sous-systèmes du système de stockage distincts du dispositif de stockage, ou par l'hôte.
PCT/US2014/053941 2013-09-03 2014-09-03 Procédé et appareil pour réduire la capacité déclarée d'un dispositif de stockage par réduction d'une plage d'adresses logiques WO2015034954A1 (fr)

Applications Claiming Priority (36)

Application Number Priority Date Filing Date Title
US201361873326P 2013-09-03 2013-09-03
US201361873324P 2013-09-03 2013-09-03
US61/873,326 2013-09-03
US61/873,324 2013-09-03
US14/470,580 US9519577B2 (en) 2013-09-03 2014-08-27 Method and system for migrating data between flash memory devices
US14/470,580 2014-08-27
US14/470,596 US9442670B2 (en) 2013-09-03 2014-08-27 Method and system for rebalancing data stored in flash memory devices
US14/470,596 2014-08-27
US201462044898P 2014-09-02 2014-09-02
US201462044883P 2014-09-02 2014-09-02
US201462044976P 2014-09-02 2014-09-02
US201462044983P 2014-09-02 2014-09-02
US201462044905P 2014-09-02 2014-09-02
US201462044919P 2014-09-02 2014-09-02
US201462044936P 2014-09-02 2014-09-02
US201462044981P 2014-09-02 2014-09-02
US201462044930P 2014-09-02 2014-09-02
US201462044969P 2014-09-02 2014-09-02
US201462044963P 2014-09-02 2014-09-02
US201462044890P 2014-09-02 2014-09-02
US201462044989P 2014-09-02 2014-09-02
US201462044932P 2014-09-02 2014-09-02
US62/044,963 2014-09-02
US62/044,932 2014-09-02
US62/044,976 2014-09-02
US62/044,981 2014-09-02
US62/044,905 2014-09-02
US62/044,969 2014-09-02
US62/044,930 2014-09-02
US62/044,983 2014-09-02
US62/044,936 2014-09-02
US62/044,989 2014-09-02
US62/044,919 2014-09-02
US62/044,898 2014-09-02
US62/044,883 2014-09-02
US62/044,890 2014-09-02

Publications (1)

Publication Number Publication Date
WO2015034954A1 true WO2015034954A1 (fr) 2015-03-12

Family

ID=52628902

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2014/053941 WO2015034954A1 (fr) 2013-09-03 2014-09-03 Procédé et appareil pour réduire la capacité déclarée d'un dispositif de stockage par réduction d'une plage d'adresses logiques

Country Status (1)

Country Link
WO (1) WO2015034954A1 (fr)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106257401A (zh) * 2015-06-22 2016-12-28 三星电子株式会社 数据存储装置
CN107526691A (zh) * 2016-06-21 2017-12-29 深圳市中兴微电子技术有限公司 一种缓存管理方法及装置
CN108139871A (zh) * 2015-10-05 2018-06-08 美光科技公司 具有基于存储器寿命周期的可变逻辑容量的固态存储装置
TWI660361B (zh) * 2017-03-29 2019-05-21 美商美光科技公司 用於多維記憶體之選擇性錯誤率資訊
JP2019082813A (ja) * 2017-10-30 2019-05-30 東芝メモリ株式会社 メモリシステムおよび制御方法
TWI684985B (zh) * 2019-02-21 2020-02-11 旺宏電子股份有限公司 記憶體裝置與其編程方法
CN111149091A (zh) * 2017-10-02 2020-05-12 西部数据技术公司 基于跨多个装置的协调内部地址方案的冗余编码条带
WO2020259942A1 (fr) * 2019-06-28 2020-12-30 Siemens Mobility GmbH Procédé de fonctionnement d'un ensemble informatique et ensemble informatique

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100008140A1 (en) * 2008-07-11 2010-01-14 Chang-Hyun Lee Nonvolatile memory devices supporting memory cells having different bit storage levels and methods of operating the same

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100008140A1 (en) * 2008-07-11 2010-01-14 Chang-Hyun Lee Nonvolatile memory devices supporting memory cells having different bit storage levels and methods of operating the same

Cited By (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20160150497A (ko) * 2015-06-22 2016-12-30 삼성전자주식회사 데이터 저장 장치, 이를 포함하는 데이터 처리 시스템, 및 이의 작동 방법
CN106257401A (zh) * 2015-06-22 2016-12-28 三星电子株式会社 数据存储装置
KR102397582B1 (ko) 2015-06-22 2022-05-13 삼성전자주식회사 데이터 저장 장치, 이를 포함하는 데이터 처리 시스템, 및 이의 작동 방법
US11385797B2 (en) 2015-10-05 2022-07-12 Micron Technology, Inc. Solid state storage device with variable logical capacity based on memory lifecycle
CN108139871A (zh) * 2015-10-05 2018-06-08 美光科技公司 具有基于存储器寿命周期的可变逻辑容量的固态存储装置
EP3360035A4 (fr) * 2015-10-05 2019-05-22 Micron Technology, Inc. Dispositif de stockage à semi-conducteur avec capacité logique variable selon un cycle de vie de la mémoire
US11704025B2 (en) 2015-10-05 2023-07-18 Micron Technology, Inc. Solid state storage device with variable logical capacity based on memory lifecycle
CN107526691A (zh) * 2016-06-21 2017-12-29 深圳市中兴微电子技术有限公司 一种缓存管理方法及装置
TWI660361B (zh) * 2017-03-29 2019-05-21 美商美光科技公司 用於多維記憶體之選擇性錯誤率資訊
CN111149091B (zh) * 2017-10-02 2023-09-12 西部数据技术公司 基于跨多个装置的协调内部地址方案的冗余编码条带
CN111149091A (zh) * 2017-10-02 2020-05-12 西部数据技术公司 基于跨多个装置的协调内部地址方案的冗余编码条带
JP2019082813A (ja) * 2017-10-30 2019-05-30 東芝メモリ株式会社 メモリシステムおよび制御方法
JP7074453B2 (ja) 2017-10-30 2022-05-24 キオクシア株式会社 メモリシステムおよび制御方法
TWI684985B (zh) * 2019-02-21 2020-02-11 旺宏電子股份有限公司 記憶體裝置與其編程方法
WO2020259942A1 (fr) * 2019-06-28 2020-12-30 Siemens Mobility GmbH Procédé de fonctionnement d'un ensemble informatique et ensemble informatique

Similar Documents

Publication Publication Date Title
EP3189437B1 (fr) Procédé et appareil pour réduire la capacité déclarée d'un dispositif de mémoire par ajustement de manière conditionnelle
US9582220B2 (en) Notification of trigger condition to reduce declared capacity of a storage device in a multi-storage-device storage system
US9519577B2 (en) Method and system for migrating data between flash memory devices
EP3189439B1 (fr) Procédé et appareil pour réduire la capacité déclarée d'un dispositif de stockage en modifiant un format de codage
WO2015034954A1 (fr) Procédé et appareil pour réduire la capacité déclarée d'un dispositif de stockage par réduction d'une plage d'adresses logiques
US9665311B2 (en) Process and apparatus to reduce declared capacity of a storage device by making specific logical addresses unavailable
US9519427B2 (en) Triggering, at a host system, a process to reduce declared capacity of a storage device
EP3189438B1 (fr) Déclenchement d'un processus pour réduire la capacité déclarée d'un dispositif de mémoire dans un système de mémoire à dispositif de mémoire multiple
US9582203B2 (en) Process and apparatus to reduce declared capacity of a storage device by reducing a range of logical addresses
US9524112B2 (en) Process and apparatus to reduce declared capacity of a storage device by trimming
US9652153B2 (en) Process and apparatus to reduce declared capacity of a storage device by reducing a count of logical addresses
US9563362B2 (en) Host system and process to reduce declared capacity of a storage device by trimming
US9552166B2 (en) Process and apparatus to reduce declared capacity of a storage device by deleting data
US9582202B2 (en) Process and apparatus to reduce declared capacity of a storage device by moving data
US9582212B2 (en) Notification of trigger condition to reduce declared capacity of a storage device
US9563370B2 (en) Triggering a process to reduce declared capacity of a storage device

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14767225

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14767225

Country of ref document: EP

Kind code of ref document: A1