DE102017112446A1 - Electrically buffered NV-DIMM and method of use therewith - Google Patents

Abstract

An electrically buffered NV-DIMM and method of use therewith are provided. In one embodiment, a memory system is provided that includes a plurality of non-volatile memory devices; a controller in communication with the plurality of non-volatile storage devices; a plurality of data buffers in communication with the controller and configured to store data sent between the controller and an input / output bus; and a command and address buffer configured to store commands and addresses sent from a host, the command and address buffer further configured to synchronize data flow into and out of the plurality of data buffers ,

Description

Cross-reference to related application

This application claims the benefit of US Patent Application No. 62 / 380,217, filed on Aug. 26, 2016, which is hereby incorporated herein by reference.

background

Many computer systems use one or more dual in-line memory modules (DIMMs) connected to a central processing unit (CPU) to store data. Some DIMMs include dynamic random access memory (DRAM) chips. However, DRAM is relatively expensive, requires a relatively large amount of power, and fails to scale capacity with rate matching processor performance, which is undesirable when used in servers such as enterprise and hyperscale systems in data centers where large amounts of data are stored may, To address these issues, non-volatile DIMMs (NV-DIMMs) have been developed which replace volatile DRAM chips with non-volatile memory devices. Compared to DRAM-based DIMMs, NV-DIMMs can provide lower cost per gigabyte, lower power consumption, and longer data retention, especially in the event of a power outage or system crash. Like some DRAM-based DIMMs, some NV-DIMMs are designed to communicate via a clock data-parallel interface, such as a double data rate (DDR) interface.

Brief description of the drawings

1 Figure 12 is a block diagram of a host and memory systems of one embodiment.

2A Figure 4 is a block diagram of a memory system of an embodiment wherein the memory system takes the form of a non-volatile dual in-line memory (NV-DIMM) non-volatile memory module.

2 B FIG. 10 is a block diagram of a distributed system memory system of one embodiment. FIG.

3 Fig. 10 is a block diagram showing signals between a host and memory systems of an embodiment.

4 FIG. 10 is a flowchart of a method of reading data from a DRAM DIMM. FIG.

5 Figure 4 is a timing diagram of a method of reading data from a DRAM DIMM.

6 FIG. 10 is a flowchart of a method of an embodiment for a host to send a read command.

7 Figure 10 is a flow diagram of a method of an embodiment for a host to request a return of read data by using a send command and processing received data.

8A and 8B Figure 9 are timing diagrams of a non-deterministic method for reading data from a memory system of one embodiment.

8C Figure 10 is a timing diagram of a non-deterministic method for writing data to a memory system of one embodiment.

9 Figure 12 is a block diagram of a controller of a memory system of one embodiment.

10 FIG. 10 is a flow chart of a method of reading data from a memory system of one embodiment. FIG.

11 FIG. 10 is a flowchart of a method of writing data to a memory system of one embodiment. FIG.

12 and 13 are representations showing read and write flows of a DRAM-based DIMM, respectively.

14 is a representation of the internal states of the data flow in a DRAM-based DIMM.

15 Figure 4 is a block diagram of a memory system of an embodiment wherein the memory system takes the form of a non-volatile dual in-line memory (NV-DIMM) non-volatile memory module.

16 Fig. 10 is a block diagram illustrating a read operation of a memory system of an embodiment.

17 Fig. 10 is a block diagram illustrating a write operation of a memory system of an embodiment.

18A and 18B FIG. 10 are flowcharts of a read operation of an embodiment. FIG.

19A and 19B FIG. 10 are flowcharts of a write operation of an embodiment. FIG.

20 Fig. 10 is a diagram showing a change of the clock speed of an embodiment.

21 is a block diagram of a data buffer.

22 FIG. 12 is a block diagram of a data buffer of one embodiment. FIG.

23A Figure 12 is a block diagram of a memory system of an embodiment in which nonvolatile memory devices are connected to data buffers without going through an NVM controller.

23B FIG. 10 is a block diagram of a Registered Clock Driver (RCD) of one embodiment. FIG.

Detailed description

overview

By way of introduction, the following embodiments relate to an electrically buffered NV-DIMM and a method of use therewith. In one embodiment, a memory system is provided that includes a plurality of non-volatile memory devices; a controller in communication with the plurality of non-volatile storage devices; a plurality of data buffers in communication with the controller and configured to store data sent between the controller and an input / output bus; and a command and address buffer configured to store commands and addresses sent from a host, the command and address buffer further configured to synchronize data flow into and out of the plurality of data buffers ,

In some embodiments, the controller is configured to associate read and / or write commands with identifiers so that the read and / or write commands can be processed in a different order than in an order received from the host.

In some embodiments, the command and address buffer includes a registered clock driver.

In some embodiments, the plurality of data buffers include random access memory.

In some embodiments, the command and address buffer is further configured to reduce a frequency of a clock received from the host.

In some embodiments, the command and address buffer is further configured to perform bandwidth conversion.

In some embodiments, physical layers and instruction layers of the memory system are designed to be compatible with a DRAM DIMM communication protocol.

In some embodiments, physical layers and instruction layers of the memory system are designed to be compatible with one or more of the following: unbuffered DIMM (UDIMM), registered DIMM (RDIMM), and load-reduced DIMM (LRDIMM).

In some embodiments, the controller is further configured to: transmit a ready signal after data requested by the host is ready to be sent to the host; Receiving a send command from the host; and in response to receiving the send command from the host, sending the data to the host.

In some embodiments, the data is sent to the host after a time delay, wherein the time delay is selected based on a communication protocol used with the host.

In some embodiments, the controller is configured to communicate with the host using a clock data parallel interface.

 In some embodiments, the clock data parallel interface comprises a double data rate (DDR) interface.

In some embodiments, at least one of the plurality of non-volatile memory devices includes a three-dimensional memory.

Other embodiments are possible, and each of the embodiments may be used alone or in combination.

General introduction to an implementation of an embodiment

As explained in the background section above, dual in-line memory modules (DIMMs) can be connected to a central processing unit (CPU) of a host to store data. Non-Volatile Dual In-line Memory Modules (NV-DIMMs) have been developed to provide chips with volatile DRAM on standard DIMMs with non-volatile memory devices such as non-volatile memory devices NAND, to replace. Compared to DRAM-based DIMMs, NV-DIMMs can provide lower cost per gigabyte, lower power consumption, and longer data retention, especially in the event of a power outage or system crash. Like some DRAM-based DIMMs, some NV-DIMMs are designed to communicate via a clock data-parallel interface, such as a double data rate (DDR) interface.

However, existing standards that are suitable for DRAM-based DIMMs may not be suitable for NV-DIMMs. For example, some existing standards require that read and write operations be completed within a specified ("deterministic") period of time. Although the completion of read and write operations in the specified amount of time is typically not a problem for DRAM memories, the mechanism of reading and writing to the non-volatile memory may result in delays exceeding the specified time period. That is, DRAM-based DIMM protocols expect consistent, predictable, and fast responses that nonvolatile storage may not be able to provide. To take this into account, some emerging standards (eg JEDEC's NVDIMM-P standard) allow "non-deterministic" read and write operations to introduce "slack" into the communication between the storage system and the host. According to such standards, read and write operations to the NV-DIMM need not be completed in a certain amount of time. In the case of a read operation, the NV-DIMM instead informs the host when the requested data is ready so that the host can retrieve it. In the case of a write operation, the host may be constrained to have more than a certain number of pending write instructions to ensure that the non-volatile memory device does not receive more write commands than it can handle.

The approach of allowing non-deterministic timed operations at a protocol level is just one possible approach to dealing with the unpredictable nature of non-volatile storage. Other approaches do not use non-deterministic modifications of the DDR standard. Instead, they rely on software approaches to construct composite read and write procedures from conventional DDR primitives. Each DDR primitive may correspond either to direct access to the non-volatile memory itself, or it may correspond to indirect operations performed by the use of interconnect elements such as control registers or buffers. Although the read or write algorithms themselves may require an unspecified number of iterations or DRR commands to complete - and thus may not complete within a particular timeframe - each individual DDR primitive completes operation within the well-defined time limits set by the usual (e.g. deterministic timed) DDR standards.

Some of the following embodiments benefit from the non-deterministic aspect of the emerging standard to allow the NV-DIMM to perform time-consuming actions that may not have the time, according to conventional DRAM-based DIMM standards. These actions are sometimes referred to herein as indefinite-duration operations from the perspective of the host, and may include memory and data management operations. These memory and data management operations may be important to the operation of the NV-DIMM. For example, a non-volatile memory device may have lower endurance (i.e., the number of writes prior to a failure) compared to DRAM and may store data less reliably (eg, due to internal memory errors that cause bits to be improperly stored). These problems can be even more pronounced with emerging non-volatile memory technologies that would likely be used as a DRAM replacement in an NV-DIMM. As such, in one embodiment, the NV-DIMM exploits that it is not "under pressure" to perform indefinite-duration operations from the perspective of the host, such as memory and data management operations (e.g., wear-leveling and data management operations) Error correction operations) which it may not be able to perform in the allocated time according to conventional DRAM based DIMM standards.

It should be noted that this introduction discusses only a particular implementation of an embodiment, and that other implementations and embodiments may be used, as discussed in the following paragraphs. Further, although some of these embodiments are discussed with respect to an NV-DIMM connected to a CPU of a host, any type of memory system may be used in any suitable type of environment. Accordingly, specific architectures and protocols discussed herein should not be read into the claims unless expressly stated therein.

General discussion of timing data interfaces and new protocols

Clock data parallel interfaces are an easy way to transfer digitized data and commands between any two devices. Each transmission line carrying data or instructions from one device to another is accompanied by a separate "clock" transmission line which provides a time reference for sample changes in the data and command buses. In some embodiments, the clock may be disabled when the interface is inactive and does not transmit data or commands. This provides a convenient way to reduce power loss during inactivity. In some embodiments of clock data parallel interfaces, the clock is a single-ended transmission line, which means that the clock consists of an additional transmission line whose voltage is compared to a common voltage reference shared by many transmission lines between the CPU and the memory devices. In other embodiments, the time reference may be a differential clock having both a positive clock reference and a clock complement switching to a low voltage concurrently with each of the positive to low-to-high voltage switching - an event known as the "rising edge" of the clock and vice versa, the clock complement switches to the high voltage state with each high to low voltage transition of the positive clock reference - an event known as the "falling edge" of the clock. Timing data parallel interfaces are often classified according to how many data beats are sent along with the clock. In single-data-rate or SDR interfaces, the command or data buses alternate once per clock cycle, often with the rising edge of the reference clock. In Double Data Rate (DDR) interfaces, the command and data buses send twice as much data per clock cycle, allowing the command and data busses to switch twice per period, once on the rising edge of the clock and once on the clock falling edge of the clock. There are also Quad Data Rate (QDR) protocols, which allow four data or command transitions per clock. Typically, clock data parallel interfaces are efficient and of low latency because of their simplicity, and the receiver circuits can be as simple as a single bank of logic flip-flops. However, there may be additional complexity caused by the need to synchronize the newly-latched data to the internal clock of the devices themselves, one of the many jobs handled by a collection of signal conditioning circuits called the "physical communication layer" or simply "Phy layer" are known.

Conversely, serial interfaces are typically based on clock data recovery processes to extract the time reference from a single electrical transmission line which switches the voltage at regular time intervals but in a pattern that also communicates commands and / or data (in some embodiments, many different lines will be used in parallel for increased bandwidth, and thus each line may encode data for a whole instruction and a whole sequence of data, or only part of a instruction or data sequence). Coding the clock and data in the same physical transmission line reduces timing uncertainties caused by mismatching delays between clock and data or command lines, thus allowing clock frequencies of 25 GHz or higher for very high bandwidth communication. However, such interfaces also have some disadvantages. Due to the nature of clock data recovery, the transmission line must remain continuously active to maintain the synchronization of the inferred clock reference between the communication partners. Power saving modes are possible, but switching back to active mode requires significant retraining delays. Moreover, the very nature of clock data recovery requires slightly more time to decode each message, and one-way communication delays are common even with a well-trained serial link. This adds extra latency to each data request.

The interface between computer CPUs and their corresponding storage devices is an example of an interface where optimization of both performance and latency is desired. Thus, while high bandwidth serial CPU memory interfaces exist, such as Hybrid Memory Cube, most of today's interfaces between CPUs and memory devices still use clock data parallel interfaces. For example, synchronous dynamic random access memory (SDRAM) uses a single clock to synchronize commands on a command bus, which consists of a plurality of transmission lines, each of which encodes a bit of the command sequence information. Depending on the embodiment, instructions in an SDRAM command sequence may include, but are not limited to: activating a series of cells in a two-dimensional data array for future reading or writing; Reading from some Columns in a currently active row; Writing some columns in a currently active line; Selecting another bank of cells for reading or writing; Writing a few bits into the memory mode registers to change aspects of the behavior of the memory device; and reading back values from the mode registers to identify the status of the memory device.

Data associated with these instructions are sent or received along a separate data bus consisting of a separate and parallel plurality of data transmission lines called the DQ bus. In some embodiments, the DQ bus may be half-duplex and bi-directional, meaning that the same lines are used for receiving and transmitting data, and data may not be sent simultaneously from the memory device to the CPU while data is in the opposite direction flow, or vice versa. In other embodiments, the DQ bus may be full-duplex with separate lines for receiving or transmitting data. It can be safely assumed that the data on the DQ bus is synchronous with the device command clock. However, for longer transmission lines or faster operating frequencies, this can lead to poor synchronization. Thus, there are other embodiments in which the entire DQ bus is divided into a plurality of smaller DQ groups, each with its own "DQ Strobe" signal, DQS, which is used as a separate timing frequency for the wires in that DQ. Group serves. For example, in one embodiment, a 64-bit DQ bus may be divided into 8 groups (or "byte lanes") of 8 DQ lines, each synchronized by its own DQS strobe. Depending on the embodiment, the DQS strobes can be differential or single-ended. In some embodiments, some DQ lines may provide encoding not only for data stored by the host, but also additional parity or other signal data for the purpose of recording additional error correction codes. Depending on the embodiment, many DDR protocols include a range of other control signal transmission lines that are driven by the CPU to the memory device, which may, for example, command, but is not limited to, the following functions in some embodiments: Command Suppression Lines (CS_N) , Clock Enable (CKE) or On-Die-Termination (ODT) activation.

 An electronic system may consist of one or a plurality of data processing elements - wherein the process of processing may include computation, analysis, storage of data or transmission of data over a network or peripheral bus - connected to a plurality of storage devices are. Examples of data processing elements include, but are not limited to, CPUs, CPU caches, application specific integrated circuits, peripheral buses, Direct Memory Access (DMA) engines, or network interface devices. In the many DRAM configurations, a plurality of memory circuits are bundled together into modules; for example, in modules described by the Dual Inline Memory Module (DIMM) standard. Within a module, some devices may transmit data in parallel along separate DQ groups, while others may all be connected in parallel to the same transmission lines within a DQ group. To reiterate, in many typical DRAM configurations, a plurality of modules may then be connected in parallel to form a channel. In addition to the memory modules, each channel is connected to exactly one data processing element, hereinafter referred to as host. Each memory device may be connected to the host via a portion of a half duplex DQ bus (as opposed to a full duplex DQ bus), or may also be connected to the same DQ transmission lines as several other memory devices - either on the same module or on other neighboring modules in the same channel. Therefore, there is a risk that a memory device might choose to assert data on the DQ bus or at the same time as other memory devices on the same bus, and thus there is a need for arbitration on the bus. Therefore, SDRAM protocols are based on a time-window centralized arbitration scheme: by default, the host is the only device allowed to transmit data on the DQ bus, and by default, all storage devices leave their DQ lines high impedance most of the time. When a command requiring a response is sent to a particular memory device, the device is allowed to transmit data on the DQ bus, but only within a certain time window after the first pulse of the command. The window starts a fixed number of clock cycles after the instruction and has a typical duration of only one or two clock cycles longer than the time required to transfer the data. Storage devices that transfer data outside of this window either fail to successfully send their data to the host, or will corrupt data coming from neighboring storage devices.

The DQ bus arbitration scheme used by these clock data parallel SDRAM protocols works well for DRAM. The Technology behind DRAM devices has advanced to the point where their data access times are extremely consistent and predictable. However, DRAM is a relatively power hungry technology because it requires frequent refreshing thousands of times per second.

Non-volatile memory, such as Phase-Change Random Access Memory (PCM), Oxidative Resistive Random Access Memory (OxRAM or ReRAM), Conductive Bridge Random Access Memory (CBRAM), NAND Flash (NAND), Magnetic Tunnel Junction-based Magnetic Random Access Memory (MRAM), Memristor, NOR Flash (NOR), Spin Torque Transfer Magnetic Memory (STT-MRAM), and Ferroelectric Random Access Memory (FeRAM), all promise data access at low latency for data, allowing for lower power consumption Many data-heavy workloads can be optimized and can soon provide higher density random access memory than DRAM. However, they require somewhat looser data access protocols than DRAM. All of these non-volatile memories have non-deterministic read and write latencies. It is impossible at the time a read or write command is written to know exactly how long it would take to access the data to or from a non-volatile memory cell for all NVM options and for all Access or commit NVM device architectures. However, it is possible to mimic deterministic latencies. Deterministic latencies can be mimicked by accepting worst case timing conditions or abandoning a read that may take too long. Modifications of DDR SDRAM protocols could be specified based on pessimistic read or write latency specifications. For example, a memory that commits most writes within 100 ns, but occasionally, for unpredictable reasons, may take 10 us to commit data, use a DDR protocol that does not allow writes to as many as 10 μs after the previous write, and no reads in this period of time (because in some storage technologies, write operations mean that reads must also be delayed). However, this would represent a drastic limit on the maximum bandwidth achievable by such a device and could further limit the performance of other devices on the same channel. Conversely, one can envision a modification of the standard DDR or SDR or QDR SDRAM protocols which allow flexibility for non-deterministic read latencies and non-deterministic writes. In one embodiment, this protocol is referred to as a synchronous non-volatile RAM (hereinafter SNVRAM, Synchronous Non-Volatile RAM) protocol.

For example, in some embodiments, the read command may be partitioned by SNVRAM protocols into three smaller commands. Previously, where a read command sequence consisted of two parts: an activation command followed by a read to indicate the row and column of requested data, the command would now consist of an activation command, a read command, and finally, after an indefinite delay, a send command. The activation / reading combination would specify the two-part requirement to read a particular area. However, no response would be sent after the read command; instead, the memory device would assert a signal, referred to for example as "READ READY" (sometimes referred to herein as "R_RDY") back to the host at some undetermined time after the read command. This assertion would then prompt the host to issue the SEND command because another SDRAM activity is allowed to transfer the fully extracted data from the storage device back to the host. The response from the SEND command would go out over the shared DQ bus within a predetermined window following the SEND command. In this way, the typical read command would support non-deterministic read latencies; however, performance characteristics such as the average minimum latency or overall bandwidth of the system are not limited by the slowest possible read operation. The average performance of the protocol is consistent with the typical performance of the device, while there is still some flexibility for outliers, which are clearly expected as a physical consequence of media choice.

• – Ähnlich wie bestehende SDRAM- oder DDR-Protokolle unterstützt es die Kommunikation zwischen einem einzelnen Host und einer Mehrzahl von Speichervorrichtungen auf dem gleichen Speicherkanal. Hosts können an separate Speicherkanäle angeschlossen sein, obwohl jeder Kanal unabhängig arbeitet, und somit spezifiziert das Protokoll nicht das Verhalten von Vorrichtungen in anderen Kanälen. Übertragungsleitungen für den Betrieb eines Kanals können ausschließlich von diesem Kanal verwendet werden. In anderen Ausführungsformen kann der Host an eine einzelne Speichervorrichtung angeschlossen sein, und diese Speichervorrichtung kann die Befehle und Daten an eine zweite Vorrichtung in einem verketteten Stil der Bereitstellung weiterleiten.
• – Wie bei bestehenden SDRAM- oder DDR-Protokollen kann jedes Signal oder jeder Bus vom Host zum Kanal synchron zu einem Taktsignal im Anschluss an eine parallele Übertragungsleitung sein.
• – Wie bei bestehenden SDRAM- oder DDR-Protokollen existieren logische Befehle, wie beispielsweise „Adressblock aktivieren“, „Element innerhalb des aktiven Adressblocks lesen“ oder „Zum Element innerhalb des aktiven Adressblocks schreiben“, welche entlang eines Befehlsbusses gesendet werden können.
• – Wie bei bestehenden SDRAM- oder DDR-Protokollen kann der Befehlsbus mit einem Master-Takt oder Master-Befehls-Strobe für den Kanal synchronisiert werden.
• – Wie bei bestehenden SDRAM- oder DDR-Protokollen können Daten, welche von der Speichervorrichtung zurückkehren, entlang eines separaten Datenbusses gesendet werden, der aus einer Mehrzahl von Übertragungsleitungen besteht, die als DQ-Bus bezeichnet werden.
• – Wie bei bestehenden SDRAM- oder DDR-Protokollen kann jede Leitung im DQ-Bus in einigen Ausführungsformen mit dem Master-Takt synchron sein. In anderen Ausführungsformen ist der DQ-Bus synchron mit einem separaten DQ-Strobe-Signal (das entweder durch den Host oder durch die Speichervorrichtung erzeugt wird), nachfolgend als DQS bezeichnet. In einigen Ausführungsformen können mehrere DQS-Leitungen vorhanden sein, welche jeweils einer Teilmenge der DQ-Busleitungen entsprechen.
• – Wie bei bestehenden SDRAM- oder DDR-Protokollen existieren einige Ausführungsformen, in denen der DQ-Bus bidirektional sein kann und speicherbare Daten vom Host zur Speichervorrichtung aufnehmen kann. Andere Ausführungsformen können einen separaten Schreib-DQ-Bus umfassen.
• – Wie bei bestehenden SDRAM- oder DDR-Protokollen können Daten vom Host zur Speichervorrichtung auf einem DQ-Bus je nach der betrachteten Ausführungsform synchron entweder mit dem Master-Takt oder den entsprechenden DQS-Leitungen übertragen werden.
• – Wie bei bestehenden SDRAM- oder DDR-Protokollen können die DQ-Busse zusätzlich zum einzelnen Host an mehrere Speichervorrichtungen angeschlossen sein. Die Arbitrierung auf diesem Bus erfolgt auf der Basis von Zeitfenstern. Wenn eine Speichervorrichtung vom Host einen Befehl empfängt, welcher eine Antwort erfordert, hat sie ein schmales Zeitfenster, in dem sie den DQ-Bus besitzt und Daten assertieren kann.
• – Wie bei bestehenden SDRAM- oder DDR-Protokollen können innerhalb eines Kanals Speichervorrichtungen als eine Mehrzahl gruppiert werden, um koordinierte Module auszubilden.
• – SNVRAM-Protokolle unterscheiden sich typischerweise von SDRAM-Protokollen, da es zusätzliche Steuerleitungen gibt, welche Signale vom Speichersystem zum Host senden. (Typische SDRAM-Schnittstellen umfassen nur Steuersignale, welche vom Host zum Speichersystem gesendet werden.) Diese zusätzlichen Steuerleitungen werden nachfolgend als „Antwortbus“ (oder RSP) bezeichnet. Der Antwortbus kann in einigen Ausführungsformen synchron mit dem Master-Takt sein oder kann in anderen Ausführungsformen ein eigenes Strobe-Signal aufweisen, welches durch das Speichermodul generiert wird. Der Antwortbus umfasst, ist aber nicht darauf beschränkt, Signale, welche für unsere Zwecke hier als „LESEN BEREIT“ (R_RDY) und „CREDIT-INKREMENT SCHREIBEN“ (WC_INC) identifiziert werden. Es sollte jedoch beachtet werden, dass verschiedene Ausführungsformen von SNVRAM-Protokollen elektrische Signale mit ähnlichen Funktionen aufweisen können, obwohl das Protokoll sie mit einem anderen Namen bezeichnen kann. Dementsprechend versteht es sich, dass spezielle Signalnamen, welche hierin verwendet werden, lediglich Beispiele sind.
• – In einigen Ausführungsformen von NVRAM-Protokollen kann der Antwortbus von allen Modulen in einem Kanal gemeinsam genutzt und durch den Host arbitriert werden, oder in anderen Ausführungsformen kann der Antwortbus aus unterschiedlichen Übertragungsleitungen bestehen – die nicht von irgendwelchen Modulen geteilt werden – welche nur von jedem Modul zum Host weitergeleitet werden, ohne elektrischen Kontakt mit anderen Modulen zu machen.
• – Genau wie verschiedene Ausführungsformen der SDRAM- oder DDR-Protokolle Daten mit protokollspezifizierten Raten übertragen, können Daten auf jedem Befehlsbus für die Übertragung bei SDR-, DDR- oder QDR-Raten durch die spezielle Protokollausführungsform spezifiziert werden.
• – Daten auf einem beliebigen Befehlsbus, Takten oder Strobes können je nach den in der Ausführungsform des SNVRAM-Protokolls enthaltenen Spezifikationen single-ended oder differentiell gesendet werden.
• – SNVRAM-Protokolle bieten eine einfache Möglichkeit, das unregelmäßige Verhalten von nicht-deterministischen, nicht-flüchtigen Medien zu berücksichtigen, ohne deren Bandbreite unnötigerweise einzuschränken. Jedoch gibt es viele andere Möglichkeiten, welche durch solche Protokolle realisiert werden können. Zusätzlich zur Kompensierung für das nicht-deterministische Verhalten des Speichers können diese Protokolle auch verwendet werden, um Zeit für verschiedene Wartungsaufgaben und Datenqualitätsverbesserungen bereitzustellen, wie beispielsweise Fehlerkorrektur, I/O-Scheduling, Speicher-Wear-Levelling, In-situ-Mediencharakterisierung und Protokollierung von controllerspezifischen Ereignissen und Funktionen. Sobald die Hardware, welche diese Funktionen implementiert, komplexer wird, wird die Konkurrenz für Hardwareressourcen, die diese Funktionen durchführen, zu einer weiteren potentiellen Verzögerungsquelle. Alle derartigen Verzögerungen können bei der Verwendung eines standardmäßigen SDRAM-Kommunikationsprotokolls signifikante Leistungs- oder Zuverlässigkeitsprobleme bewirken. Die Verwendung eines nicht-deterministisch zeitgesteuerten SNVRAM-Protokolls ermöglicht jedoch einen flexiblen Betrieb und Freiheit von Hardwarekomplexität. Ferner ermöglichen nicht-deterministische Lese-Timings die Möglichkeit einer gelegentlich schnelleren Leseantwort durch Caching.

In one embodiment, the SNVRAM includes the following properties:

• Similar to existing SDRAM or DDR protocols, it supports communication between a single host and a plurality of storage devices on the same memory channel. Hosts can be connected to separate memory channels, although each channel operates independently, and thus the protocol does not specify the behavior of devices in other channels. Transmission lines for the operation of a channel can only be used by this channel. In other embodiments, the host may be connected to a single storage device, and this storage device may forward the commands and data to a second device in a chained style of deployment.
• As with existing SDRAM or DDR protocols, any signal or bus from the host to the channel may be synchronous with a clock signal following a parallel transmission line.
• - As with existing SDRAM or DDR protocols, there are logical commands such as "Enable Address Block", "Read Element Within Active Address Block", or "Write To Element Within Active Address Block" which can be sent along a command bus.
• - As with existing SDRAM or DDR protocols, the command bus can be synchronized with a master clock or master command strobe for the channel.
• As with existing SDRAM or DDR protocols, data returning from the memory device can be sent along a separate data bus consisting of a plurality of transmission lines called a DQ bus.
• As with existing SDRAM or DDR protocols, in some embodiments each line in the DQ bus may be synchronous with the master clock. In other embodiments, the DQ bus is synchronous with a separate DQ strobe signal (generated either by the host or by the memory device), hereinafter referred to as DQS. In some embodiments, multiple DQS lines may be present, each corresponding to a subset of the DQ bus lines.
• As with existing SDRAM or DDR protocols, there are some embodiments in which the DQ bus may be bidirectional and may receive storable data from the host to the storage device. Other embodiments may include a separate write DQ bus.
• As with existing SDRAM or DDR protocols, data from the host to the memory device on a DQ bus may be transmitted synchronously with either the master clock or the corresponding DQS lines, depending on the embodiment under consideration.
• - As with existing SDRAM or DDR protocols, the DQ buses may be connected to multiple storage devices in addition to the single host. The arbitration on this bus is based on time slots. When a storage device receives a command from the host requiring a response, it has a narrow window of time in which to own the DQ bus and assert data.
• As with existing SDRAM or DDR protocols, memory devices may be grouped as a plurality within a channel to form coordinated modules.
• SNVRAM protocols typically differ from SDRAM protocols because there are additional control lines that send signals from the storage system to the host. (Typical SDRAM interfaces include only control signals sent from the host to the memory system.) These additional control lines are hereafter referred to as a "response bus" (or RSP). The response bus, in some embodiments, may be synchronous with the master clock or, in other embodiments, may have its own strobe signal generated by the memory module. The response bus includes, but is not limited to, signals identified for our purposes as "READY READY" (R_RDY) and "WRITE CREDIT INCREMENT" (WC_INC). It should be noted, however, that various embodiments of SNVRAM protocols may have electrical signals with similar functions, although the protocol may refer to them by a different name. Accordingly, it should be understood that specific signal names used herein are merely examples.
• In some embodiments of NVRAM protocols, the response bus of all modules in a channel may be shared and arbitrated by the host, or in other embodiments, the response bus may consist of different transmission lines - which are not shared by any modules - which only by each Module to be forwarded to the host without making electrical contact with other modules.
• Just as various embodiments of the SDRAM or DDR protocols transmit data at protocol-specified rates, data on each command bus may be specified for transmission at SDR, DDR, or QDR rates by the particular protocol embodiment.
• Data on any command bus, clocks or strobes may be sent single-ended or differential depending on the specifications contained in the SNVRAM protocol embodiment.
• - SNVRAM protocols provide an easy way to account for the irregular behavior of non-deterministic, non-volatile media without unnecessarily limiting their bandwidth. However, there are many other possibilities that can be realized by such protocols. In addition to compensating for non-deterministic memory behavior, these protocols can also be used to provide time for various maintenance tasks and data quality enhancements, such as error correction, I / O scheduling, memory wear leveling, in-situ media characterization, and the like Logging of controller-specific events and functions. As the hardware implementing these functions becomes more complex, the competition for hardware resources performing these functions becomes another potential source of delay. Any such delays can cause significant performance or reliability problems when using a standard SDRAM communication protocol. However, the use of a non-deterministic timed SNVRAM protocol allows flexible operation and freedom from hardware complexity. Furthermore, non-deterministic read timings allow the possibility of occasionally faster read response through caching.

Discussion of the drawings

Turning now to the drawings, is 1 a block diagram of a host 100 in communication with memory systems of one embodiment. As used herein, the term "in communication with" may mean directly in communication with or indirectly in communication with over one or more components described or shown herein or not. In this illustration, two memory systems are shown (memory system A and memory system B); however, it should be understood that more than two memory systems may be used or only one memory system may be used. In this embodiment, the host includes 100 one or more central processing units (CPUs, Central Processing Units) 110 and a memory controller 120 , In this illustration, there are two CPUs (CPU A and CPU B); however, it is understood that more than two CPUs may be used or only a single CPU may be used. The memory controller may also be connected to devices other than just CPUs, and may be configured to handle memory requests on behalf of other devices, such as, but not limited to, network cards or other storage systems (eg, hard disk or solid state drive). SSD)), forward. Further, the memory controller may forward memory requests to the memory controller on behalf of one or more software applications executing on the CPU 120 for accessing the connected storage systems.

In this embodiment, the host includes 100 also a memory controller 120 in communication with the CPUs 110 (although in other embodiments, no memory controller is used) that communicates with the memory systems using a communication interface, such as a clock data parallel interface (eg, DDR), and operates according to a particular protocol (eg, one from Joint Electron Device Engineering Council (JEDEC)). In one embodiment, the memory controller correlates 120 Access requests to the storage systems from the CPUs 110 and sorts responses from the storage systems and delivers them to the appropriate CPUs 110 ,

As well as in 1 The storage system A comprises a media (non-volatile memory) controller 130 in communication with a plurality of non-volatile memory devices 140 , In this embodiment, storage systems A and B contain the same components; the memory system A thus also includes a media (non-volatile memory) controller 150 in communication with a plurality of non-volatile memory devices 160 , It should be noted that in other embodiments, the storage systems may include different components.

The media controller 130 (sometimes referred to as a "non-volatile memory (NVM) controller" or "controller") may take the form of processing circuitry, a microprocessor or processor, and a computer-readable medium having computer-readable program code (e.g. B. Firmware), for example, by the (micro) processor, logic gates, switches, an application specific integrated circuit (ASIC, Application Specific Integrated Circuit), a programmable logic controller and an embedded microcontroller executable. The controller 130 can be configured with hardware and / or firmware to perform the various functions described below and shown in the flowcharts.

In general, the controller receives 130 Requirements for accessing the storage system from the storage controller 120 in the host 100 , processes and sends the requests to the non-volatile memory 140 and returns responses back to the memory controller 120 , In one embodiment, the controller 130 take the form of a non-volatile (e.g., flash) memory controller which can format the non-volatile memory to ensure that the memory operates properly, can set bad non-volatile memory cells and can allocate spare cells, which will be replaced for future faulty cells. A portion of the spare cells may be used to accommodate firmware for operating the non-volatile memory controller and implementing other features. When operating when the host 100 Data must read from the non-volatile memory or data in the non-volatile memory To write memory, it communicates with the non-volatile memory controller. If the host 100 provides a logical address where data is to be read / written, the flash memory controller can receive the information from the host 100 convert received logical address into a physical address in non-volatile memory. (Alternatively, the host 100 provide the physical address.) The non-volatile memory controller may also perform various operations of indefinite duration from the perspective of the host, such as, but not limited to, wear-leveling (distribution of writes to reduce the wear on specific ones) Avoid memory blocks that would otherwise be written to repeatedly) and garbage collection (after a block is full, only the valid pages of data are moved to a new block so that the full block can be erased and reused). Learn more about a specific embodiment of the controller 130 are in connection with 6 explained.

A non-volatile storage device 140 may also take any suitable form. For example, a non-volatile memory device 140 a single memory-contains one or more memory dies and may be equipped with or without an internal controller. As used herein, the term "die" refers to a collection of nonvolatile memory cells and associated circuitry for managing the physical operation of these nonvolatile memory cells formed on a single semiconductor substrate. A non-volatile memory die 104 may include any non-volatile storage medium, including NAND flash memory cells, NOR flash memory cells, PCM, RRAM, OxRAM, CBRAM, MRAM, STT-RAM, FeRAM, or any other non-volatile technology. Also, volatile memory that mimics non-volatility may be used, such as a volatile memory that is battery backed or otherwise protected by an auxiliary power source. The memory cells may take the form of solid-state (e.g., flash) memory cells and may be one-time programmable, few-programmable or many-time programmable. The memory cells may also be single-level cells (SLC), multiple-level cells (MLC), triple-level cells (TLC) or use other memory cell level technologies now known or later developed. Also, the memory cells may be fabricated in a two-dimensional or three-dimensional manner. Some other storage technologies have been discussed above, and further discussion of possible storage technologies that may be used is further provided below. Also, different storage technologies may have different algorithms (eg, existing program and wear-leveling) that apply to this technology.

For simplicity shows 1 a single line representing the controller 130 and the non-volatile storage device 140 links; it is understood that this connection may contain a single channel or multiple channels. For example, in some architectures, 2, 4, 8, or more channels may exist between the controller 130 and a storage device 140 exist. Accordingly, in any of the embodiments described herein, more than a single channel may exist between the controller 130 and the storage device 140 exist even if a single channel is shown in the drawings.

The host 100 and the storage systems may take any suitable form. For example, in one embodiment (in 2A shown) the form of a non-volatile dual in-line memory module (NV-DIMM, non-volatile Dual In-line Memory Module) 200 to, and the host 100 takes the form of a computer with a motherboard that accepts one or more DIMMs. Im in 2A shown NV-DIMM 200 There are nine non-volatile storage devices 40 , and the NV-DIMM 200 has an interface 210 which comprises 9 data input / output DQ groups (DQ0-DQ8), a command bus and a response bus. Of course, these are just examples, and other implementations may be used. For example, shows 2 B an alternative embodiment in which the storage system is a distributed controller 31 and a master controller 212 (although not shown, with all distributed controllers 31 is connected). Compared to the storage system in 2A communicates each NVM device 41 with their own NVM controller 31 instead of all NVM devices communicating with a single NVM controller. In one embodiment, the master controller performs 212 through any required synchronization activity, including determining when all distributed controllers 31 are read to send the RD_RDY signal, which will be discussed in more detail below.

As mentioned above, multiple memory systems may be used in which signals may be passed through one memory system to accomplish another. This is in 3 shown. In 3 Storage system A is closer to the host 100 as the storage system B. The arrow 300 represents shared memory input signals from the host 100 be sent to the command pin in both the first and second memory systems. Examples of shared memory input signals, which may be used include, but are not limited to, an address signal, a read chip select signal, a bank group signal, a command signal, an enable signal, a clock enable signal, a termination control signal, and a command identifier (ID) signal. The arrow 310 represents a memory channel clock that can also be sent on the command pin. The arrow 320 represents shared memory output signals that can be sent on the DQ0-DQ8 groups. Examples of shared memory output signals include, but are not limited to, data signals, parity signals, and data strobe signals. The arrow 330 represents dedicated memory input signals to the memory system B, and the arrow 350 represents dedicated memory input signals to the memory system A. Examples of dedicated memory input signals that may be transmitted on the command pin include, but are not limited to, clock enable signals, data strobe signals, chip select signals, and termination control signals. The arrow 340 represents a device dedicated response line to the storage system B, and the arrow 360 represents a device dedicated response line to the memory system A. Examples of signals sent on the device dedicated response lines that can be sent on the command pin include, but are not limited to, Read Data Readiness (R_RDY) signals, a Read Identifier (ID). Signal and a write flow control signal. These signals will be discussed in more detail below.

One aspect of these embodiments is like the NVM controller 130 handles read and write commands in the storage system. Before discussing this aspect of these embodiments, the flowchart becomes 400 in 4 to illustrate how a conventional host reads data from a conventional DDR-based DRAM DIMM. The flowchart 400 becomes in connection with the time diagram 500 in 5 discussed. As in 4 shown when the host receives data from the DIMM (as a "device" in 4 designated) (operation 410 ), the memory controller in the host sends an activation command with the upper address (Operation 420 ). The memory controller in the host then sends a read command with the lower address (Operation 430 ). This is referred to as the "Act." And "Rd" fields on the command / address line in 5 shown. The memory controller in the host then waits a predetermined period of time (sometimes referred to as the "preamble time") (act 440 ). This is called a "predefined delay" in 5 shown. After the predetermined ("deterministic") time has elapsed, the memory controller in the host accepts the data (with data strobes for fine-grained time synchronization) (act 450 ) (Fields D1-DN on the data line in 5 ), and the data is provided to the host (operation 460 ).

As mentioned above, although this interaction between a host and the storage system is adequate when the storage system is a DRAM DIMM, complications may arise when using a deterministic protocol with an NV DIMM, since the mechanism behind reading and writing in the non-volatile memory may result in delays exceeding the specified time duration for a read or write operation according to the protocol. To accommodate this, some emerging standards allow for "non-deterministic" read and write operations. According to such standards, read and write operations to the NV-DIMM need not be completed in a certain amount of time.

In the case of a read operation, the NV-DIMM informs the host 100 when the requested data is ready so that the host can retrieve it. This is in the flowcharts 600 . 700 in 6 and 7 and the timing diagram 800 in 8A shown. As in 6 shown when the host 100 Request data from the storage system (operation 610 ), the host generates 100 a double data rate identifier (DDR-ID) for the request (operation 620 ). The host 100 then associates the DDR ID with a host request ID (eg, an ID from the CPU or other entity in the host 100 who requested the data) (operation 630 ). Next, the host sends 100 the activation command and the upper address (Action 640 ) and then sends the read command, the lower address and the DDR ID (Action 650 ). This is indicated by the fields "Act." And "Rd + ID" on the command / address line in 8A shown. ( 8B is another time chart 810 for the reading process discussed above, but here there are two read commands, and the later received read command (read command B) is completed before the first received read command (read command B). As such, the data B before the data A is sent to the host 100 returned.)

In response to receiving the read command, the controller takes 130 an indefinite amount of time to the data from the non-volatile memory 140 to read. After the data has been read, the controller shares 130 the host 100 by sending a R_RDY signal on the reply bus indicating that the data is ready (act 710 in 7 ). In response, the host sends 100 a "Send" command on the command / address line (Action 720 ), and after a predefined delay gives the controller 130 the data to the host 100 back (operation 730 ) (as indicated by the fields "D1" - "DN" on the data line and the field "ID" on the ID line in 8B shown). The memory controller 120 in the host 100 then accepts the data and the DDR ID (operation 740 ). Next, the memory controller determines 120 whether the DDR ID with a specific host ID from any of the CPUs 110 in the host 100 is associated (process 750 ). If so, the memory controller gives 120 the data to the correct CPU 110 back (operation 760 ); otherwise, the memory controller ignores 120 the data or returns an exception (operation 770 ).

In the case of a write operation, the host may 100 be limited in that it has more than a certain number of pending write commands to ensure that the non-volatile memory device receives no more write commands than it can handle. This will be in the writing time diagram 820 in 8C shown. As in 8C shown, the host decreases 100 each time he issues a write command, his write flow control credits (marked "WC" in the drawing). When a write operation completes, the media controller sends 130 a response to the host 100 for him to increase his write flow control credits.

The protocol discussed above is an embodiment of an NVRAM protocol that supports read and write operations of unpredictable duration. As previously discussed, the controller may 130 in some embodiments, exploit the non-deterministic aspect in read and write operations to perform time-consuming actions (which may be referred to herein as indefinite-duration operations from the perspective of the host) for which it may be in accordance with conventional DRAM-based DIMM standards does not have the time. These operations, which have an indeterminate duration from the perspective of the host, such as memory and data management operations, may be important to the operation of the NV-DIMM. For example, a non-volatile memory device 140 have lower stamina (ie, number of writes before failure) compared to DRAM and store data less reliably (eg, due to internal memory errors that cause bits to be stored improperly). These problems can be even more pronounced with emerging non-volatile memory technologies that would likely be used as a DRAM replacement in an NV-DIMM. As such, in one embodiment, the NV-DIMM exploits that it is not "pressurized" to perform operations of indeterminate duration from the perspective of the host (eg, wear-leveling and error-correcting operations) according to conventional practices DRAM-based DIMM standards may not perform in the assigned time.

In general, an operation that has an indeterminate duration from the perspective of the host refers to an operation which (1) naturally does not have a predetermined duration (e.g., since the duration of the operation depends on one or more variables) or ( 2) has a predetermined duration, but this duration is not known to the host (eg, a decryption operation may have a predetermined duration, but this duration is indeterminate from the perspective of the host because the host does not know if the storage system is decrypting will perform or not). An "operation having an indefinite duration from the perspective of the host" may take any suitable form. For example, such an operation may be a "storage and data management function" which is an action on the part of the controller 130 is to manage the health and integrity of the NVM device. Examples of storage and data management functionality include, but are not limited to, wear-leveling, data movement, metadata read / write (e.g., logging, controller status and status tracking, wear-leveling tracking updates), data decoding variations (ECC engine). Variations (syndromes, BCH vs. LDPC, soft-bit decoding), soft reads or re-reads, layered ECC requiring increased transfers and reads, RAID or parity reads with their compounded decode and component latencies), resource conflict (ECC) Engine, channels, NVM property (die, block, level, I / O, buffers), DRAM access, scrambler, other hardware engines, other RAM conflict), controller exceptions (bugs, peripherals (temperature, NOR) Media characterization activities (determining the effective age of memory cells, determining bit error rate (BER), or probing memory defects Further, the media controller may introduce elements, such as caches, that have the inverse effect (fast programs, temporary writes with reduced retention or other properties) and serve to speed up read or write operations in a way that would be difficult to predict deterministically ,

Further, indefinite duration operations from the perspective of the host may include program refreshes, verification steps (eg, skip verification, regular settings, tight settings), data movement from one media / state to another storage location, or other state (e.g., SLC TLC, ReRam to NAND, STT-MRAM to ReRam, burst settings to Hardened settings, low ECC to high ECC), and longer media settings (eg, simpler voltage transients) include, but are not limited to. Such operations can for example, for endurance expansion, retention improvement or attenuation, and performance acceleration (eg, fast writing of this data burst or more programming of that data in the preferred direction so that future reads will stabilize faster).

The media / NVM controller 130 may be equipped with various hardware and / or software modules to perform these memory and data management operations. For example, as used herein, a "module" may take the form of a packaged functional hardware unit designed for use with other components, a portion of program code (e.g., software or firmware) provided by a (micro) processor or processing circuitry executable, which typically perform a particular function of related functions, or a stand-alone hardware or software component interfaced to a larger system.

9 is a block diagram of an NVM controller 130 an embodiment showing various modules that may be used to perform memory and data management functions. In this particular embodiment, the controller is 130 Designed to perform encryption, error correction, wear-leveling, command-planning and data aggregation. It should be noted, however, that the controller 130 may be designed to perform other types and numbers of memory and data management functions.

As in 9 shown includes this NVM controller 900 a physical layer 900 and a nonvolatile RAM ("SNVRAM") protocol logic interface (including instruction and memory location decoding) 905 which is used to communicate with the host 100 (via the memory controller 120 ) to communicate. The physical layer 900 is responsible for the latching in the data and commands, and the interface 905 separates the commands and memory locations and handles additional signaling pins between the host 100 and the controller 130 , The controller 130 also includes a number N of memory state machines (MemFSMs, Memory Finite State Machines) 910 and a physical NVM layer (Phy) 910 , which are connected to a number M of non-volatile memory devices 140 communicate.

The controller points between these input and output parts 130 a write path on the right, a command path in the middle, and a read path on the left. Although not shown, the controller may 130 a processor (eg, a CPU that executes firmware) having the various in 9 can control elements shown and connected via an interface with them. Turning first to a write operation after a command and location through the interface 905 the address becomes a wear-leveling address translation module 955 Posted. In this embodiment, the host sends 100 a logical address with a command to write data and the wear-leveling address translation module 955 translates the logical address into a physical address in memory 140 , In this translation, the wear leveling translation module shuffles 955 the data to be placed at a physical address that has not been very worn. The wear-leveling data movement module 960 is responsible for rearranging the data if a sufficiently unused space within the address translation scheme can not be found. The resulting physical address, along with the associated command and associated address, where the data resides in local buffers within the controller 130 can be found in the NVM I / O scheduling module 940 which reads and writes to memory 140 plans. The NVM I / O scheduling module 940 may include other functions for planning, such as, but not limited to, deletion, setting changes, and defect management.

In this embodiment, the data, parallel to the address translation, for a write operation is first obtained by the encryption engine 925 encrypted. Next, the media error correction code (ECC) generates an error correction code encoder 930 ECC protection for data in idle state in NVM memory 140 , Data protection at rest may be preferred because non-volatile memories are much more susceptible to errors than DRAM when retrieving previously stored data. However, decoding data with error correction is not always a constant time operation, so it would be difficult to perform such operations according to deterministic protocols. Although ECC is used in this example, it will be understood that any suitable privacy scheme may be used, such as, but not limited to, cyclic redundancy check (CRC), redundant array of independent hard drives (RAID, redundant array of Independent disks), scrambling, data weighting / modulation, or other changes to protect against deterioration due to physical events such as temperature, time, and voltage stress (DRAM is also prone to error, but NVM is vulnerable to other errors.) Thus, each NVM in idle state probably requires one other Protection scheme. Often it is a trade-off between latency and cost). Although not shown to simplify the drawing, it should also be noted that other privacy systems from the controller 130 can be used to protect data when "on the go" between the host 100 and the controller 130 are and if they are in the controller 130 move (eg using CRC, ECC or RAID).

As mentioned above, other privacy schemes may be used as ECC. The following paragraphs provide some additional information about different privacy schemes.

With respect to ECC, some embodiments of error checking codes, such as BCH or other Hamming codes, allow the decoding engine, which can use near-instantaneous syndrome, to validate the validity of the data. However, a syndrome check error can bring about the solution of complex algebraic equations, which can lead to a significant delay. In addition, if multiple syndrome check errors occur simultaneously, hardware resource generated backlogs may be present due to the unavailability of hardware resources for decoding. However, these occasional delays can be handled by delaying read ready notification to the host. Other coding schemes, such as LDPC or additional CRC checks, may also be included for more efficient use of space or higher reliability, and although these other schemes are likely to have additional temporal variations to process the data coming from the storage medium, these variations can can also be handled by a simple delay of read ready signal.

Another form of data protection may take the form of soft bit decoding wherein the binary value of the data stored in the medium is measured with a higher confidence by measuring the analog values of the data stored in the physical storage medium several times, relative to several threshold values. Such techniques will take longer to implement and may add additional variability to the combined data read and decode process. However, these additional delays can be handled well if desired by delaying the read ready signal back to the host.

Furthermore, reliability can still be added using nested or layered error correction schemes. For example, the data in the medium can be encoded so that the data can survive N errors from all read A bytes and M (where M> N) can survive errors from all read B (where B> A) bytes. A size A small read may thus be optimal for fast operation, but suboptimal for data reliability given a very poor data block with more than N errors. Occasional problems in this scheme can be corrected by first reading and validating A bytes. If errors persist, the controller has the ability to read the much larger block with the penalty of a delay, but with the successful decoding of the data. This is another emergency decoding option enabled by the non-deterministic read timings granted by the SNVRAM supported media controller.

Also, gross malfunctions of a particular storage device could be encoded using RAID techniques. Data could be distributed across a plurality of memory devices to accommodate the complete malfunction of a particular number of memory devices within that set. Spare memory devices could be included as a fail-in-place replacement in a memory module to obtain redundancy data once a bad memory device is encountered.

Returning to 9 after the Media Error Correction Code (ECC) encoder 930 ECC protection for the data generated, the data becomes the write cache management module 935 which determines whether to buffer space in the write data cache 945 exists or not and where the data in these buffers 945 should be stored. The data is buffered in the write data cache 945 stored where they are stored until reading. Thus, if there is a delay in the scheduling of the write command, the data in the write data cache buffer 945 be stored indefinitely until the memory 140 is ready to receive the data.

As soon as the write command associated with this write cache buffer entry comes forward in the queue, the data entry is sent to the NVM write I / O queue 950 forwarded. If through the NVM I / O scheduler 940 is displayed, the command is issued by the NVM I / O Scheduler 940 to the NVM Data Routing, Command Routing, and Data Aggregation module 920 forwarded and the data is taken from the NVM Write I / O Queue 950 to the NVM data routing, command routing and Data aggregation module 920 forwarded. The command and the data are then forwarded to the appropriate channel. The memory state machine (MemFSM, Memory Finite State Machine) 910 This is for parsing the commands into finer-grained, NVM-specific commands and controlling the timing of these commands to the NVM devices 140 be responsible. The NVM Phy 915 controls the timing to an even finer level, ensuring that the data and command pulses are located at well-synchronized intervals with respect to the NVM clock.

Turning now to the read path when data read commands from the NVM devices 140 return, provides the NVM Data Routing, Command Routing and Data Aggregation module 920 the read data to the NVM read I / O queue 965 , In this embodiment, the read data may take one of three forms: data requested by a user, NVM register data (for internal use by the controller 130 ) and write validation data. In other embodiments, one or more of these data classes may be held in different queues. If the data was read for internal purposes, it will be taken from the internal read processing module 960 (for example, to check if previously written data was written correctly before sending an acknowledgment back to the host 100 is sent or a rewrite request to the scheduler 940 is sent). If the data was requested by the user, metadata indicating the command ID associated with the read data is appended to the data. These command-ID metadata are associated with the read data as they pass through the read pipeline (as indicated by the double-headed arrow). The data then becomes the media ECC decoder 975 which decodes the data and then to the decryption module 980 which decrypts the data before going to the read data cache 955 be sent. The data remains in the read data cache 955 until the host 100 requests them by identifying the command ID block. At this point, the data is sent to the interface 905 and physical layer 900 for transmission to the host 100 Posted.

10 is a flowchart 1000 a method of reading data using the controller 130 out 6 , As in 10 shown, the host sends 100 first a read request to the storage system (operation 1050 ). The NVM controller 130 in this embodiment, the following elements then extract from the request: Address, Read Request ID, and Request Length (Operation 1010 ). The NVM controller 130 then converts the logical address from the request to a physical address for wear-leveling (act 1015 ).

The NVM controller 130 then determines if the physical address corresponds to a portion of the storage array that is busy or unavailable for reads (operation 1020 ). If the memory part is busy or unavailable, the NVM controller is planning 130 reading the non-volatile memory devices 140 for a later date (operation 1022 ). At this later time, if the physical address becomes available (operation 1024 ), determines the NVM controller 130 whether there are any other higher priority operations that prevent reading (act 1026 ). If so, the NVM controller waits 130 (Process 1028 ).

If / when the memory part becomes available, the NVM controller sends 130 Read commands to the NVM devices 140 to read the requested data (operation 1030 ). The NVM devices 140 then return the requested data (operation 1035 ). Depending on the type of devices used, the NVM devices 140 return the data after a fixed, predetermined period of time. The NVM controller 130 can then process the returned data. For example, after the data, which from the various NVM devices 140 be returned, were aggregated (operation 1040 ), the NVM controller can 130 determine if the data passes an Error Correction Code (ECC) check (operation 1045 ). If the data fails the ECC check, the NVM controller can 130 initiate an error recovery process (operation 1046 ). After the error recovery process is completed (Operation 1048 ) or if the aggregated data has passed the ECC test, the NVM controller determines 130 whether the data is encrypted (act 1050 ). If the data is encrypted, the NVM controller initiates 130 a decryption process (operation 1052 ).

After the decryption process is completed (operation 1054 ) or if the data was not encrypted, the NVM controller determines 130 optional, whether the host 100 previously agreed to use non-deterministic reads (operation 1055 ). (Process 1055 allows the NVM controller 130 is used for both deterministic and non-deterministic reads, but may not be used in certain embodiments.) If the host 100 has a previous agreement, holds the NVM controller 130 the read data for a future send command (or set aside) (as discussed below) (act 1060 ). The NVM controller 130 also sends a signal on the line "READY TO READ" to the host 100 (Process 1065 ). When he is ready, the memory controller sends 120 in the host 100 a send command (operation 1070 ). In response to receiving the send command from the host 100 transmits the NVM controller 130 the processed read data together with the command ID to the host 100 (eg after a predefined delay (there may be global timeouts from the memory controller in the host)) 1075 ).

If the host 100 has not previously agreed to use non-deterministic reads (operation 1055 ), becomes the NVM controller 130 handle the reading as in the conventional system discussed above. That is, the NVM controller 130 will determine if the elapsed time exceeds the previously agreed transmission time (Operation 1080 ). If the elapsed time has not exceeded the previously agreed transmission time, the NVM controller transmits 130 the data to the host 100 (Process 1075 ). However, if the elapsed time has exceeded the previously agreed transmission time, the read operation failed (operation 1085 ).

Turning now to a write operation is 11 a flowchart 1100 which starts when the host 100 Has data to write (operation 1105 ). Next, the host checks 1110 to see if there is any available flow control credit for the write operation (operations 1110 and 1115 ). If a flow control credit is available, the host issues 100 the write request (operation 1130 ), and the media controller 130 receives the write request from the host 10 (Process 1125 ). The controller 130 then extracts the destination address and user data from the request (operation 1130 ). Since a non-deterministic protocol is used in this embodiment, the controller may 130 now spend time performing memory and data management operations. For example, if the data requires encryption (operation 1135 ), the controller encrypts 130 the data (operation 1140 ). Otherwise, the controller encodes 130 the data for error correction (operation 1145 ). As mentioned above, any suitable error correction scheme may be used, such as, but not limited to, ECC, Cyclic Redundancy Check (CRC), redundant Array of Independent Disks (RAID), scrambling, or data weighting /-modulation. Next, the controller uses 130 Wear leveling hardware (or software) to convert the logical address into a physical (NVM) address (act 1150 ). The controller 130 then determines if the write cache is full (operation 1155 ). If so, the controller signals 130 an error (operation 1160 ). An error may be signaled in any suitable manner, including, but not limited to, using a series of voltages on a dedicated pin or dedicated pins on the response bus, writing the error to a protocol (eg, in the NVM). Controller) or incrementing or annotating the error in the Serial Presence Detect (SPD) data. If this is not the case, the controller associates 130 a write cache entry with the current request (operation 1165 ) and write the data into the write cache (operation 1170 ).

The controller 130 then determines if the physical media is busy at the required physical address (act 1175 ). If so, the controller is planning 130 the write operation for future processing (operation 1180 ). If this is not the case, the controller waits 130 until the current operation is completed (Operation 1182 ), and then determines if a higher priority request is still outstanding (operation 1184 ). If this is not the case, the controller distributes 130 the data by write commands to the NVM devices 140 (Process 1186 ). The controller 130 then waits as there are typical delays in writing to NVM devices (Operation 1188 ). Next up is the controller 140 optionally, make sure the write commit was successful (operation 1190 ) by determining whether the write was successful (act 1192 ). If the write was unsuccessful, the controller determines 130 whether further attempts are justified (Act 1193 ). If this is not the case, the controller can 130 optionally apply error correction techniques (Operation 1194 ). If and when the write succeeds, the controller issues 130 the write cache entry free (operation 1195 ) and notifies the host 100 via additional write buffer space (operation 1196 ), and the write completes (operation 1197 ).

The flowcharts in 10 and 11 both describe the process for performing a single read operation or a single write operation. However, in many media controller embodiments, multiple read or write operations may occur in parallel, thus providing a continuous pipeline of read or write processes. Many of these steps will in turn support out-of-order processing. The flowcharts serve as an example of the steps that may be required to process a single read or write request.

In summary, some of the above embodiments provide a media controller that interfaces with a particular embodiment of the SNVRAM protocol is connected to a host and is also connected via an interface with a plurality of storage devices. In addition to using non-deterministic read and write timing features of the SNVRAM protocol, the media controller is specifically designed to improve the life of the media (NVM), optimally correct media errors, and plan for media requirements to optimize throughput while presenting a low latency, high bandwidth memory interface to the host. In this way, the media controller can manage the health and integrity of the storage medium by "massaging" storage idiosyncrasies. In addition, the media controller can collect and aggregate data from NVM chips for more efficient data processing and error handling.

There are many alternatives that can be used in these embodiments. For example, while a clock data parallel interface was present in the above examples, other types of interfaces may be used in various embodiments, such as, but not limited to, Serial Advanced Technology Attachment (SATA), Peripheral Component Interface Express (PCIe), NVMe (NonMedia). Volatile Memory Express), RapidIO, Industry Standard Architecture (ISA), Lightning, Infiniband or FCoE (Fiber Channel over Ethernet). Accordingly, while a parallel DDR interface was used in the above example, other interfaces, including serial interfaces, may be used in alternative embodiments. However, current serial ports may encounter long latencies and I / O delays (while a DDR interface provides fast access times). Also, as noted above, although the memory system takes the form of an NV-DIMM in the above examples, other types of memory systems may be used, including, but not limited to, embedded and removable devices such as a solid-state drive. SSD) or a memory card (such as Secure Digital (SD), Micro Secure Digital (Micro SD) card, or Universal Serial Bus (USB) drives).

As a further alternative, NVM chips can be built that can speak either standard DDR or newer SNVRAM protocols without the use of a media controller. However, the use of a media controller is currently preferred because currently existing NVM devices have much larger features than more sophisticated DRAM devices; On NVM chips can not be relied on to speak at current DDR frequencies. The memory controller can slow down DDR signals to communicate with the NVM chips. Moreover, the functions that the media controller performs may be relatively complex and expensive to integrate into the memory chips themselves. Furthermore, media controller technology is likely to evolve, and it may be desirable to enable the media controller to be separately updated to better handle a particular type of memory chip. That is, sufficient isolation of the NVM and NVM controller allows the incubation of new memories while also providing DRAM speed for mature NVMs. In addition, the media controller enables error checking codes and wear-leveling schemes that distribute data across all chips and handle defects, and there is an advantage of aggregating data together through a device.

As discussed above, the controller may 130 in some embodiments, exploit the non-deterministic aspect in read and write operations to perform time consuming actions that have an indeterminate duration from the perspective of the host. Although memory and data management operations have been mentioned above as examples of such actions, it will be understood that there are many other examples of such actions, such as monitoring the health of individual non-volatile media cells to protect them from wear, identifying failures in the circuits used to access the cells, ensuring that user data is transferred to or removed from the cells in a timely and consistent with the operational requirements of the NVM device, and ensuring that user data is reliably stored, not due to be lost or damaged by bad cells or media circuit errors. Further, in cases where sensitive data may be stored on such a device, operations of indefinite duration from the perspective of the host may include encryption as a management service to prevent theft of non-volatile data by malicious entities.

More generally, an indefinite-duration operation may include, but is not limited to, one or more of the following: (1) NVM activity, (2) protection of data stored in the NVM, and ( 3) Data movement efficiencies in the controller.

Examples of NVM activity include, but are not limited to, user data handling, non-user media activity, and scheduling decisions. Examples of user data handling include, but are not limited to, enhancement and mitigation of Endurance from the NVM (eg, wear-leveling data movement, where wear-leveling is the distribution of localized user activity over a larger physical space to extend the endurance of the device, and writing or reading the NVM in a manner that enhances endurance characteristics affecting that location), enhancement or mitigation of NVM retention (e.g., program refresh, data movement, and retention verification), different media latency handling to better manage the wear on the media during media activity (writes, reads, deletions, verifications, or others Interactions) (eg, use of longer or shorter latency techniques, as required for NVM handling, to enhance a desired property (persistence, retention, future read latency, BER, etc.)), and convolution of data from temporary Memory (SLC or STT-MRAM) to more permanent memory (TLC or ReRam). Examples of non-user media activities include, but are not limited to, device logs (e.g., errors, debug information, host usage information, warranty support information, settings, activity trace information, and device history information), controller status, and status tracking (e.g. B. Algorithm and condition tracking updates for improved or continuous behavior in case of power failure or power handling and inter-verification status conditions for media write confirmations, defect identifications and data protection updates for ECC (updating parity or layered ECC values), media characterization activities (eg characterizations of NVM age or BER and investigation of the NVM for defects) and remapping of defective areas.

Examples of the protection of data stored in the NVM include, but are not limited to, various ECC engine implementations (eg, BCH or Hamming (hardware implementation options of size, implementation parallelization, syndromes, and encoding implementation options such as Generator polynomial, protection level or special case arrangements), LDPC (e.g., size hardware implementation options, implementation parallelization, array size and clock rate, and encoding implementation options such as protection level and polynomial selection to favor media BER characteristics), parity (e.g. Layered protection of one of the above in any order (e.g., CRC on the user data, ECC over the user data, and CRC, two ECC blocks together receive another ECC, calculating the RAID over several blocks with ECC for a full strip of RAID), decode new attempt paths (eg. B. Options for initiating and using the other protection layers (eg, speculative soft-reading, waiting for failure before reading the entire RAID stripe, low power ECC engine modes vs. high performance)), ECC Retries with or without any of the following: speculative bit-flips, soft bit decodes, soft reads, new reads (eg, re-reads and soft reads (re-reading the same data with different settings) and decoding errors) and Data shaping for improved memory performance (eg, reduced interference between cells (eg, using a scrambler or weighted scrambler for improved detection circuit performance)).

Examples of data movement efficiencies in the controller include, but are not limited to, scheduling architecture and scheduling decisions. The scheduling architecture can affect the availability of individual vs. relate multiple paths for each of the following: prioritization, speculative early launches, parallelization, component acceleration, resource arbitration, and implementation options specific to that component. The amount, throughput, latencies and connections of each device resource will implicitly affect the planning. The scheduling architecture can also handle internal bus conflicts during transfers (eg AXI bus conflicts), ECC engines, NVM communication channels (eg bandwidth, speeds, latencies, idle times, congestion of traffic to other NVM, order - or prioritization options and efficiencies of use for command, data, status, and other NVM interactions), NVM access conflicts often due to the placement and internal circuit access of each particular NVM (e.g., die, block, level, I / O circuits, buffers, bays, arrays, word lines, strings, cells, combs, layers and bit lines), memory access (eg external DRAM, SRAM, eDRAM, internal NVMs and ECC on these memories), scrambler, internal data transfers, Interrupt delays, polling delays, processors and firmware delays (eg, processor code execution speed, code efficiency and function, thread or interrupt interchange), and cache engines (eg, efficiency of cache search operations, cache insertion costs, cache filling strategies, successful cache hits, and efficient cancellation of parallel NVM and controller activity and cache ejection strategies). Scheduling decisions may include command overlap detections and ordering, memory location decode and memory schemes (e.g., cached lookup tables, hardware controlled tables, and layered tables), controller exceptions (e.g., firmware hangs, component timeouts, and unexpected component states), peripheral handling (e.g. , Alternative NVM handling such as NOR or EEPROM, temperature, SPD (Serial Presence Detect) interactions on the NVDIMM-P, and alternative device access paths (eg, low-power modes and out-of-band commands); Power circuit status), and reduced power modes (eg, off, reduced power states, idle, idle active, and higher power states that may serve for accelerations or bursts) include, but are not limited to.

The memory system discussed above may benefit from the use of a command and address buffer and data buffers (DBs). An example of a command and address buffer is a Register Clock Driver (RCD). Although an RCD is used in the following examples, it should be understood that other types of command and address buffers may be used. In addition, a command and address buffer may have other functionality. For example, a command and address buffer, such as an RCD, may also include data-parallel decode synchronization capabilities to synchronize data flow into and out of the DBs.

RCDs and DBs have been used with DRAM-based DIMMs to improve signal integrity. For example, if long, stray electrical lines in the DIMM cause poor electrical characteristics on the command and address group of signals, the RCD will receive and repeat 1220 the command and the address to the DRAM chips 1210 to make sure they receive it. RDIMM (Registered DIMM) is an example of a DIMM with an RCD, and LRDIMM (Load Reduced DIMM) (or FBDIMM (Fully Buffered DIMM)) is an example of a DIMM having both an RCD and DBs ( a UDIMM (unbuffered DIMM) enforces electrical routing rules that affect the bus). Signal integrity and other issues may arise when using an NV-DIMM, particularly one with a media controller such as that discussed above. The following paragraphs discuss the general use of RCDs and DBs in this regard before turning to their use in an NV-DIMM.

Returning to the drawings are 12 and 13 Illustrations of a DRAM DIMM 1200 containing a plurality of DRAM chips 1210 , an RCD 1220 and a plurality of DBs 1230 having. Although not to simplify the drawings 12 and 13 shown is the RCD 1220 in communication with all DRAM chips 1210 and the DBs 1230 , In general, the DBs store 1230 Data related to the DIMM 1200 be sent or read from it, and the RCD 1220 serves as a repeater to send the command and address received on the CMD / Addr line of the DIMM to the DRAM chips 1210 to repeat. The RCD 1220 Also controls when the DBs 1230 release the data that they store.

12 shows the read flow in the DIMM 1200 , and 13 shows the write flow in the DIMM. As in 12 is shown, a read command from the RCD 1220 received on the CMD / Addr line (arrow 1). Next, the RCD communicates 1220 a "read" command to the address in each DRAM block 1210 , since each DRAM block is addressed here the same (arrow 2). The data will then be from each of the DRAMs 1210 read and to the appropriate DB 1230 moved (arrow 3). In the DRAM-based DIMM protocol, after receiving the read command, the DIMM has some time to deliver the data back to the host. After this time, the RCD signals 1220 then the DBs 1230 to release the data to the host (arrow 4). Between each of these steps a variation is allowed in this scheme. In this architecture, the RCD takes 1220 simply state that the data expires after the time in the DBs 1230 This is usually a safe assumption due to the reliable DRAM latency when reading data.

Turning now 13 to, a write command is issued in a write operation from the RCD 1220 received on the CMD / Addr line (arrow 1). Almost immediately thereafter, the RCD communicates 1220 to the DRAM blocks 1210 the writing process (arrow 2). Next, the DBs receive 1230 after a fixed time delay tWL, the data to be written (arrow 3) and then transmit the data to the DRAM blocks 1210 (Arrow 4).

14 is a representation of the internal states of the data flow in a DRAM-based DIMM. The earlier layer of decoding and forwarding makes it possible to assume that each sub-block in this representation is correctly decoded and understood as a group. Theoretically, each of the subgroups can be advanced to a larger set of data moving together. The dashed boxes in this drawing convey four of the groups that can be treated together. Although there are times when CMD / ADDR can go in earlier than the DQ data, the relationships are well formed so that this time delay can be ignored. In either case, a maximum of DQ and CMD / ADDR can describe the state of the physical layer.

Now that the general background of RCDs and DBs has been provided, the following paragraphs discuss the use of RCDs and DBs in an NV DIMM. Returning to the drawings 15 a block diagram a storage system 1500 which is the storage system discussed above 200 in 2A similar. Like the storage system 200 includes this storage system 100 an interface 1510 which includes 9 data input / output pins (DQ0-DQ8), command pins and response pins, an NVM controller 1530 and nine non-volatile memory devices 1240 includes. New in this embodiment are the RCD 1520 and the DBs 1550 ,

An advantage of this embodiment is that the RCD 1520 and the DBs 1550 act to electrically buffer the NV-DIMM. For example, as in the storage system 200 in 2A shown that the DQ traces are long and difficult to route, which may affect the signal integrity (SI) quality of the buses. In contrast, the tracks are 1560 between the DRAM bus pins and the RCD 1520 and the DBs 1560 relatively short, thereby ensuring signal integrity from the DRAM bus. These tracks 1560 can be rigorously specified for maximum SI and NV DIMM P operability in any of UDIMM, RDIMM, LRDIMM and any other DIMM configurations (now or later developed) without degrading bus integrity (this can increase supplier competition and reduce system integration challenges). That is, the speed of the lines 1560 may have sufficient signal integrity and speed to match other physical DRAM communications. In contrast, the wires can 1570 between the RCD 1520 and the DBs 1550 and the NVM controller 1530 as well as the lines 1580 between the NVM controller 1530 and the NVM devices 1540 be specified with looser specifications, as the communication on these lines 1570 . 1580 can be absorbed into the existing JEDEC specification latency responses (ie latency may be behind the RCD 1520 and the DBs 1550 isolated), or the electrical routing contained entirely within the DIMM can ensure sufficient SI for transmission. This enables multi-vendor development of DB and RCD chips and "agnostic" placement of the NVM devices and the NVM controller. Further, this allows sufficient isolation of the NVM devices and the NVM controller to allow the incubation of new memories while also providing DRAM speed for mature NVMs. In addition, the RAM buffers in the DBs 1550 and in the RCD 1520 with non-deterministic protocol sufficient to separate and align behaviors of NV DIMM P internals and DRAM bus externals.

In one embodiment, each DQx inferiert a grouping of data, strobe and clock signals from the memory controller 120 in the host 100 come. The number of sets of DQs could have a maximum of DQ7 or DQ8 in one deployment, but there are other maxima, such as DQ9. (Some specifications refer to these as CBs (check bits)). Accordingly, these embodiments may apply to any number of data group signals, and the maximum DQ group number is referred to herein as N. DQ and RCD signal timing and constraints within each group (eg, message content lines, strobes, and bars) can be very stringent. For example, the "message lines" may be either data in the case of DQ or command and address in the case of RCD. This ensures that each eight bytes of data and the instructions and addresses are received together and decoded correctly by group. Each message can be received and correctly received by the DBs 1550 or the RCD 1530 (depending on the appropriate group), so the overall timing constraints between each DQ and the RCD 1530 can be more lenient. The structure of delays of the entire DRAM bus can be much looser than a single edge of the DRAM bus clock rate. Thus, the DQ and the RCD 1530 able to correctly decode and encode to the corresponding and related buffers. In one embodiment, the memory controller sends 1530 the newsgroups all at once, and the correct placements and signal integrity rules are ensured so that the data will reach each component and be correctly decoded.

The basic operation of the RCD 1520 and the DBs 1550 similar to the operation of the RCD 1220 and the DBs 1230 in the example above with a DRAM-based DIMM, with some differences to the use of NVM devices 1540 and the NVM controller 1530 to take into account. In other words, the DBs generally store 1550 Data related to the NVM devices 1540 be sent or read from them, and the RCD 1520 serves as a repeater to the command and the address appearing on the CMD / Addr line of the storage system 1500 received to the NVM devices 1540 to repeat. However, the DRAM-based DIMM uses a deterministic protocol wherein the RCD 1220 the DBs 1230 instructs to release their data to the host after a predetermined amount of time. As mentioned above, due to the mechanism of read data from a non-volatile memory, the requested data may not be ready to be sent to the host in the predetermined amount of time. Examples of this mechanism include, but are not limited to, media selection (e.g., MRAM, PRAM, RRAM, etc.) and media material, process nodes, I / O circuit behavior, I / O Circuit Log, Intermittent Logic Dies, Controller Delays, Data Errors (BER, Defects) which require a higher or lower ECC, which means a higher or lower number of NVM Dies, Placements of NVM Devices and Controllers, NVM Communication Channel Delays (e.g. B. Command vs. Data Groups of Commands, Shared Data and Command, Serializer / Deserializer (SerDes) vs. Parallel), and NVM Channel Connection Options (eg, Through Silicon Via (TSV), Through Silicon Side Wall (TSW), directly, interposed).

Accordingly, the RCD 1520 in the in 15 shown embodiment (eg by programming a processor in the RCD 1520 with firmware / software or by providing a pure hardware implementation) to receive and respond to the new read command discussed above. In particular, the RCD 1520 in this embodiment, to provide a ready signal on the CMD / Addr line when the DBs 1550 The data contained in response to a read command, and is further designed to the DBs 1550 to instruct their data (after a predefined delay) to the host in response to release the RCD 1520 receives a send command.

16 Fig. 16 is a block diagram illustrating a read operation. As in 16 is shown, a read command from the RCD 1520 received from the memory controller in the host (arrow 1). The address and the read command are then sent from the RCD 1520 to the NVM controller 1530 transferred (arrow 2). The read command is processed and sent to the relevant NVM devices 1540 Transmit (arrow 3), and the read data returns to the NVM controller and then on to the DBs 1550 back (arrow 4). If the RCD 1520 know that the DBs 1550 contain the data (eg by polling or otherwise communicating with the DBs 1550 or as directed by the NVM controller 1530 ), the RCD sends 1520 the signal RD_RDY to the memory controller in the host (arrow 5). In response, the memory controller in the host issues a SEND command on the command bus (arrow 6), and in response the RCD points 1520 the DBs 1550 to transfer the data (after an optional specified delay (tsend)) to the host (arrow 7).

Turning now to the write operation (see 17 ), the memory controller in the host first checks the write count to ensure there is a remaining credit for the write operation. If so, the memory controller in the host transmits a write command and an address to the RCD 1520 (Arrow 2), and the memory controller decrements its write credit. Next, the memory controller in the host transfers data to the DBs after a specified JEDEC delay 1550 (Arrow 3). Then the command and the data from the RCD 1520 and the DBs 1550 to the NVM controller 1530 transferred (arrow 4), although the RCD 1520 the address and the command can forward before the data from the DBs 1550 received. Next, the write data becomes the NVM devices 1540 committed (arrow 5), and the write credit is returned to the memory controller in the host on the bus (arrow 6). It should be noted that actions 5 and 6 can be reversed. However, if persistence is required before a write-credit confirmation, then it may be preferable to perform action 5 before 6. If persistence is not required before a write-credit confirmation, then it may be preferable to perform action 6 before 5. In either case, the memory controller in the host increments the write-credit count (the write-credit response back to the host 100 Either single credits or multiple credits per message to the host 100 be).

Due to the mechanism of reading and writing in NVM memory devices, read and write commands may not be completed in the order in which they were received. As discussed above, a second received read command (read B) may be completed prior to a first received read command (read A), for example, if read B has a higher priority or if the physical address of read A is not available for reads and read A for a later date is planned. This is not a problem for DRAM-based DIMMs because read and write commands are processed in the order in which they are received. However, this can be a problem with NV-DIMMs because the data shared by the NV-DIMM to the host may not be the data that the host expects (eg, the host expects data from read A, but instead receives data) from reading B). To address this problem, an identifier (ID) is associated with various commands to keep track of which data belongs to which commands. This will be in 18 and 19 illustrated.

18A FIG. 10 is a flowchart of a read operation of an embodiment using the memory system. FIG 1500 in 15 , As in 18A As shown, the host commands a read from an address (and specifies an optional read ID) 1880 ). The RCD then forwards the command, address, and ID (act 1882 ). It should be noted that this ID (which may be used to enable out-of-order operations) may or may not be the same as the ID received from the host. Next, the data from NVM is ready (operation 1884 ), and the RCD shares tells the host that the read data is ready (and optionally includes the read read ID) 1886 ). The host then outputs the transmission signal (act 1888 ), and the RCD tells the NVM controller to transmit (operation 1890 ). The data ( 1892 ) are then transferred (act 1894 ), along with a response that includes the ID (act 1896 ).

18B Fig. 10 is a flowchart of a read operation of another embodiment. As in 18B shown, the host orders 100 a read from an address and includes an optional read identifier (ID) 1805 ). The RCD 1520 receives the command, address and ID to the NVM controller 1520 (Process 1810 ). The RCD 1520 also directs the command and the ID (but not the address) to the DBs 1550 continue (process 1815 ). In response, the DBs arrange 1550 Allocate space for the read data and reference the assigned seat with the ID (operation 1820 ). (In another embodiment, the DBs always have some space available, and the ID is correlated in a delayed manner with the ID contained within the RCD.) After the NVM controller 1530 has read the requested data from the NVM devices (act 1825 ), sends the NVM controller 1520 the data and the ID to the DBs 1550 whereby the data is placed in the allocated space identified by the ID (act 1835 ). The NVM controller 1520 Also sends a termination signal and the ID to the RCD 1520 (Process 1840 ), which can either wait until the DBs 1550 confirm that the data is placed or can wait a predefined time (Action 1845 ). After either the DBs 1550 confirming the storage of the data or after the predefined time has elapsed, the RCD communicates 1520 the host 100 that the read is ready (and may include the ID as well) (Action 1850 ). The host 100 later sends a send command (with the ID) to request the read data (Operation 1855 ). The RCD then tells the NVM controller to transmit (operation 1859 ). In response, the NVM controller shares the DBs 1550 with transmitting the data associated with the ID after an optional predetermined delay specified by a standard (act 1860). The DBs 1550 then transmit the data associated with the ID (act 1865 ), and the RCD transmits its corresponding information (operation 1870 ).

Turning now 19A to, is 19A a flowchart of a write operation of an embodiment. As in 19A shown, the host determines 100 first, if he can send a write command by checking whether credits have remained in the write counter and / or checking if the persistence level is greater than 0 (act 1904 ). It should be noted that the write counter and the persistence counter are optional and that an implementation may have one, both or none of the counters. This particular example uses both write and persistence counters, and if the write is allowed, the host decreases 100 the count in both counters (operation 1908 ). If the RCD 1520 the write command from the host 100 receives, sends the command and the address to the NVM controller 1530 (Process 1912 ) and sends the data to be written to the DBs 1550 (Process 1922 ). The RCD 1520 may also include the optional ID in embodiments where the NVM controller 1530 the data from the DBs 1550 brings (process 1925 ). The data is then repeated (operation 1926 ). The NVM controller 1530 then accepts the data from the DBs 1550 into its write buffer (operation 1932 ). The NVM controller 1530 then moves the data through its buffers and may eventually be in an optional state that it is protected in the event of a power failure and is sure to be writing (act 1934 ). The NVM controller 1530 then writes the data to the NVM devices 1540 (Process 1936 ).

In this embodiment, there are three locations where the storage system 100 back to the host 100 can communicate that the write is completed. The protocol may or may not differentiate between them, and it may or may not track them separately. Also, it may happen that customers or manufacturers will implement different behaviors. As in 19 In one embodiment, the write persist indicator and counter are incremented (operations 1944 and 1948 ). In another embodiment, the write persistence indicator and counter are incremented (operations 1952 and 1956 ). In yet another embodiment, the write completion indicator and counter are incremented (operations 1964 and 1968 ).

19B FIG. 10 is a flowchart of a write operation of another embodiment. FIG. As in 19B shown, the host determines 100 first, if he can send a write command by checking whether credits have remained in the write counter and / or checking if the persistence level is greater than 0 (act 1905 ). It should be noted that the write counter and the persistence counter are optional and that an implementation may have one, both or none of the counters. This particular example uses both write and persistence counters, and if the write is allowed, the host decreases 100 the count in both counters (operation 1910 ). If the RCD 1520 the write command from the host 100 receives, sends the command and the address to the NVM controller 1530 (Process 1915 ) and sends the to be written Data to the DBs 1550 (Process 1920 ). The RCD 1520 may also include the write ID in embodiments where the NVM controller 1530 the data from the DBs 1550 brings (process 1925 ). If the NVM controller 1530 the data is not from the DBs 1550 gets, push the DBs 1550 the write data to the NVM controller 1520 as by the RCD 1520 coordinates to request data for the ID (operation 1930 ). The data then becomes the NVM controller 1530 moved (operation 1932 ). The NVM controller 1530 then accepts the data from the DBs 1550 into its write buffer (operation 1935 ). The NVM controller 1530 then moves the data through its buffers and may eventually be in an optional state that it is protected in the event of a power failure and is sure to be writing (act 1940 ). The NVM controller 1530 then writes the data to the NVM devices 1540 (Process 1945 ).

In this embodiment, there are three locations where the storage system 100 back to the host 100 can communicate that the write is completed. The protocol may or may not differentiate between them, and it may or may not track them separately. Also, it may happen that customers or manufacturers will implement different behaviors. As in 19 In one embodiment, the write persist indicator and counter are incremented (operations 1955 and 1960 ). In another embodiment, the write persistence indicator and counter are incremented (operations 1970 and 1975 ). In yet another embodiment, the write completion indicator and counter are incremented (operations 1985 and 1990 ).

Another problem that may be due to the use of an NVM controller 1520 must be addressed, the clock rate, as the NVM controller 1520 may need a slower clock than the one provided by the host 100 is generated on the SDRAM bus. High speed bus lines of conventional DIMMs can handle complex circuits in the input / output connections on the NVM controller 1520 as well as careful routing in the storage system 1500 require. To address this, the RCD 1520 in one embodiment, changing the clock speed to data in the internal lines in the memory system 100 to transmit at a slower rate. (Alternatively, the RCD 1520 This functionality can be performed by the NVM controller 1520 or another component in the storage system 100 change the clock speed.) This becomes diagrammatic in 20 shown for incoming data (the same conversion can be reversed to send data back to the host 100 respectively). 20 shows the clock, DQ and DQ strobe signal from the host side 100 (left part of 20 ) and from the side of the NVM controller 1530 (right part of 20 ). As shown in this drawing, the clock signal is from the host 100 on a frequency Thost, which due to the DDR protocol causes data and data strobes to occur at a relatively high frequency, which may be too much to get from the NVM controller 1530 without significant changes to its circuits. In contrast, data and data strobes, as shown by the right part of 20 shown by slowing down the clock to Tnvsdimm be slowed to a relatively low frequency, indicating the NVM controller 1530 is easier.

The RCD 1520 may be configured to slow down the clock using any suitable method. For example, the RCD 1520 Include clock dividers to generate slower clocks from the source clock (eg, by dividing the frequency by an integer to produce a slower frequency). The RCD 1520 may also include a phase-locked loop (PLL) to increase the clock frequency, which may be important for the division of the clock frequency by a non-integral fraction. For example, to divide the clock frequency by 3/2 (or in other words multiply by 2/3), a PLL can be used to first double the clock frequency before dividing by three. As another example, the RCD 1520 Delay compensation circuits (e.g., a phase locked loop may contain the delay to compensate in its feedback loop therefor, and thus the delay would be automatically subtracted from the clock output, or explicit delay locked loops may be added to explicitly adjust the delays). As yet another example, the RCD 1520 Have data synchronizer, which slow down the data, not just the clock. This can be done using a first-in-first-out memory, which has the advantage of moving the data securely from one clock domain to another.

As mentioned above, rather than these clock change components in the RCD 1520 to implement them in the NVM controller 1520 be implemented. In addition, the RCD 1502 The clock and data reclocking functions include loosening the signal integrity and routing requirements of the DIMM's internal wiring. Furthermore, three clocks can be used (one to talk to the host (very fast), one to send data to the media controller (less fast) and one to talk to the NVM (even less fast)), in which case both the NVM controller 1520 as well as the RCD 1520 could perform a clock conversion.

In embodiments where the data clock rate decreases as it passes through the RCD, the clock is preferably distributed to all DBs. Thus, the DBs can receive a copy of the host clock and the media controller page clock. Also, the RCD preferably knows how slow the media controller page clock is, so it can still fulfill its task of synchronizing the DB data transfers.

In addition, bandwidth considerations may be present in addition to clock conversion. For example, the bandwidth in the left part of 20 defined as: N bits * (1 ns) / (thost) * 1 GHz or N / (thost / 1 ns) [Gbits / s]. In the right part of 20 the bandwidth would be defined as: N / (nvdimm / 1 ns) [Gbits / s]. There are several approaches that can be used to account for the bandwidth difference. For example, one approach uses serializers and deserializers to achieve the same bandwidth as a DDR over the DIMM. The deserializer can take a narrow bus of N bits with a frequency of f cycles / second and a transfer rate of f * N bits / second and make it a wider bus of N * a bits with a frequency of f / b cycles / second and For a = b, the bandwidth is the same for the wider, slower bus). The use of the serializer can transform the width back to N bits at a frequency of f cycles / second.

In another approach, queues may be used to compensate for the bandwidth mismatch. The bus width is the same for DB input and output. In this approach, incoming data (from the host 100 to the NVM controller 1330 ) is held in a buffer which may or may not be a first-in-first-out (FIFO) memory. Using a buffer can cause transmission to the NVM controller 1520 takes longer, but the buffer provides a temporary hold during the transfer. Outgoing data (from the NVM controller 1530 to the DBs) can be collected in a buffer (such as, but not limited to, a FIFO) when they are dropping with a low bandwidth. The data can only be retransmitted to the host when a complete packet is received.

Changes to the DBs 1550 can also be made to use non-volatile memory and the NVM controller 1530 to take into account. To understand these changes, an in 21 shown DB 2100 considered. This DB 2100 includes a set of components for the DQ signals and for the DQ strobe signals. As in 21 As shown, the components for the DQ signals include I / O buffers 2110 . 2120 , Input and output FIFOs 2130 . 2140 and synchronization / phasing logic 2115 , The components for the DQ strobe signals include I / O buffers 2150 . 2160 and strobe generators 2170 . 2180 , The DB 2100 also contains a command parsing logic 2190 which has the clock and command bus signals as input. In this embodiment, the FIFOs 2130 . 2140 used for caching data and are synchronized by the RCD and DQ strobe generators. In another implementation, the FIFOs are not used, and the DB 2100 is configured in pass-through mode.

If a DB is designed to down-convert data to a lower frequency, additional components may be used, as in FIG 22 shown. Like the DB 2100 in 21 include the components for the DQ strobe signals I / O buffer 2250 . 2260 and strobe generators 2270 . 2280 , and the components for the DQ signals include I / O buffers 2210 . 2220 and synchronization / phasing logic 2215 , However, the DB includes 22 in 22 instead of input and output FIFOs, I / O buffers 2230 . 2240 , and the command parsing logic 2290 contains the following inputs: clock A (host side), clock B (NV DIMM page), and command bus signals from the RCD. In addition, the DB contains 2200 Dual port, dual clock random access memory 2235 to allow out-of-order processing because the input and output buffers 2230 . 2240 serve both as a data store and as a staging area for synchronization (a second FIFO can be used for further synchronization).

Returning to the drawings. 23 is an illustration of an alternative architecture to that in 15 shown.

As in 23A shown are the NVM devices 2540 with the DBs 2350 connected without the NVM controller 2330 to go. This embodiment may be useful when NVM devices that operate at DRAM speed are capable of data rates with the DBs 2350 and the bus 2310 to bring into line. Writes and reads that conflict with media locations that cause unforeseen latencies can be handled by the DBs 2350 be absorbed without the bus 2310 to impair. The NVM controller 2330 can the DBs 2350 , the RCD 2320 and coordinate the NVM activity while allowing data directly between the DBs 2350 and the NMV devices 2340 to get redirected.

In addition, as noted above, the memory system may include one RCD and DBs in various variations of DIMMs (eg, UDIMM, RDIMM and LRDIMM). There are variations in each of these DIMM formats. For example, UDIMMs have short lines with respect to electrical routing rules. UDIMMs generally have a small number of DIMMs, DRAM banks / ranks per packet, and next physical layout in the server motherboard. The DRAM packets and instruction routing lines are all specified for repeatable system integration and system electrical interactions. As a result, UDIMMs have the cheapest production costs. RDIMMs have an RCD and generally have a larger number of DIMMs. DRAM banks / ranks per package are possible. DRAM packets, terminations, data routing, and RCD details are specified. RCD-to-DRAM connections are loose specifications. Compared to UDIMM, there are incremental costs for RCD. LRDIMMs have insulators across all electrical communication groups, and DB and RCD connections to the memory controller are tightly specified. LRDIMMs have the highest cost among these three formats, but most DIMMs, BGAs, and banks / ranks per storage controller are allowed.

For each DRAM bus (UDIMM, RDIMM, LRDIMM), the storage system may use specifications for the external interacting components. These specifications may include physical and electrical properties for maximum interoperability. This may include changes in both the physical signaling layer (eg, to adapt electrical specifications) and the instruction layer (eg, to provide the appropriate instruction decode). Changes to the physical signaling layer may include the insertion of additional transmission lines in the headset, or changes in the geometry, impedance, and / or termination of any of the clock, command, data, or header lines (including both standard SDRAM / DDR headset lines) Response bus). In the instruction layer, these changes may also include selection between different Tsends, depending on the delay to which these different formats are exposed, or adding new interpretation to new instructions (eg, associating particular row decode bits not with addresses within a ranks, but instead inferred selection of additional ranks within a DIMM).

Also, parameterized specifications for the internal connections from an NVM controller to the RCD and DBs can be set. The internal connections can be optional to enable vendor-specific optimizations, package integrations, or ASIC integration. The specifications can be robust enough to handle diverse NVM controller placements, diverse data communication rates, and signal integrity characteristics. The specifications for RAM buffer sizing and RCD timing behavior can also be used for successful supplier diagnostic interoperability.

Returning to the drawings 23B an illustration of a RCD 2360 an embodiment. As in 23B shown includes the RCD 2360 input buffer in this embodiment 2363 , Latches / FFs 2363 , Control register 2364 , Output buffer 2365 , CS, CKE, decoding logic 2366 , Control logic 2367 , Clock buffer 2368 , a PLL 2369 and a PLL feedback delay compensation module 2370 , Many of the circuit elements in this RCD 2360 may be similar to those found in the RCD discussed above. However, the configuration of the control logic 2367 to accommodate the nature of the non-deterministically timed SNVRAM command sequences to support SNVRAMs. The control logic 2367 is responsible for the behavioral response of the RCD, and changes can be made so that the DRAM DIMM RCDs are as shown in the flowcharts in 18 and 19 orchestrate the command flows shown. The RCD also has the differentiating ability to understand more commands, controls and addresses. There may be additional outputs and inputs to synchronize new parts such as the NVM controller.

Finally, as mentioned above, any suitable type of memory may be used. Semiconductor memory devices include volatile memory devices such as dynamic random access memory ("DRAM") or static random access memory ("SRAM") devices, non-volatile memory devices such as resistive random access memory ("Resistive Random Access Memory") , "ReRAM"), electrically erasable programmable read only memory ("EEPROM"), flash memory (which may also be considered as a subset of EEPROM), ferroelectric random access memory , "FRAM") and Magnetoresistive Random Access Memory ("MRAM") and other semiconductor elements that can store information. Each type of storage device may have different configurations. For example, flash memory devices may be configured in a NAND or a NOR configuration.

The memory devices may be formed of passive and / or active elements in any combination. As a non-limiting example, passive semiconductor memory devices include ReRAM device elements, which in some embodiments comprise a resistance switching element, such as an anti-fuse phase change material, etc., and optionally a steering element, such as a diode, etc. Further, as a non-limiting example, active semiconductor memory elements include EEPROM and flash memory device elements, which in some embodiments include elements that include a charge storage region, such as a floating gate, conductive nanoparticles, or a charge storage dielectric material.

Multiple memory elements may be designed to be connected in series or so that each element may be individually accessed. As a non-limiting example, flash memory devices in a NAND (NAND) configuration typically include series connected memory elements. A NAND memory array may be arranged such that the array consists of multiple strings of memory, where a string consists of several memory elements which share a single bit line and are accessed as a group. Alternatively, memory elements may be designed so that each element can be accessed individually, e.g. B. a NOR memory array. NAND and NOR memory configurations are exemplary, and memory elements may be otherwise designed.

The semiconductor memory elements disposed within and / or over a substrate may be arranged in two or three dimensions, such as in a two-dimensional memory structure or in a three-dimensional memory structure.

In a two-dimensional memory structure, the semiconductor memory elements are arranged in a single level or a single memory device level. Typically, memory elements in a two-dimensional memory structure are disposed in a plane (eg, in a plane in the x-z direction) that extends substantially parallel to a major surface of a substrate that supports the memory elements. The substrate may be a wafer over or in which the layer of memory elements is formed, or it may be a carrier substrate which is attached to the memory elements after being formed. As a non-limiting example, the substrate may include a semiconductor such as silicon.

The memory elements may be arranged in a single array device level in an ordered array, such as a plurality of rows and / or columns. However, the memory elements may be arranged in non-regular or non-orthogonal configurations. The memory elements may each comprise two or more electrodes or contact lines, such as bit lines and word lines.

A three-dimensional memory array is arranged such that memory elements occupy multiple levels or multiple memory device levels, thereby forming a structure in three dimensions (ie, in the x, y, and z directions, with the y direction substantially perpendicular and the x and z directions are substantially parallel to the main surface of the substrate).

As a non-limiting example, a three-dimensional memory structure may be vertically arranged as a stack of a plurality of two-dimensional memory device levels. As a further non-limiting example, a three-dimensional memory array may be arranged as a plurality of vertical columns (eg, columns extending substantially perpendicular to the main surface of the substrate, ie, in the y-direction), each column having multiple memory elements in it each column. The columns may be in a two-dimensional configuration, e.g. In an x-z plane, resulting in a three-dimensional array of storage elements with elements on multiple vertically stacked storage levels. Other configurations of storage elements in three dimensions may also represent a three-dimensional storage array.

As a non-limiting example, the memory elements in a three-dimensional NAND memory array may be coupled together to form a NAND string within a single horizontal (eg, x-z) memory device level. Alternatively, the memory elements may be coupled together to form a vertical NAND string that traverses multiple horizontal memory device levels. Other three-dimensional configurations are conceivable in which some NAND strings contain storage elements at a single storage level, while other strings contain storage elements that extend through multiple storage levels. Three-dimensional memory arrays can also be designed in a NOR configuration and in a ReRAM configuration.

Typically, in a monolithic three-dimensional memory array, one or more memory device levels are formed over a single substrate. Optionally, the monolithic three-dimensional memory array may also include one or more memory layers at least partially within the single substrate. As a non-limiting example, the substrate may include a semiconductor such as silicon. In a monolithic three-dimensional array, the layers that make up each array device level of the array are typically formed on the layers of the underlying memory device level of the array. However, layers of adjacent memory device levels of a monolithic three-dimensional memory array may be shared or have intervening layers between memory device levels.

Then, again, two-dimensional arrays may be separately formed and then packaged together in a package to form a non-monolithic memory device having a plurality of memory layers. For example, non-monolithic stacked memories can be constructed by forming memory levels on separate substrates and then stacking the memory levels together. The substrates may be thinned or removed from the memory device levels prior to stacking, but since the memory device levels are initially formed over separate substrates, the resulting memory arrays are not monolithic three-dimensional memory arrays. Further, a plurality of two-dimensional memory arrays or three-dimensional memory arrays (monolithic or non-monolithic) may be formed on separate chips and then packaged together in a package to form a stacked-chip memory device.

Associated circuits are typically required for the operation of the memory elements and for communication with the memory elements. As non-limiting examples, memory devices may include circuits used to control and drive memory elements to perform functions such as programming and reading. These associated circuits may be on the same substrate as the memory elements and / or on a separate substrate. For example, a memory read / write controller may be located on a separate controller chip and / or on the same substrate as the memory elements.

Those skilled in the art will recognize that this invention is not limited to the described two-dimensional and three-dimensional exemplary structures, but covers all relevant memory structures within the spirit and scope of the invention as described herein and as understood by those skilled in the art.

It is intended that the foregoing detailed description be understood to be illustrative of selected forms that the invention may take and not as a definition of the invention. It is only the following claims, including all equivalents, that are intended to define the scope of the claimed invention. Finally, it should be noted that any aspect of any of the preferred embodiments described herein may be used alone or in combination with one another.

Claims (26)

 Storage system comprising: a plurality of non-volatile memory devices; a controller in communication with the plurality of non-volatile memory devices, the controller being configured to: Receiving a read command from a host; in response to receiving the read command from the host, reading data from the plurality of non-volatile storage devices; Performing an operation of indeterminate duration from the perspective of the host; Sending a ready signal to the host after performing the operation; Receiving a send command from the host; and in response to receiving the send command from the host, sending the data to the host; a plurality of data buffers in communication with the controller and configured to store data sent between the controller and the host; and a command and address buffer configured to store commands and addresses sent by the host, the command and address buffer further configured to synchronize data flow into and out of the plurality of data buffers.  The memory system of claim 1, wherein read and / or write commands are associated with identifiers so that the read and / or write commands can be processed in a different order than in an order received from the host.  The memory system of any one of the preceding claims, wherein the command and address buffer comprises a registered clock driver. The memory system of any one of the preceding claims, wherein the plurality of data buffers comprises random access memory.  The memory system of any one of the preceding claims, wherein the command and address buffer is further adapted to change a frequency of a clock received from the host.  The memory system of any preceding claim, wherein the command and address buffer is further configured to perform bandwidth conversion.  The memory system of any one of the preceding claims, wherein physical layers and instruction layers of the memory system are adapted to be compatible with a DRAM DIMM communication protocol.  The memory system of claim 7, wherein physical layers and instruction layers of the memory system are configured to be compatible with one or more of the following: unbuffered DIMM (UDIMM), registered DIMM (RDIMM), and load-reduced DIMM (LRDIMM).  The memory system of any one of the preceding claims, wherein the data is sent to the host after a time delay, and wherein the time delay is selected based on a communication protocol used with the host.  The memory system of any one of the preceding claims, wherein the controller is configured to communicate with the host using a clock data parallel interface.  The memory system of claim 10, wherein the clock data parallel interface comprises a double data rate (DDR) interface.  A memory system according to any one of the preceding claims, wherein at least one of the plurality of non-volatile memory devices comprises a three-dimensional memory.  Storage system comprising: a plurality of non-volatile memory devices; a controller in communication with the plurality of non-volatile memory devices, the controller being configured to: Receiving a write command from a host, allowing only a certain number of pending write commands to the host, as tracked by a write counter in the host; Performing an operation of indeterminate duration from the perspective of the host; Writing data to the plurality of non-volatile storage devices; and after writing the data, sending a write counter increment signal to the host; a plurality of data buffers in communication with the controller and configured to store data sent between the controller and the host; and a command and address buffer configured to store commands and addresses sent by the host, the command and address buffer further configured to synchronize data flow into and out of the plurality of data buffers.  The memory system of claim 13, wherein read and / or write commands are associated with identifiers such that the read and / or write commands can be processed in a different order than in an order received from the host.  The memory system of claim 13 or 14, wherein the command and address buffer comprises a registered clock driver.  The memory system of any of the preceding claims claims 13 to 15, wherein the plurality of data buffers comprises random access memory.  The memory system of any one of the preceding claims, claims 13 to 16, wherein the command and address buffer is further adapted to change a frequency of a clock received from the host.  The memory system of any of the preceding claims claims 13 to 17, wherein the command and address buffer is further adapted to perform bandwidth conversion.  The memory system of any one of the preceding claims, claims 13 to 18, wherein physical layers and instruction layers of the memory system are adapted to be compatible with a DRAM DIMM communication protocol.  The memory system of claim 19, wherein physical layers and instruction layers of the memory system are configured to be compatible with one or more of the following: unbuffered DIMM (UDIMM), registered DIMM (RDIMM), and load-reduced DIMM (LRDIMM).  The memory system of any one of the preceding claims, claims 13 to 20, wherein the data is sent to the host after a time delay, and wherein the time delay is selected based on a communication protocol used with the host. A memory system according to any one of the preceding claims, claims 13 to 21, wherein the controller is designed to communicate with the host using a clock data parallel interface.  The memory system of claim 22, wherein the clock data parallel interface comprises a double data rate (DDR) interface.  The storage system of any one of the preceding claims claims 13 to 23, wherein at least one of the plurality of non-volatile storage devices comprises a three-dimensional storage.  Storage system comprising: a plurality of non-volatile memory devices; Means for receiving a read command from a host; Means for reading data from the plurality of non-volatile memory devices in response to receiving the read command from the host; Means for performing an operation of indeterminate duration from the perspective of the host; Means for sending a ready signal to the host after performing the operation; Means for receiving a send command from the host; Means for sending the data to the host in response to receiving the send command from the host; Means for storing data sent between the controller and the host; and Means for storing commands and addresses sent by the host, the command and address buffer further being arranged to synchronize the flow of data into and out of the plurality of data buffers.  The storage system of claim 25, further comprising: Means for receiving a write command from the host, allowing only a certain number of pending write commands to the host, as tracked by a write counter in the host; Means for performing an operation of indeterminate duration from the perspective of the host; Means for writing data to the plurality of non-volatile storage devices; and Means for sending a write counter increment signal to the host after writing the data.

Images