RU2780441C2 - Permanent records for non-volatile memory - Google Patents

Abstract

FIELD: computer technology.

SUBSTANCE: method for performing permanent operations contains: reception by means of a non-volatile dual in-line memory module (hereinafter – NVDIMM) configured to support permanent records of a permanent record command and associated record data from host; performance of a permanent record of record data to non-volatile memory in NVDIMM based on the permanent record command; reception of record identification (hereinafter – WID) associated with the permanent record command from host after successful completion of the permanent record; reception from host of a request for sending a status for a set of permanent records together with associated WID; and reception by means of NVDIMM of a group of two or more permanent record commands with common WID, wherein the last permanent record command of the group has Persist bit set in 1, and the other permanent record commands have corresponding Persist bits set in 0, and provision of an indication on completion of the permanent records for the last permanent record command.

EFFECT: increase in the performance and efficiency of permanent record operations.

30 cl, 8 dwg

Description

disclosure area

[0001] The disclosed aspects are directed to memory systems. More specifically, exemplary aspects are directed to persistent write operations and their protocols for non-volatile memory.

Prior Art

[0002] Storage class memory (SCM) generally refers to a large capacity memory that can also have high performance. The SCM may be used in applications such as servers or other processing systems, wherein the operating data set for a processor or central processing unit may be stored in the SCM while the complete data set may be stored in a back-up memory or hard disk drive (HDD). An important expectation of the SCM is write persistence, which means that information written to the SCM should not be lost if, say, the server crashes or loses power. Traditional non-volatile memory that can meet such persistence expectations may not, however, be able to meet the capacity and performance metrics that SCMs may desire. Therefore, technologies such as phase change memory (PCM), spin moment transfer magnetic random access memory (STT MRAM), resistive RAM (ReRAM), etc., are becoming more popular in SCM implementations.

[0003] When using the SCM, the application can use memory write operations to update the corresponding persistent memory. For a write to the SCM to be persistent, an application requesting a write operation can wait for explicit confirmation that the write operation has reached persistent memory. In contrast, write operations to non-persistent memory (such as dynamic random access memory (DRAM) or other volatile memory) are traditionally considered complete or done, from the point of view of the application, when the write operation and associated data have been transferred to memory, and no explicit confirmation is required. that the data has been written. Thus, for applications that use SCM with persistence expectation, high performance methods are desirable that provide explicit acknowledgment of writes to persistent memory, and high performance methods are also compatible with different data sizes to maximize efficiency.

[0004] There are two types of conventional schemes for persistent memory operations. The first scheme assumes that the entire memory system (eg, a dual-sided memory module (DIMM) containing a string of DRAM integrated circuits, as is known in the art) is energy supported. In this case, a write operation to an intermediate buffer at the receiving end of the DIMM may be sufficient to satisfy persistence expectations. In one implementation, when a write operation on the channel interface between the application requesting the write operation and the DIMM is successfully completed, the write operation may be considered to be permanent. However, the implementation of such circuits may involve the use of energy storage devices such as supercapacitors or batteries that provide power/charge to reset intermediate buffers on the DIMM when a power failure is detected. But such power storage devices may not be available on all DIMMs, and furthermore, even if available, they are associated with high costs.

[0005] In the second scheme, all previous write operations may be flushed to read-only memory while the application is waiting for the completion status from the DIMM. However, this scheme can incur significant implementation costs. For example, in cases where an application may require constant fine-grained writes to a DIMM, but there may be other concurrent but independent writes streaming to the DIMM, flushing all previous read-only writes while waiting for the completion status may only slow down persistent write requests, but also concurrent write operations.

[0006] Accordingly, there is a need in the art for high performance, high efficiency persistent write operations that support different granularities or sizes of persistent records, avoiding the aforementioned disadvantages of conventional approaches.

Brief description of the essence of the invention

[0007] Exemplary aspects of the invention include systems and methods for ongoing operations. The memory system, after receiving the persistent write command and the associated write data from the host, permanently writes the write data to the non-volatile memory in the memory system based on the persistent write command. The memory system may also receive a write identification (WID) associated with the permanent write command from the host and provide, upon successful completion of the permanent write, an indication of completion of the permanent write along with the associated WID to the host.

[0008] For example, an exemplary aspect is directed to a method for performing persistent operations, the method comprising receiving, at the memory system, a persistent write command and associated write data from a host, and performing a persistent write of the write data to non-volatile memory in the memory system based on the persistent write command.

[0009] Another exemplary aspect is directed to a method for performing persistent operations, the method comprising providing, from a host to a memory system, a persistent write command and associated write data, wherein the persistent write command instructs the memory system to permanently write the write data to non-volatile memory.

[0010] Another exemplary aspect is directed to an apparatus comprising a memory system configured to receive a persistent write command and associated write data from a host, and permanently write the write data to non-volatile memory in the memory system based on the persistent write command.

[0011] Another exemplary aspect is directed to an apparatus comprising a host configured to provide a persistent write command and associated write data to a memory system, wherein the persistent write command instructs the memory system to permanently write the write data to non-volatile memory.

[0012] Another exemplary aspect is directed to an apparatus comprising means for storing data, comprising means for receiving a persistent write command and associated write data from a host, and means for performing persistent writing of the write data to non-volatile memory in the storage means, based on the persistent write command. .

[0013] Another exemplary aspect is directed to an apparatus comprising means for processing comprising means for providing a persistent write command and associated write data to a memory system, wherein the persistent write command instructs the memory system to permanently write the write data to non-volatile memory.

[0014] Another exemplary aspect is directed to a non-volatile computer-readable storage medium comprising code that, when executed by a processor, causes the processor to perform persistent operations, the non-volatile computer-readable storage medium comprises code for receiving, in a memory system, a permanent write command and associated write data from the host, and code for permanently writing the recording data to the non-volatile memory in the memory system based on the persistent write command.

[0015] Another exemplary aspect is directed to a non-volatile computer-readable storage medium comprising code that, when executed by a processor, causes the processor to perform persistent operations, the non-volatile computer-readable storage medium comprises code to provide, from the host to the memory system, a persistent write instruction and associated the write data, wherein the permanent write command instructs the memory system to permanently write the write data to the non-volatile memory.

Brief description of the drawings

[0016] The accompanying drawings are presented to assist in describing aspects of the invention and are provided to illustrate the aspects only and not to limit them.

[0017] FIG. 1 illustrates a processing system in accordance with aspects of the present disclosure.

[0018] FIG. 2A-C illustrate transactions for processing persistent records, in accordance with various aspects of the present disclosure.

[0019] FIG. 3 illustrates an exemplary coding for a persistent write command in accordance with the present disclosure.

[0020] FIG. 4A-B illustrate sequences of events regarding exemplary methods for performing persistent entries, in accordance with aspects of the present disclosure.

[0021] FIG. 5 depicts an exemplary computing device in which an aspect of the disclosure may advantageously be used.

Detailed description

[0022] Aspects of the invention are disclosed in the following description and associated drawings, directed to specific aspects of the invention. Alternative aspects may be developed without departing from the scope of the invention. Additionally, well-known elements of the invention will not be described in detail or will be omitted so as not to obscure relevant details of the invention.

[0023] The word “exemplary” is used herein to mean “serving as an example, exemplary, or illustrative”. Any aspect described herein as "exemplary" is not necessarily to be understood as being preferred or advantageous over other aspects. Similarly, the term “aspects of the invention” does not require that all aspects of the invention include the feature, advantage, or mode of operation discussed.

[0024] The terminology used here is only intended to describe specific aspects and is not intended to limit aspects of the invention. As used here, the singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the terms "comprises", "comprising", "includes" and/or "comprising", when used here, define the presence of the indicated features, integers, steps, operations, elements and/or components. , but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groups thereof.

[0025] Additionally, many aspects are described in terms of sequences of actions to be performed, for example, by elements of a computing device. It will be understood that the various actions described herein may be performed by specific circuits (eg, application specific integrated circuits (ASICs)), software instructions executed by one or more processors, or a combination of both. Additionally, these sequences of actions described herein may be considered to be embodied entirely in any form of a computer-readable storage medium having stored therein an appropriate set of computer instructions that, when executed, would cause an associated processor to perform the functionality described herein. Thus, various aspects of the invention may be embodied in a variety of different forms, all of which are intended to be within the scope of the claimed subject matter. In addition, for each of the aspects described here, the corresponding form of any such aspects can be described, for example, as "logic configured to" perform the described action.

[0026] Exemplary aspects of the present disclosure are directed to efficient and high performance persistent write operations for non-volatile memory, such as a non-volatile DIMM (or NVDIMM). Accordingly, a persistent NVDIMM or NVDIMM-P is disclosed as one exemplary memory system that supports persistent write operations in accordance with the exemplary aspects. The host device may be configured to provide exemplary requests/commands for, for example, persistent operations and corresponding data to the exemplary storage system, and the storage system may be configured to perform the requested persistent write operations and provide appropriate signaling to the host device, as will discussed in more detail in the following sections.

[0027] With reference now to FIG. 1, an exemplary processing system 100 is shown, including a host 120 and a memory system 130. Host 120 may include one or more processing elements such as a general purpose processor (CPU), digital signal processor (DSP), media processor, system processor, graphics processing unit (GPU), modulator/demodulator (modem), application processor, etc. ., even though they have not been explicitly illustrated. These processing elements may make requests to access the memory system 130 . A memory controller (not shown) may reside in host 120 to manage these access requests.

[0028] The memory system 130 may be read-only memory, such as an NVDIMM-P in accordance with the present disclosure. The memory system 130 is shown as including an input/output (I/O) block 132 and a memory bank 134 . Memory bank 134 may include flash memory, DRAM, and so on.

[0029] Communication mechanism 110 is shown between host 120 and memory system 130, with a data (DQ) bus 112, a command and address (CA) bus 114, and a separately identified response bus 116. Host 120 may be capable of providing commands and associated addresses for requests memory access via CA 114 and send/receive data via DQ 112 (shown as a bi-directional bus). Response bus 116, although shown separately, may be configured as part of CA 114 and may be implemented as a bi-directional bus in some cases. Response bus 116 may be used to provide information, such as the status of persistent entries, in some exemplary aspects. Various other buses/wires may also be present in the communication mechanism 110, although they are not identified separately. In some examples, memory system 130 may use separate buses for deterministic and non-deterministic responses, as will be explained further below.

[0030] In an implementation in which memory system 130 may be configured as an NVDIMM, with additional support for persistent NVDIMM (NVDIMM-P) configuration for at least some operations, host 120 may be capable of providing one or more of the following exemplary commands to the system 130 memory, e.g. on CA 114:

- a READ command (for example, with an encoding length equal to multiples of 64 B), together with a read identification (RID);

- a WRITE command (for example, a traditional write command);

- a P-WRITE command (e.g. a permanent write command, together with a write identification (WID) for persistent writes, together with a persistent bit that indicates when records with a given WID need a permanent write complete (W_PER) signal (e.g., provided over the response bus 116) from the memory system 130);

- ADRx command: extended addressing;

- a SEND command (for example, a command for the memory system 130 to provide the status of a data read request);

- command SEND_Status (for example, a command for the system 130 memory to ensure the output of the error count, WID, etc. related to permanent operations, from the system 130 memory);

- FLUSH command (to reset previous records to be entered into permanent memory);

- NOP (empty statement);

- a read-ahead command (eg, used to read cached memory); and

- other caching commands that may be implementation specific.

[0031] As mentioned earlier, separate buses may be provided in communication engine 110 for deterministic and non-deterministic responses from memory system 130 to host 120. Deterministic responses include metadata, error/parity information such as error correction coding (ECC ) regarding read data sent over DQ 112 to host 120, etc., which may be multiplexed on buses originating from pins connected to I/O 132, such as check bit pins.

[0032] Among the ECC bits, there may be, for example, media ECC bits specific to memory system 130 implementations (such as NVDIMM), and channel-specific ECC bits in DQ 112, which can be standardized to allow interoperability across various implementations.

[0033] The metadata bits may include delayed RIDs for read requests sent out of program order (and, for in-order operations, the RID may be set to don't care). The metadata bits may also include write permission (WC), which refers to the unused quota for write operations allocated to certain hosts or processing elements of host 120. The metadata bits may further include data poisoning bits for data from user equipment, as is known in the art, and other user-defined bits.

[0034] Non-deterministic responses in accordance with the present disclosure may relate to persistent operations and may be sent via dedicated signaling, such as response 116 from memory system 130 to host 120, and may indicate the following:

- R_RDY: signal from memory system 130 to host 120 to indicate that read data is available;

- Wr_Per: signal from the memory system 130 to the host 120 to indicate that the persistent write has ended; and

- ERROR: Signal from memory system 130 to host 120 to indicate error conditions such as CRC checking, permission (credit) violation, media timeout, etc.

[0035] To implement the memory system 130 as NVDIMM-P, the following protocol pins can be defined in I/O 132, for example. Using pulse width modulation, the following pin and response signal configurations can be implemented. For example, in a Double Data Rate 5 (DDR5) NVDIMM-P implementation, one wire labeled RSP_n (one dedicated per sub-channel) can be used to provide the following signaling: 2 clock low for R_RDY, 4 clock low for R_RDY for W_PER and 6 low clocks for MESSAGE. Each low pulse may be followed by at least 2 high clock pulses. If a separate ERROR signal is needed, it can be defined as 8 low clock pulses.

[0036] For DDR4 implementation: two pins can be used to solve single pin performance issues (ODT1 and CKE1), where ODT1 represents 2 low clock width for R_RDY and 4 low clock for MESSAGE, and CKE1 represents 2 low clock width for W_PER. Each low pulse can be followed by at least 2 high clocks, and if a separate ERROR signal is needed, it can be defined as 6 low clocks on ODT1.

[0037] In exemplary implementations of persistent entries, suitable combinations of hardware, software, firmware, etc. (eg, applications, drivers, etc.) may be configured to allow notifications to be provided to host 120 from storage system 130 when one or more write requests from host 120 to storage system 130 reach persistence. These notifications may be implementation specific, as explained below.

[0038] When the data to be written for a write operation reaches the power-failure buffer on the media controller (eg, the fail-safe memory of the memory system 130), the write operation may be considered persistent during normal operations. However, for certain infrequent cases, or when the media controller's buffers are not protected from power failure, the software will ensure that the writes are entered end-to-end into the NVM media.

[0039] To implement the memory system 130 as an NVDIMM-P, power backed DIMMs use configurations in which the aforementioned buffers are protected from power failure, which means that the NVDIMM-P Write command can be used even when persistence is required for normal cases. Additionally, the NVDIMM-P Flush command, as defined herein, can be used to flush all writes in the media controller's buffers to non-volatile memory. In the case of a reset command, only entries that occurred prior to the reset are guaranteed to be made permanent for non-volatile memory. Software-implemented commands, such as appropriate barrier operations, can be used after the last write and before issuing a reset command to ensure that the correct order of the writes is maintained (e.g., when host 120 can be configured to send persistent write requests out of order programs).

[0040] Although non-power-backed DIMMs may be used less frequently than the power-backed DIMMs discussed above, the NVDIMM-P Persistent Write command may be used when persistence is also required for non-power-backed DIMMs. Host memory controller 120, as previously mentioned, may be configured to determine when to issue a permanent write command. In this case, memory system 130 is expected to provide explicit notification when persistent recording is completed, as will be discussed with reference to FIG. 2A. Additionally, the NVDIMM-P reset command can also be used, as before, to flush all writes (even non-persistent writes) to non-volatile memory.

[0041] With reference now to FIG. 2A, an exemplary set of transactions between host 120 and storage system 130 is shown to illustrate aspects of a persistent write command. There are some features of an exemplary write (Wr_Per) command (or simply, "write permanent") that may be common to the Read command from host 120 described above. These include a common write identification (WID), which may be multi-bit identifier to identify instructions specific to the entry. An exemplary set of WIDs 210 is shown, of which each can be up to 16 bits wide, which includes one valid bit (Valid) “Vld” (correspondingly, up to 31 WIDs can be represented in a 64-byte command packet sent to CA 114, for example). The permanent write command may also have a reserved field in the WID encoding for reset command status returns, which will be further explained in the following sections.

[0042] In one aspect, host 120 may be configured to issue a permanent entry only when host 120 has associated permanent entry permissions available. Persistent write permissions (similar to read permissions known in the art) may be determined at configuration time and administered by host 120 and may reflect the number of pending persistent writes that host 120 is allowed to issue.

[0043] Once issued, host 120 may be configured to keep track of pending persistent entries based on their respective WIDs 210. FIG. 2A (referring also to FIG. 1) shows two persistent writes (P-Write 1 with first address and WID, and P-Write 2 with second address and WID), labeled 202a and 204a, issued from host 120 to the system. 130 memory over CA 114, along with the respective data, write data 202b and write data 204b over DQ 112, for example.

[0044] The memory system 130 is configured to issue a “Wr_Per” response on the response bus 116 for a particular persistent entry after all data for that persistent entry has been written to non-volatile memory in the memory system 130. Wr_Per 202c and Wr_Per 204c are shown for respective persistent entries 202a and 204a. However, Wr_Per 202c and Wr_Per 204c are shown as being sent in an order that is different from the order in which persistent entries 202a and 204a were received by memory system 130 to illustrate that responses need not be in program order or in the order in which requests for persistent a write is received from host 120. In one aspect, memory system 130 may assert a “Req” signal on response bus 116 along with an appropriate encoding for the “Write_Rdy” message for Wr_Per responses.

[0045] Additionally, host 120 may also be configured to issue a “Send-Status for WID” command, denoted at 206a, at any time to determine the status of its pending persistent entries. In response, the memory system 130 may be configured to issue a status packet with the WID of completed persistent writes, such as when a packet of length 8 or "BL8" is carried over DQ 112.

[0046] As previously mentioned, up to 31 WIDs 210 may be packed into each 64 B status packet, with each WID 210 having a total of 16 bits assigned for a 15-bit WID and a Valid bit. Additionally, storage system 130 may also use the previously mentioned metadata field to return status for other entries. Host 120 may use the returned WIDs 210 in the WID status packet 206b, for example, to complete tracking of pending persistent entries.

[0047] In some aspects, two or more persistent entries may be grouped. For example, a set of 64B permanent entries may be grouped for commit (or write to non-volatile memory) in the case of DIMMs without power support, for example. An exemplary implementation may use a persistent record block to be issued to memory system 130 from host 120, wherein memory system 130 may be configured to aggregate the persistent record block in a buffer and commit the entire persistent record block at once, which may result in increased efficiency. It will be understood, however, that the grouping of persistent entries and their latching into a block is not required for power-backed DIMMs in which buffers are protected from power failure.

[0048] The following modifications may be made to the persistent write command to implement the group commits discussed above. Host 120 may select one WID (from WID 210, for example) for a set of two or more entries. An additional bit, called “Persist”, may be added to the persistent write command when sent on CA 114, for example. The Persist bit may be used to determine when the entire group of persistent records has been sent to the memory system 130 .

[0049] For example, three 64 B persistent entries may be grouped together as follows using WID=5 in an illustrative example. The first persistent entry (WID=5, Persist=0), the second persistent entry (WID=5, Persist=0), and the third persistent entry (WID=5, Persist=1) may be sent on the CA 114. The memory system 130 may be configured to collect persistent records with WID=5 in the buffer while the Persist bit is set to 0, and when the last persistent record arrives with the Persist bit set to 1, initiate persistence commit processes.

[0050] In one implementation, only the persistent entry with the Persist bit set to 1 can be configured to receive a Wr_Per response from the memory system 130 (eg, only the third persistent entry in the example above) for a group of persistent entries. This may reduce traffic on the 116 response bus.

[0051] In some aspects, persistent records with different WIDs may be interleaved, for example, across CA 114. Accordingly, grouping persistent records for a permanent commit does not imply that persistent records in a group of the same WID are sent sequentially from host 120.

[0052] In some aspects, to address the problem of race conditions that may occur in Wr_Per responses to persistent entries, the entry group ID (WGID) status method can be used to group the statuses of one or more persistent entries using different bitmaps, such as a completed WGID bitmap and a pending WGID bitmap, as will be explained with reference to FIG. 2B below. Given persistent entries with a corresponding WID, memory system 130 may assert a corresponding Wr_Per (referred to as “W_PER” in this case) for each persistent entry with Persist=1 and for each flush completion. Host 120 may use another Send-W_PER-Status command upon receipt of one or more W_PERs (wherein, host 120 may also maintain a W_PER count, referred to as W_PER-Count). The memory system 130 may only return a WGID-Completed Status with completed bits based on already asserted W_PERs. In turn, host 120 may update the list for the WGID or "WGID list" and decrement W_PER-Count based on the number of terminations.

[0053] In some cases, a fatal error (UE) may occur in transactions, which will be discussed with reference to FIG. 2C. When there is a UE in Send-W_PER-Status, host 120 may stop issuing new persistent writes/resets and Send-W_PER-Status. Host 120 may send a status read command, referred to as Xread-Status, to memory system 130. The memory system 130 responds by collecting all persistent records prior to receiving the Xread-Status to return a WGID-Pending Status to the host 120 (status packets cover the W_PER assertion prior to receiving the RD_RDY), and the memory system 130 may continue to issue W_PER during status reads. Host 120 may update the WGID list maintained by host 120 and decrement W_PER-Count based on pending entries. Host 120 may then start reissuing permanent writes/drops.

[0054] For DIMM implementations with energy support of the memory system 130, in the normal protocol, the host 120 may issue permanent writes (with Persist=0/1) and reset commands, but the memory system 130 will not assert W_PER for every permanent write with Persist= 1, and the memory system 130 will assert W_PER for the reset instruction when the reset is completed. In the case of WGID implementations, the W_PER processing by the memory system 130 remains the same as in the normal protocol, only for resets. The completed WGID status bitmap provided by the memory system 130 will have the WGID reset bits set when they are completed. When there is a UE in Send-W_PER-Status, the operation remains the same as the normal case, except that the pending WGID status is only applicable for resets.

[0055] Permissions for WGID implementations may be processed as follows. Separate permissions may be maintained for status or Xwrites records and for persistent records, where host 120 may determine how the permission pool may be allocated by memory system 130 . Incremental permission return may be provided by reading metadata, and an encoding scheme may be used to return Xwrite or persistent write permissions. X-Read-Status returns may be available for Xwrite buffer segments and persistent writes based on permission allocation.

[0056] In one implementation, for example, to be described with reference to FIG. 2B-C, memory system 130 may complete persistent writes (referred to here as PWRITE) and flushes in any order. To store a particular PWRITE on the media, host 120 may issue a PWRITE for a given WGID with Persist=1, or issue a PWRITE with Persist=0 followed by any of the reset types. The memory system 130 may issue W_PER for each completed PWRITE that has Persist=1 in command, as well as each completed reset. If multiple PWRITEs are grouped with the same WGID with Persist=1 only in the last PWRITE terminating the group, the memory system 130 may issue W_PER only when the entire PWRITE group is completed.

[0057] With reference now to FIG. 2B, W_PER processing for the normal case will be described. System 250 is shown with host 120 and memory system 130. If both write permissions and a free WGID are available, then host 120 may issue one or more PWRITE or FLUSH commands, shown as 252a, 254a. Host 120 may monitor issued PWRITE or FLUSH commands 252a, 254a in a pending host WGID list (not shown, but may be supported in host 120).

[0058] Accordingly, memory system 130 may receive and monitor pending PWRITE or FLUSH commands 252a, 254a in a pending DIMM-WGID list (not shown). The memory system 130 may execute the pending PWRITE or FLUSH commands 252a, 254a and assert the corresponding W_PERs 254b and 252b (shown in reverse order of the received PWRITE or FLUSH commands 252a, 254a) to the host 120 after the respective completion of each received command.

[0059] The memory system 130 may collect completed received PWRITE or FLUSH commands 252a, 254a in a completed WGID bitmap 260 for which various updates 260a, 260b, 260c, etc. are shown. Memory system 130 may also remove completed PWRITE or FLUSH commands 252a, 254a from the pending DIMM-WGID list.

[0060] Host 120 may maintain a count of received W_PER events, for example, to receive W_PER 254b, 252b, referred to as W_PER-Count. Concurrently, host 120 may process received W_PER events as follows: if W_PER-Count>0, then host 120 may issue a status request, shown as Send-W_PER-Status 256a. After a predetermined time, referred to as time Tsend, memory system 130 may send a snapshot of the status of the completed WGID bitmap 260 at that time (260b in this case) in a response shown as WGID_Status 256b to host 120. The snapshot may include completions for W_PERs issued prior to migration of WGID_Status 256b to host 120.

[0061] In some aspects, at least 1 termination is registered in the snapshot. The memory system 130 clears the bit positions of the completed WGID bitmap 260 based on the terminations sent to WGID_Status 256b, indicated by the transfer of the completed WGID bitmap 260b to the completed WGID bitmap 260c after resetting or clearing the bit positions.

[0062] Host 120 receives WGID-Status 256b and may output information regarding completed WGIDs. Accordingly, host 120 may release completed WGIDs from the host's list of pending WGIDs and decrement W_PER-Count by the number of terminations received in completed WGID bitmap 260 (e.g., decrement count by 2 based on two received W_PERs as indicated by completed WGID bitmap 260b) . Host 120 may repeat the above process, starting with monitoring W_PER-Count, and if W_PER-Count>0, then issue another status request Send-W_PER-Status 256a to memory system 130.

[0063] In exemplary implementations, host 120 and memory system 130 may continue to issue and execute a new PWRITE while the W_PER event is being processed. While W_PER-Count and waitlists such as host WGID pending list, DIMM-WGID pending list, etc. have been discussed for exemplary implementation, alternative structures to achieve the above functionality may be used without departing from the scope of this disclosure.

[0064] With reference now to FIG. 2C, a system 270 is shown for handling an unrecoverable link error (UE) following a SEND-W_PER-Status from host 120 resulting in loss of terminations sent to WGID_Status 256b (explained in FIG. 2B above) from memory system 130. Additionally, it should be noted that the memory system 130 may have been cleared prior to terminations from the completed WGID bitmap 260 in FIG. 2b.

[0065] Accordingly, in a protocol for recovering from such errors in system 270, host 120 may initiate the recovery process by stopping the issuance of new PWRITE or FLUSH commands (e.g., PWRITE-3 or FLUSH-3 272a is not issued, shown with dotted lines to indicate time scales when they would have been issued if the error had not occurred), while the memory system 130 may continue to issue RD_RDY and/or W_PER events for completed reads or PWRITE or FLUSH commands (e.g., W_PER 254b is shown as being issued in while 252b is not issued until recovery from the error). Host 120 may also continue to issue SEND and update W_PER-Count.

[0066] After a predetermined minimum time delay for the write enable signal, referred to as TWE_Delay, following the last PWRITE, the host 120 issues an XREAD-Status 274a to the memory system 130, and the memory system 130 may prepare a full status packet with a snapshot of the status of the bitmap 280 the pending WGID, which is another bitmap provided in addition to the completed WGID bitmap 260 discussed above, where the pending WGID bitmap 280 includes the status of all pending PWRITE/FLUSH. Memory system 130 may assert RD_RDY 276b and host 120 may issue SEND 278a in response.

[0067] Memory system 130 may then return a prepared status packet 278b from which host 120 may retrieve and process the pending WGID bitmap 280 received in status packet 278b. Host 120 may release eligible WGIDs from its pending host WGID tracking list and decrement W_PER-Count by the number of WGIDs released. Host 120 may then repeat the processes starting with the issuance of new PWRITE/FLUSH commands and process pending W_PERs at this time as per the previous call.

[0068] In some aspects, status packet 278b is configured to indicate whether it has a completed WGID bitmap 260 or a pending WGID bitmap 280. The W_PER response status packets contain a completed WGID bitmap 260, while all other status packets contain a pending WGID bitmap 280. The TWE_Delay time is configured to take into account the time to receive error notification from memory system 130 for the last PWRITE issued from host 120, and the time to wait from UE discovery to XREAD-Status 274a issued from host 120 may vary depending on when the last PWRITE was issued.

[0069] With reference to FIG. 3, exemplary coding for persistent entries is shown, for example, for a DDR5 implementation of memory system 130. The CA1 field is commonly used to distinguish between the 1UI and 2UI commands in DDR5 technology, and may be retained for NVDIMM-P implementations. The CA 114 in some implementations may be configured at DDR speeds for DDR5 with only 7 pins, and in such cases, single instruction encoding may be used for persistent entries, such as shown in FIG. 3.

[0070] FIG. 3, if the persistent bit Persist=1, this indicates that the memory system 130 should place all persistent entries associated with the respective WID into non-volatile memory. If there is one 64 B persistent entry in a group, the Persist bit may be set to 1. For persistent entries greater than 64 B, all persistent entries may have the same WID, with the last persistent entry having its Persist bit set to 1, in while the remaining persistent entries have their Persist bits set to 0.

[0071] In addition to the transactions above, as presented in the previous sections, another command may also be used in association with persistent records, referred to as the FLUSH command. The FLUSH command is configured to indicate to the memory system 130 that all previous buffered writes (e.g., in non-persistent or volatile memory) should be placed in read-only memory, given that future writes may not be affected in this manner or placed in read-only memory when the FLUSH command is used. .

[0072] When the FLUSH execution is completed, the memory system 130 may assert Wr_Per again, for example, over the response bus 116 to the host 120, similar to the case of persistent writes discussed above.

[0073] Additionally, host 120 may also provide a Send-Status for WID command (similar to persistent writes) to memory system 130 in the case of a FLUSH command, to which memory system 130 may respond with a WID status packet with a unique reserved WID to indicate completion of the FLUSH execution. (eg, a WID with all bits set to 1 could be such a reserved WID used to indicate completion of a FLUSH execution).

[0074] In one implementation, only one pending FLUSH command from host 120 may be allowed. Thus, in this implementation, host 120 may be forced to wait for a FLUSH completion response from memory system 130 before sending another FLUSH command. In alternative implementations, FLUSH commands may be accompanied by appropriate FLUSH IDs (eg, selected from reserved WID fields), and an appropriate Send-Status response may cause memory system 130 to return a FLUSH ID for which FLUSH execution has been completed.

[0075] It will be understood that aspects include various methods for performing the processes, functions, and/or algorithms disclosed here. For example, FIG. 4A illustrates an exemplary method 400 for performing persistent operations.

[0076] Block 402 includes receiving, at the memory system (eg, memory system 130), permanent write commands (eg, permanent write 202a) and associated write data (eg, data 202b) from the host (eg, host 120).

[0077] Block 404 comprises performing persistent writing of write data to non-volatile memory (eg, non-volatile memory in memory system 130) in the memory system based on the persistent write command. The write identification (WID) associated with the persistent write command may be received from the host, and upon successful completion of the persistent recording, a permanent recording completion indication (Wr_Per) along with the associated WID (eg, Wr_Per 202c) may be provided to the host.

[0078] Similarly, FIG. 4B illustrates another exemplary method 450 for performing persistent operations.

[0079] Block 452 includes providing, from the host (eg, host 120) to the storage system (eg, storage system 130), permanent write instructions (eg, permanent write 202a) and associated write data (eg, data 202b), wherein the command permanent write instructs the memory system to permanently write the recording data to non-volatile memory.

[0080] Block 454 comprises providing a write identification (WID) (eg, WID 210) associated with a permanent write command to the memory system from the host.

[0081] An exemplary apparatus in which aspects of the present disclosure may be used will now be discussed with respect to FIG. 5. FIG. 5 shows a block diagram of computing device 500. Computing device 500 may correspond to an exemplary implementation of processing system 100 of FIG. 1, where processor 120' may be one of the processing elements of host 120. Processor 120' is shown by way of example as connected to memory system 130 via communication mechanism 110, with additional details of communication mechanism 110 omitted from this representation for the sake of clarity. Processor 120', communication engine 110, and memory system 130 may be configured to perform methods 400-450 as discussed above. It will be appreciated that other memory configurations known in the art, such as using one or more cache levels, although not shown, may be present in computing device 500.

[0082] FIG. 5 also shows a display controller 526 that is coupled to processor 120' and to display 528. In some cases, computing device 500 may be used for wireless communication, and FIG. 5 also shows optional blocks in dotted lines such as encoder/decoder (codec) 534 (eg, audio and/or voice codec) associated with processor 120', and speaker 536 and microphone 538 may be coupled to codec 534; and a wireless antenna 542 associated with a wireless controller 540 that is associated with the processor 120'. When one or more of these optional blocks are present, in a particular aspect, processor 120', display controller 526, memory system 130, and wireless controller 540 are included in system-on-a-chip or system-on-a-chip device 522.

[0083] Accordingly, in a specific aspect, the input device 530 and the power supply 544 are connected to the system-on-a-chip device 522. Moreover, in a specific aspect, as illustrated in FIG. 5, where one or more option blocks are shown, display 528, input device 530, speaker 536, microphone 538, wireless antenna 542, and power supply 544 are external to system-on-a-chip device 522. However, each of display 528, input device 530, speaker 536, microphone 538, wireless antenna 542, and power supply 544 may be associated with a component of system-on-a-chip device 522, such as an interface or controller.

[0084] It should be noted that although FIG. 5 generally depicts a computing device, processor 120' and memory system 130 may also be integrated into a set-top box, server, music player, video player, entertainment device, navigation device, personal digital assistant (PDA), fixed location data unit, computer, laptop, tablet, communication device, mobile phone or other similar devices.

[0085] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips that may be referred to throughout the foregoing description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination of them.

[0086] Additionally, those skilled in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

[0087] The methods, sequences, and/or algorithms described in connection with the disclosed aspects may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage media known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. Alternatively, the storage medium may be integrated into the processor.

[0088] Accordingly, an aspect of the invention may include computer-readable media embodying a method for performing persistent recordings. Accordingly, the invention is not limited to the illustrated examples, and any means for performing the functionality described herein are included within aspects of the invention.

[0089] Although the foregoing disclosure shows illustrative aspects of the invention, it should be noted that various changes and modifications may be made without departing from the scope of the invention as defined by the appended claims. The functions, steps and/or actions of the claims on the method in accordance with the aspects of the invention described here do not need to be performed in any particular order. Moreover, while elements of the invention may be described or claimed in the singular, the plural is intended unless limitation to the singular is expressly stated.

Claims (49)

1. A method for performing permanent operations, the method comprising: receiving via a non-volatile dual-sided memory module (NVDIMM) configured to support persistent writes, persistent write commands, and associated write data from a host; performing a permanent write of the write data to the non-volatile memory in the NVDIMM based on the permanent write command; receiving a write identification (WID) associated with the permanent write command from the host upon successful completion of the permanent write; receiving from the host a request to send a status for a plurality of persistent entries along with associated WIDs; and receiving by the NVDIMM a group of two or more persistent write commands with a common WID, wherein the last persistent write command of the group has the Persist bit set to 1 and the remaining persistent write commands have their respective Persist bits set to 0, and providing an indication of completion of the persistent write for the last permanent write command. 2. The method of claim 1, further comprising sending, from the NVDIMM upon successful completion of the persistent write, an indication of the completion of the persistent write along with an associated WID to the host. 3. The method of claim 2, wherein sending the permanent write complete indication comprises sending two or more permanent write complete indications from the NVDIMM to the host in an order different from the order in which the corresponding two or more permanent write commands were received from the host. 4. The method of claim 1, further comprising sending a status packet from the NVDIMM to the host, the status packet containing the WIDs for the permanent write commands that have completed execution. 5. The method of claim 1, wherein the WID contains a multi-bit permanent entry identification and a valid bit. 6. The method of claim 1, further comprising receiving, by the NVDIMM, a FLUSH command from the host, wherein the FLUSH command indicates that all previous writes buffered in volatile media should be placed in nonvolatile or read only memory. 7. The method of claim 6, further comprising sending a FLUSH completion indication upon completion of execution of the FLUSH command from the NVDIMM to the host. 8. The method of claim 1, further comprising receiving by the NVDIMM one or more persistent write commands, maintaining the statuses of one or more persistent write commands completed in the completed bitmap, and the statuses of one or more persistent write commands pending in the pending bitmap, and after a status request from the host, providing a completed bitmap if there is no unrecoverable error, or a pending bitmap if there is an unrecoverable error. 9. A method for performing permanent operations, the method comprising: sending, from the host, to a non-volatile dual-sided memory module (NVDIMM) configured to support persistent writes, persistent write commands, and associated write data; providing a write identification (WID) associated with the persistent write command to the NVDIMM from the host upon successful completion of the persistent write; sending, from the host to the NVDIMM, a request to send status for a plurality of persistent entries along with associated WIDs; and sending from the host to the NVDIMM of a group of two or more persistent write commands with a common WID, with the last persistent write command from the group having the Persist bit set to 1 and the remaining persistent write commands having their respective Persist bits set to 0, and receiving from the NVDIMM specifying the end of the permanent recording for the last permanent recording, wherein the permanent write command instructs the NVDIMM to permanently write the write data to the non-volatile memory. 10. The method of claim 9, further comprising receiving, at the host, a persistent write completion indication along with an associated WID from the NVDIMM upon successful completion of the persistent write. 11. The method of claim 10, wherein receiving the persistent write completion indication comprises receiving two or more persistent write completion indications from the NVDIMM in an order different from the order in which the corresponding two or more persistent write commands were sent from the host to the NVDIMM. 12. The method of claim 11, further comprising receiving a status packet from the NVDIMM by the host, the status packet containing the WIDs for the permanent write commands that have completed execution. 13. The method of claim 9, wherein the WID comprises a multi-bit persistent entry identification and a valid bit. 14. The method of claim 9, further comprising sending a FLUSH command from the host to the NVDIMM, wherein the FLUSH command indicates that all previous writes buffered in volatile media should be placed in non-volatile or persistent memory by the NVDIMM. 15. The method of claim 14, further comprising receiving at the host a FLUSH completion indication after the NVDIMM has completed execution of the FLUSH command. 16. A device for performing permanent operations, comprising: host; and non-volatile dual-sided memory module (NVDIMM) configured to support persistent writes and configured to: receive a persistent write command and associated write data from the host; permanently write the write data to the non-volatile memory in the NVDIMM based on the permanent write command; receive a write identification (WID) associated with the permanent write command from the host after successful completion of the permanent write; receive from the host a request to send a status for a plurality of persistent entries along with associated WIDs; and receive a group of two or more persistent write commands with a common WID, where the last persistent write command of the group has the Persist bit set to 1 and the remaining persistent write commands have their respective Persist bits set to 0, and provide a permanent write completion indication for the last permanent recording commands. 17. The apparatus of claim 16, wherein the NVDIMM is further configured to provide, upon successful completion of the persistent write, an indication of the completion of the persistent write along with an associated WID to the host. 18. The apparatus of claim 17, wherein the NVDIMM is further configured to provide two or more permanent write completion indications to the host in an order different from the order in which the respective two or more permanent write commands were received from the host. 19. The apparatus of claim 18, wherein the NVDIMM is further configured to provide a status packet to the host, the status packet contains the WID for the permanent write commands that have completed execution. 20. The apparatus of claim 16, wherein the WID contains a multi-bit persistent entry identification and a valid bit. 21. The apparatus of claim 16, wherein the NVDIMM is further configured to receive a FLUSH command from the host, the FLUSH command indicating that all previous writes buffered in volatile media should be placed in non-volatile or persistent memory. 22. The apparatus of claim 21, wherein the NVDIMM is further configured to provide a FLUSH completion indication after execution of the FLUSH command to the host has completed. 23. The apparatus of claim 16, wherein the NVDIMM is further configured to receive one or more permanent write commands; maintain the statuses of one or more persistent write commands completed in the completed bitmap and the statuses of one or more pending persistent write commands in the pending bitmap; and after a status request from the host, provide a completed bitmap if there is no unrecoverable error, or a pending bitmap if there is an unrecoverable error. 24. A device for performing permanent operations, comprising: a host configured to provide a persistent write command and associated write data; and a non-volatile reversible memory module (NVDIMM) configured to receive a persistent write command and associated write data sent from a host, wherein the persistent write command instructs the NVDIMM to permanently write the write data to the NVDIMM non-volatile memory, and the host is further configured to provide write identification (WID) associated with a permanent write command to the NVDIMM, and send to the NVDIMM, upon successful completion of the permanent write, a request to send a status for a plurality of permanent writes along with the associated WIDs, and send a group of two or more permanent write commands to the NVDIMM with a common WID, with the last persistent write command from the group having the Persist bit set to 1 and the remaining persistent write commands having their respective Persist bits set to 0, and receive from the NVDIMM a permanent write completion indication for the last persistent write. 25. The apparatus of claim 24, wherein the host is further configured to receive a persistent write completion indication along with an associated WID from the NVDIMM upon successful completion of the persistent write. 26. The apparatus of claim 25, wherein the host is further configured to receive two or more permanent write completion indications from the NVDIMM in an order different from the order in which the corresponding two or more permanent write commands were sent to the NVDIMM. 27. The apparatus of claim 26, wherein the host is further configured to receive a status packet from the NVDIMM, the status packet contains the WID for the permanent write commands that have completed execution. 28. The apparatus of claim 27, wherein the WID contains a multi-bit permanent entry identification and a valid bit. 29. The apparatus of claim 24, wherein the host is further configured to send a FLUSH command to the NVDIMM, wherein the FLUSH command indicates that all previous writes buffered in volatile media should be placed in non-volatile or persistent memory by the NVDIMM. 30. The apparatus of claim 29, wherein the host is further configured to receive a FLUSH completion indication after the NVDIMM completes execution of the FLUSH command.

Images