WO2017131724A1

WO2017131724A1 - Host devices and non-volatile memory subsystem controllers

Info

Publication number: WO2017131724A1
Application number: PCT/US2016/015531
Authority: WO
Inventors: Siamack Ayandeh
Original assignee: Hewlett Packard Enterprise Development Lp
Priority date: 2016-01-29
Filing date: 2016-01-29
Publication date: 2017-08-03

Abstract

In one example, a system may include a host device having at least one port and a non-volatile memory subsystem having, at least two ports, a non-volatile memory subsystem controller assigned to the host device, and a non-volatile memory. The system may further include a first nexus of a pair of one of the at least one port of the host device and one of the at least two ports of the non-volatile memory subsystem. In one example, the host device may assign the first nexus for ordered commands and may assign at least a second nexus for non-ordered commands.

Description

HOST DEVICES AND NON-VOLATILE MEMORY SUBSYSTEM CONTROLLERS

BACKGROUND

[0001] Datacenters and cloud storage arrays often use high-performance, low- latency non-volatile memory (NVM) for storage. Such storage can be accessed over a networking fabric. Storage arrays may receive tens of thousands of individual queues of instructions over a networking fabric. These instructions received over the networking fabric may be received out of order or not received at all. Logic within storage appliances of such datacenters may reorder received instructions to ensure proper execution.

BRIEF DESCRIPTION OF THE DRAWINGS

[0002] FIG. 1 illustrates an example system of the present disclosure;

[0003] FIG. 2 illustrates an additional example system of the present disclosure;

[0004] FIG. 3 illustrates an example method of the present disclosure for transmitting commands to a non-volatile memory subsystem controller;

[0005] FIG. 4 illustrates an additional example method of the present disclosure for transmitting commands to a non-volatile memory subsystem controller;

[0006] FIG. 5 illustrates an example method of the present disclosure for processing commands at a non-volatile memory subsystem controller;

[0007] FIG. 6 illustrates an additional example method of the present disclosure for processing commands at a non-volatile memory subsystem controller; and

[0008] FIG. 7 illustrates a high-level block diagram of an example computer that can be transformed into a machine capable of performing the functions described herein.

DETAILED DESCRIPTION

[0009] In one example, a system may include a host device having at least one port and a non-volatile memory subsystem having, at least two ports, a non-volatile memory subsystem controller assigned to the host device, and a non-volatile memory. The system may further include a first nexus of a pair of one of the at least one port of the host device and one of the at least two ports of the non-volatile memory subsystem. In one example, the host device may assign the first nexus for ordered commands and to assign at least a second nexus for non-ordered commands.

[0010] In another example, a host device to access a non-volatile memory subsystem controller may include a processor and a non-transitory computer- readable medium storing instructions which, when executed by the processor, cause the processor to set a first indicator in a first command to indicate that the first command is an ordered command, set a second indicator in a second command to indicate that the second command is an ordered command that is associated with the first command, transmit the first command and the second command to the non-volatile memory subsystem controller via a first nexus that is assigned for ordered commands, select a second nexus of a plurality of nexuses that are assigned for non-ordered commands, and transmit a third command comprising a non-ordered command to the non-volatile memory subsystem controller via the second nexus.

[0011] In another example, a non-volatile memory subsystem controller may receive a first command and a second command from a host device via a first nexus for ordered commands, determine a sequence for executing the first command and the second command based upon sequence information in a header of the first command and sequence information in a header of the second command, receive a third command from the host device via a second nexus for non-ordered commands, and execute the first command, the second command, and the third command, the first command and the second command being executed in the sequence for executing the first command and the second command that is determined.

[0012] The present disclosure includes a multipath architecture for a host to access a non-volatile memory. Examples of the present disclosure may be applied to protocols such as NVM Express (NVMe) and associated NVM storage arrays where the transport fabric offers connectionless datagram service, such as Fibre Channel (FC), Ethernet, or Fibre Channel over Ethernet (FCoE). Examples of the present disclosure utilize greater than one host/memory subsystem port pair for communications to a single controller of a memory subsystem, which is referred to as an "intra-controller multipath." This is in contrast to other multipath architectures for NVMe which utilize two or more memory subsystem controllers per host, or "inter-controller multipath," to access the same namespace within the non-volatile memory. For example, inter-controller multipath requires synchronization of the two (or more) controllers in accessing a shared namespace. However, using intra- controller multipath to a single controller in accordance with the present disclosure may avoid the synchronization problem.

[0013] Also, with inter-controller multipath, a session may be lost if a single associated fabric port is lost. For example, a connection-oriented transport such as Remote Direction Memory Access (RDMA) class of protocols use a transport layer connection where state is maintained at either end of the connection and the receiver ensures in-order delivery through a sequencing buffer/logic. For instance, RDMA may use Transmission Control Protocol (TCP) for the transport fabric. These connections are tied to a specific nexus for the life of the connection. However, if a connection is lost for any reason, for example loss of an administration queue-pair, the whole host-subsystem session may be reestablished, causing loss of service to the host for the duration.

[0014] In addition, in connectionless transport systems, such as Fibre Channel, each exchange is an independent unit of transport. Hence exchanges are delivered in fabric order, which may be different from the order in which a host hands off commands to a transport layer interface. Therefore a command sequence number may be added by the transport layer interface (e.g., FC-NVMe) to the datagrams, such as Fibre Channel Protocol (FCP) information unit (lUs), which allows the other end, such as a NVM subsystem FC-NVMe transport layer interface, to restore the original sequence. However this scheme works on a per- port basis. If a port is lost for any reason, the whole session may be reestablished, causing a loss of service to the host for the duration.

[0015] In contrast, examples of the present disclosure may increase up-time in the face of transport fabric transient failures, such as physical port up/down events. For instance, using a connectionless fabric, ordered commands may be assigned to a first port pair, or nexus, while non-ordered commands may be assigned to any other available port pair between a host and a memory subsystem. In the event of a failure of the first port pair for ordered commands, a different port pair may simply be reassigned. In addition, non-ordered commands may continue to be load balanced over any other remaining port pairs between the host and the memory subsystem. This mechanism ensures that a host-memory subsystem session may continue and will not trigger a loss of service. These and other aspects of the present disclosure are described in greater detail below in connection with the example FIGs. 1 -7.

[0016] FIG. 1 illustrates an example system 100. In one example, the system 100 may include a fabric 1 10, a host unit 120, and a memory system 130. In the example of FIG. 1 , the host unit 120 and the memory system 130 are connected via ports 121 and 122 of the host unit 120 and ports 131 and 132 of the memory system 130. Thus, the combination of four ports results in four nexuses 191 -194 between the host unit 120 and the memory system 130, wherein each of the nexuses 191 -194 may comprise a separate port pair. In one example, the fabric 1 10 may comprise a connectionless datagram fabric. The fabric 1 10 may include any number of intermediate devices which are omitted for ease of illustration, such as Fibre Channel switches, Ethernet switches, and the like, or any other components which may be deployed in a connectionless transport network. An additional host 160 is also illustrated in FIG. 1 with at least one controller 165, a host memory buffer 169 and ports 161 and 162. Additional host 160 is described in greater detail below. The various ports of the components illustrated in FIG. 1 , may comprise N-ports, for example, in an NVMe system.

[0017] In one example, the host unit 120, broadly a "host device," may comprise a computing device such as a desktop computer, a personal computer, a laptop computer, a blade server or server blade, or any other type of server or computing device. In one example, the host unit 120 includes a processor 125. The processor 125 may comprise a central processing unit (CPU), or the like. In one example, the processor 125 may comprise a CPU with multiple cores. The host unit 120 may further include a host memory buffer 140, which may comprise a volatile memory or working memory, such as dynamic random access memory (DRAM). The host memory buffer 140 may be used to store various queues for use in processing commands for accessing memory system 130. For instance, in the example of FIG. 1 , host memory buffer 140 may include three input/output (I/O) submission queues 141 -143 and three I/O completion queues 144-146. Host memory buffer 140 may further include an administration submission queue 147, an administration completion queue 148, and an additional portion 149 for use by a host operating system, various applications operating on the host unit 120, communication drivers operating at various layers, and so forth. In one example, I/O submission queues and I/O completion queues may be arranged in pairs. For instance, processor 125 may comprise multiple cores, where each core may be assigned at least one I/O submission queue and at least one I/O completion queue. In another example, multiple I/O submission queues may be associated with one I/O completion queue. For instance, a core of processor 125 may be assigned a plurality of I/O submission queues and at least one I/O completion queue.

[0018] The memory system 130 may include one or more memory subsystems. For instance, a first memory subsystem 138 may include a first controller 135, e.g., a non-volatile memory subsystem controller, a first namespace 181 in non-volatile memory 180, and a controller memory buffer 150. In one example, the non-volatile memory 180 may comprise a solid state drive (SSD), e.g., flash memory, or an array of SSDs in a redundant array of independent disks (RAID) arrangement, in a just-a-bunch-of-disks (JBOD) arrangement, and so forth. In one example, the controller 135 may access the non-volatile memory 180 via a PCIe interface. In one example, the controller memory buffer 150 may comprise a volatile memory or working memory, such as DRAM. In one example, the controller memory buffer 150 may be used to store various queues for use in processing commands for the first memory subsystem 138 by controller 135. For instance, in the example of FIG. 1 , controller memory buffer 150 may include three I/O submission queues 151 -153 and three I/O completion queues 154-156. Controller memory buffer 150 may further include: an administration submission queue 157, an administration completion queue 158, and an additional portion 159 for use by a subsystem operating system, various applications operating on the memory subsystem, communication drivers operating at various layers, and so forth. It should be noted that queues for I/O as well as administration queues may alternatively be instantiated in the host memory buffer 140 or the controller memory buffer 150. In one example, some queues may be instantiated in the host memory buffer 140 while other queues may be instantiated in the controller memory buffer 150. However, for illustrative purposes, FIG. 1 will now be described in connection with examples where queues for host unit 120 interacting with the first memory subsystem 138 are maintained in controller memory buffer 150.

[0019] In the example of FIG. 1 , memory system 130 may include a second memory subsystem 139 comprising a second controller 136, a second namespace 182 in non-volatile memory 180, and a controller memory buffer 170. In one example, the controller 135 and the controller 136 may comprise respective CPU cores of memory system 130. In another example, controllers 135 and 136 may comprise components of respective virtual machines which run atop a shared physical infrastructure of memory system 130. For instance, non-volatile memory 180 may comprise a storage array that includes any number of component storage devices which may be apportioned into a plurality of namespaces. In addition, different namespaces may be allocated for different controllers and/or for different host devices that are assigned to the different controllers. In addition, in one example, the number of controllers may be increased or decreased to implement various configurations of the memory system 130 and to provide storage resources to any number of host devices.

[0020] In one example, different subsystems may be identified and addressed by different NQNs. In addition, as mentioned above, ports of a memory system may be shared by different subsystems. Thus, in the example of FIG. 1 , the controller 135 and the controller 136 may both be reached via ports 131 and 132. As also mentioned above, different hosts may be assigned to different memory subsystems. In the example of FIG. 1 , host unit 120 may be assigned to the first memory subsystem 138 comprising controller 135, namespace 181 in non-volatile memory 180, and controller memory buffer 150, while the additional host 160 may be assigned to the second memory subsystem 139 comprising controller 136, namespace 182 in non-volatile memory 180, and controller memory buffer 170. Additional host 160 may use ports 161 and 162 to connect to ports 131 and 132 of memory system 130 over fabric 1 10. Thus, the additional host 160 and the memory system 130 may also have four nexuses 196-199 for communication over the fabric 1 10. In one example, to establish the various nexuses 191 -194 and 196- 199, each host port may perform a port login (PLOGI) and process login (PRLI) with each memory subsystem port. In one example, each nexus is for an independent stream of datagram I/O, i.e., connectionless transport, over fabric 1 10.

[0021] In accordance with the present disclosure, a host and a memory subsystem may communicate via at least two nexuses. For instance, a host- memory subsystem session may utilize at least two nexuses. In one example, processor 125 of the host unit 120 may assign at least a first nexus, e.g., nexus 191 for ordered commands, and may assign at least a second nexus, e.g., nexus 192, for non-ordered commands. In one example, a plurality of nexuses, e.g., nexuses 192, 193, and 194, may be assigned for non-ordered commands. Ordered commands may comprise commands for interacting with a non-volatile memory, or namespace of a non-volatile memory, that have some order in which the command should be executed in relation to another command. For instance, commands may be paired or be arranged in groups greater than two. To illustrate, a first command may be to perform a test, and a second command may be to set a flag based upon a result of the test. In this case, the second command should follow the first command since the second command depends upon the result of the first command. Thus, the order is important and the commands may be referred to as "ordered commands." Other commands, such as reading a block of memory, may not be sensitive to the order of execution. Such commands may be referred to as "non-ordered commands," or "fabric order commands."

[0022] In one example, I/O submission queues and I/O completion queues for host-subsystem interactions may be associated with different nexuses. In one example, a nexus for ordered commands may have at least one I/O submission queue and at least one I/O completion queue that are dedicated to the nexus for ordered commands. For instance, nexus 191 , dedicated for the transport of ordered commands and responses thereto, may be further assigned to and/or associated with I/O submission queue 151 and I/O completion queue 154. Other nexuses for non-ordered commands may be collectively associated with a plurality of I/O submission queues and I/O completion queues. For instance, I/O submission queues 152 and 153, and input output/completion queues 155 and 156 may be collectively associated with the nexuses 192, 193, and 194. In addition, in one example, each of the I/O submission queues 152 and 153 may be associated with at least one of the I/O completion queues 155 and 156. For instance, non- ordered commands (commands that are not order-sensitive) may be sent from host unit 120 to any one of I/O submission queues 152 or 153 over any one of nexuses 192, 193, or 194. Similarly, a response from controller 135, e.g., regarding a completion of the command, may be placed in any of I/O completion queues 155 or 156 and sent back to processor 125 via any one of nexuses 192, 193, or 194. In one example, the I/O submission queues 152 and 153 and I/O completion queues 155 and 156 may be arranged in submission queue-completion queue pairs. In another example, I/O submission queues 152 and 153 may both be associated with one of I/O completion queues 155 and 156.

[0023] In one example, processing of a non-ordered command may proceed as follows. Processor 125 of host unit 120 may implement an application layer, a host NVM protocol layer, a fabric interface layer, and a fabric transport layer. Similarly, controller 135 of memory subsystem 138 may implement a fabric transport layer, a fabric interface layer, and a controller software layer. The transport interface layer of host unit 120 can have nexuses with one or more memory subsystem ports, and subsystems. Similarly, a transport interface layer of memory subsystem 138/controller 135 can have nexuses with multiple ports from the same host, or from one or more additional hosts.

[0024] In one example, the various layers of processor 125 and memory controller 135 may utilize respective portions of host memory buffer 140 and controller memory buffer 150, e.g., portions 149 and 159, respectively, to support various operations. In one example, an application may run in the application layer of processor 125 of the host unit 120. This application may generate a service call for the host NVM protocol. For example, the application may request to read or write a unit of data in namespace 181. The host NVM protocol may generate a command, or a "command entry," for an I/O submission queue based on the service call. The command can contain information about the location of the data, associated pointers, and the data itself (if the command is a write command). The fabric interface layer may encapsulate this command into a capsule. This capsule may include control information such as a controller identifier, a NQN of the memory subsystem, and/or a queue identifier where the encapsulated command entry should be sent. For instance, processor 125 may select I/O submission queue 152 for the command.

[0025] This capsule may be further encapsulated by fabric transport layer into an information unit (IU). The IU may contain fabric-specific information. The IU may then be included in one or more frames. For example, the IU may be included in a single frame, or may be fragmented into multiple frames. As referred to herein, a frame may comprise a connectionless datagram transport unit for transmission over fabric 1 10, e.g., a connectionless datagram transport fabric/network. Each frame may include a header and a data portion. The header may include information such as a source address (i.e., the address of host unit 120) a destination address (e.g., an address of memory system 130), as well as other related fields. For instance, the fabric transport layer may select a nexus associated with non-ordered commands for transporting the frame(s). For example, nexus 193 may be selected from among nexuses 192, 193, and 194 that are assigned to non-ordered commands. Accordingly, identifiers of the selected nexus and/or a selected egress port of the host, a selected ingress port of the memory system 130, and so forth, may further be included in the frame(s) that are to be transmitted. The frame(s) may then be transmitted via the selected nexus 193, e.g., from port 122 of host unit 120 to port 131 of memory device 130 over fabric 1 10.

[0026] The frame(s) may be received by memory device 130 via fabric 1 10. The fabric transport layer of controller 135 may recover the transmitted IU(s) and the capsule. The fabric interface layer may then recover the command from the capsule, and the command may be stored in a queue controlled by the controller software. For instance, the command may be placed in I/O submission queue 152 that was selected by processor 125 of the host unit 120 and identified in the capsule. After execution of the command by the controller 135, a response may be returned to the application implemented by processor 125 of host unit 120. For instance, controller 135 may place a response in either of I/O completion queues 155 or 156. The response may then be transmitted back to processor 125 of host 125 via either of ports 131 or 132 and over any one of nexuses 192, 193, or 194. The response may comprise an acknowledgement that the command has been executed. Alternatively, or in addition, the response may comprise payload data, such as data that has been retrieved after performing a read command, a pointer to indicate a location where data has been stored after performing a write operation, and so forth. In one example, memory system 130 may designate one controller, from several controllers of the memory system 130, for routing communications to appropriate subsystems/memory controllers based upon the NQN or other information contained in the frames, lUs, and/or the capsules contained therein.

[0027] It should be noted that since multiple I/O submission queues are available for commands from the host unit 120 to the first memory subsystem 138, the processor 125 may implement any number of algorithms to determine which I/O submission queue to use for the non-ordered command. For instance, processor 125 may select an I/O submission queue using a round robin, or a weighted round robin algorithm, or any other load balancing/load distribution algorithm. Controller 135 may then select an I/O completion queue for the response based upon which I/O completion queue is assigned to or associated with the I/O submission queue that has been selected and utilized. In addition, it should be noted that since there may be multiple I/O submission queues stored in controller memory buffer 150 which may store all commands awaiting execution, the controller 135 may also implement a round robin or a weighted round robin algorithm, or any other load balancing/load distribution algorithm to determine which commands from which queue to execute and in which order.

[0028] Processing of ordered commands may proceed similar to the processing of non-ordered commands outlined above, with several distinctions. For example, since an I/O submission queue 151 has been reserved for ordered commands, processor 125 of host unit 120 may omit selecting an I/O submission queue from one or more I/O submission queues. Processor 125 of host unit 120 may also omit selecting a nexus for transporting frames, lUs, and/or capsules containing the ordered commands, since nexus 191 has also been reserved for transporting ordered commands. Rather, processor 125 may simply send one or more frames to port 121 for transport over nexus 191 and for delivery to I/O submission queue 151. Nevertheless, processor 125 may also designate the first memory subsystem 138 and/or the controller 135, e.g., by including the subsystem NQN in a capsule for the command(s) contained in the IU(s)/frame(s), identify the nexus 191 and/or the egress port 121 , and so forth. In addition, processor 125 may include a command sequence number and/or a sequence identifier (e.g., a, a+1 , a+2... a+n), in a command header of each ordered command, thus enabling the controller 135 to determine which ordered commands are associated with one another, and the correct order to arrange the ordered commands. Alternatively, or in addition, a command header may also include a flag to indicate that the command is an ordered command.

[0029] To illustrate, an application operating at an application layer implemented by processor 125 may generate a command and designate the command as being an ordered command. Alternatively, or in addition, the application layer may also include a sequence identifier and a sequence number. A host NVM protocol layer, a fabric interface layer, and a fabric transport layer may further encapsulate and prepare the commands for transport across the fabric 1 10 by including other relevant information, such as the NQN of the memory subsystem/processor 125, an identifier of host 120 and port 131 , an identifier of the nexus 191 , etc. The ordered commands may then be received in frames/IUs containing capsules, and commands within the capsules, at memory system 130 via nexus 191 and port 131. The appropriate subsystem, and hence the appropriate controller to receive the contents may be identified in the frame/IU headers. For instance, an NQN may identify controller 135.

[0030] Controller 135 may forward the commands to I/O submission queue 151 . The I/O submission queue 151 may be identified in the capsule header, or controller 135 may determine that the commands are ordered commands based upon a command header and/or based upon being received over nexus 191 , and place the commands in the I/O submission queue 151 which has been designated for ordered commands. However, the commands may be received in a different order from an order in which the commands were sent. Thus, controller 135 may rearrange the ordered commands into the correct sequence in I/O submission queue 151 . In one example, the rearrangement of ordered commands may be performed within the I/O submission queue 151 itself. However, in another example, an additional temporary buffer may be used to arrange ordered commands that may be received out of order, before placing such commands into the I/O submission queue 151 in the correct sequential order.

[0031] It should be noted that the controller 135 may also arbitrate ordered commands in I/O submission queue 151 and non-ordered commands in I/O submission queues 152 and 153 for execution. For example, as mentioned above controller 135 may implement any load balancing/load distribution algorithm for selecting which commands from which I/O submission queues for execution and in which order. In one example, a greater weighting may be given to selecting commands from I/O submission queue 151 . However, in another example, no specific positive weighting may be given to I/O submission queue 151 , since ordered commands may be relatively infrequent as compared to non-ordered commands.

[0032] In any case, when selecting a command from I/O submission queue 151 for execution, or for determining an order for execution, the controller 135 may also select a second, or any number of additional commands from I/O submission queue 151 that are paired with or grouped with the command being selected. For instance, controller 135 may read as many commands from the I/O submission queue 151 as are found with the same sequence identifier and with sequence numbers indicating an uninterrupted sequence. After executing the ordered commands, controller 135 may then place one or more responses in I/O completion queue 154 for transport over nexus 191 back to host unit 120 and processor 125.

[0033] In general, commands from host unit 120 to the first memory subsystem 138 may utilize a first nexus, e.g., nexus 191 , if the commands are ordered commands, and may be load balanced over any remaining nexuses, e.g., nexuses 192, 193, and 194, if the commands are non-ordered commands. In addition, if the nexus 191 for ordered commands fails, e.g., due to a problem with one of ports 121 or 131 , or at any intermediate device or link in the fabric 1 10, a different host device-memory subsystem nexus may be reassigned to ordered commands. For example, a nexus may be selected from among nexuses 192-194 for reassignment. Any I/O submission queue and I/O completion queue pair for ordered commands may then be associated with the different nexus that is selected. The remaining nexuses from among nexuses 192-194 may continue to be used for non-ordered commands. Thus, a host-memory subsystem session is not lost and reestablishing the session may be avoided in the face of a failure of the nexus 191 . In one example, if nexus 191 is repaired, ordered commands may be returned to nexus 191 . However, in another example, the different nexus that is selected may continue to be used to transport ordered commands and responses, while nexus 191 may be included in a pool of nexuses to transporting non-ordered commands and responses.

[0034] The foregoing describes just one example of a system of the present disclosure. In other words, various configurations and modifications of the system 100 may be provided in other examples of the present disclosure. For instance, host 160 may interact with the second memory subsystem 139 in a similar manner as described above in connection with host-memory subsystem interactions between host unit 120 and the first memory subsystem 138. Similarly, memory system 130 may include additional memory subsystems that may be accessed by the same or different hosts. In one example, non-volatile memory 180 may be reconfigured to have more or less namespaces, or different namespaces. In addition, different portions of non-volatile memory 180 and/or namespaces 181 and 182 may be reassigned to different hosts at different times. For instance, at a first time, host unit 120 may be associated with memory subsystem 138 and namespace 181 , while host 160 may be associated with memory subsystem 139 and namespace 182. At a later time, host unit 120 may be associated with namespace 182 while host 160 may be associated with namespace 181 . In addition, the controllers for accessing namepaces 181 and 182 may also be reconfigured and changed at various times, and may be assigned to different hosts. In still another example, controller memory buffer 150 may be shared among different controllers of the memory system 130, e.g., as opposed to controller 135 having exclusive use of controller memory buffer 150 and controller 136 having exclusive use of controller memory buffer 170. In another example, hosts 120 and 160 may also access different memory systems as an alternative or in addition to memory system 130.

[0035] It should be noted that the foregoing is primarily described in connection with processor 125 interacting with controller 135. However, it should be noted that processor 125 may comprise multiple cores that may access and interact with the same memory subsystem 138. In one example, the multiple cores may comprise physical cores. In another example, the utilization of processor 125 may be shared among different virtual machines such that processor 125 appears as one or more separate cores for each of the virtual machines. In such examples, different cores of processor 125 may be provided with different I/O submission queues and/or I/O completion queues to avoid locking between cores. This may include different I/O submission queues for ordered commands and non-ordered commands for each core, and corresponding I/O completion queues for ordered command and non- ordered commands for each core. In this regard, the number of nexuses between the host 120 and memory subsystem 138 may be sufficiently large such that each core may have at least two independent nexuses to communicate with the memory subsystem 138: one for ordered commands and one for non-ordered commands. However, in another example I/O submission queues and I/O completion queue may be shared by applications that are running on the different cores.

[0036] Furthermore, in accordance with the present disclosure, each memory subsystem 138 and 139 may be associated with one host, with multiple nexuses between each host device and each memory subsystem, referred to herein as "intra-controller multipath." However, in other examples of the present disclosure, the intra-controller multipath architecture may also be used in conjunction with an inter-controller multipath (with more than one controller per subsystem being accessed by a host device). To illustrate, multiple nexuses may be provided for each host device-memory subsystem session, with at least one nexus for ordered commands and the remaining nexuses for non-ordered commands. In addition, the multiple nexuses may be used to reach more than two controllers having access to a shared namespace. For example, the different controllers may be selected by the host device according to a load-balancing arrangement. The execution of commands by the controllers may therefore be coordinated in the access to the namespace. In still another example, more than one nexus may be used for delivery of ordered commands. In such case, each nexus for ordered commands may be assigned is own I/O submission queue and I/O completion queue. Thus, these and other modification are all contemplated within the scope of the present disclosure.

[0037] FIG. 2 illustrates an example system 200 of the present disclosure. In one example, system 200 includes a host unit 220 to access a memory subsystem 238 over a fabric 210. As illustrated in FIG. 2, the host unit 220 may include at least one port 221 that is connected to at least two ports 231 and 232 of memory system 230 via at least two nexuses 291 and 292 over the fabric 210. In one example, the nexus 291 may be assigned to transport ordered commands for accessing non-volatile memory 280, while the nexus 292 may be assigned to transport non-ordered commands. In one example, the nexuses 291 and 292 comprise different transport paths over fabric 210, e.g., a connectionless datagram fabric comprising one or more intermediate devices and links therebetween. In one example, memory system 230 includes at least the memory subsystem 238. Memory subsystem 238 may include ports 231 and 232, the controller 235, and at least a portion of non-volatile memory 280. In one example, the host unit 220 is to assign a first nexus for ordered commands, e.g., nexus 291 , and to assign a second nexus for non-ordered commands, e.g., nexus 292.

[0038] In one example, the host unit 220 may perform functions as described below in connection with the example methods 300 and 400 of FIGs. 3 and 4, respectively. In one example, the controller 235 may perform functions as described below in connection with the example methods 500 and 600 of FIGs. 5 and 6, respectively. In one example, system 200 may comprise a similar system to that which is illustrated in FIG. 1. For instance, host unit 220 may comprise the same or similar components as the host unit 120 in FIG. 1 , memory system 230 may comprise the same or similar components as memory system 130, and fabric 210 may comprise the same or a similar component as fabric 1 10. In this regard, system 200 may include various additional components that are omitted from the example of FIG. 2, such as additional memory subsystems, additional hosts for accessing the memory system 230 and any memory subsystems therein, additional ports, nexuses, and components of fabric 210, host and/or controller memory buffers, and so forth. For example, memory system 230 may include a second controller to receive commands from a second host unit via the at least two ports 231 and 232, and the non-volatile memory 280 may include a first namespace that is assigned to the controller 235 and the host unit 220, and a second namespace that is assigned to the second controller and the second host unit. In addition, system 200 may be modified in various other ways. For instance, in one example, system 200 may be modified such that at least two ports of host device 220 are used to connect to memory subsystem 238 via at least one port of memory system 230. Thus, these and other modifications are all contemplated within the scope of the present disclosure.

[0039] FIG. 3 illustrates a flowchart of an example method 300 for transmitting commands to a non-volatile memory subsystem controller. The method 300 may be performed, by at least one component of the system 100 of FIG. 1 or system 200 of FIG. 2, for example, by the host unit 120 of FIG. 1 or the host unit 220 of FIG. 2. However, the method 300 is not limited to implementation with the system and devices illustrated in FIGs. 1 and 2, but may be applied in connection with any number of systems and devices for transmitting commands to a non-volatile memory subsystem controller. Alternatively, or in addition, at least one of the blocks of the method 300 may be implemented by a computing device having a processor, a memory, and I/O devices as illustrated below in FIG. 7, specifically programmed to perform the blocks of the method. Although any one of the elements in system 100, system 200, or in a similar system, may be configured to perform various blocks of the method 300, the method will now be described in terms of an example where blocks of the method are performed by a processor, such as processor 702 in FIG. 7. For instance, processor 702 may represent a core and/or a processor of a host unit, or host device, as described in FIGs. 1 and 2 above. As used in connection with the description of FIG. 3, the term "processor" may also include multiple processors, hardware state machines, or hardware logic units, e.g., an application specific integrated circuit (ASIC), a programmable logic device (PLD), such as a field programmable gate array (FPGA), and so forth.

[0040] The method 300 begins in block 305 and proceeds to block 310. In block 310, the processor may set a first indicator in a first command to indicate that the first command is an ordered command. In one example, the first command may be generated via an application implemented on the processor, or on a different processor of a host unit. In one example, the indicator may comprise a flag that is set in header of the first command. For example, a bit may be set to "1" to indicate that the first command is an ordered command for in-order processing/execution, while setting the bit to "0" may indicate that the first command is a non-ordered command. Alternatively, or in addition, the indicator may comprise a sequence identifier, which indicates that the first command is an ordered command and which sequence the first command belongs to, and a sequence number.

[0041] In block 320, the processor may set a second indicator in a second command to indicate that the second command is an ordered command that is associated with the first command. In one example, the second command may be generated via a same application as the first command of block 310. In one example, the indicator may comprise a flag that is set in the header of the second command. Alternatively, or in addition, the indicator may comprise a sequence identifier that indicates the second command is part of the same sequence as the first command, and a sequence number that indicates the relative order for executing the second command in relation to the first command (and with respect to any other commands in the same sequence for in-order execution).

[0042] In block 330, the processor may transmit the first command and the second command to a non-volatile memory subsystem controller of a non-volatile memory subsystem via a first nexus that is assigned for ordered commands. The first nexus may comprise a first port pair comprising a port of a host device and a port of a memory subsystem. In one example, the processor may assign the first nexus to transport ordered commands to the memory subsystem and to receive responses thereto from the memory subsystem. The processor may also assign a plurality of other nexuses to transport non-ordered commands to the memory subsystem and to receive responses thereto from the memory subsystem. The other nexuses may comprise different port pairs, or different combinations of ports of the host device and the memory subsystem. For instance, the first nexus may comprises a first port of the host device and a first port of the non-volatile memory subsystem. A second nexus of the plurality of nexuses for non-ordered commands may then comprises one of: the first port of the host device and a second port of the non-volatile memory subsystem, a second port of the host device and the first port of the non-volatile memory subsystem, or the second port of the device and the second port of the non-volatile memory subsystem. In one example, block 330 may include encapsulating the first command and the second command in capsules, placing the capsules in one or information units (lUs), and transmitting the IU(s) in one or more frames over the first nexus. In one example, the nonvolatile memory subsystem controller may comprise controller 135 in FIG. 1 or controller 235 in FIG. 2.

[0043] In block 340, the processor may select a second nexus of a plurality of nexuses that are assigned for non-ordered commands. For instance, as mentioned above, the processor may assign a plurality of nexuses for non-ordered commands. Thus, when the processor has a non-ordered command for delivery to the non-volatile memory subsystem, the processor may select which of the available nexuses to use to transport the command to the non-volatile memory subsystem. In one example, the processor may select a nexus of the plurality of nexuses using a round robin, a weighted round robin, or other load-balancing/load- distribution algorithm.

[0044] In block 350, the processor transmits a third command comprising a non- ordered command via the second nexus that is selected. In one example, block 350 may include encapsulating the third command in a capsule, placing the capsule in one or more lUs, and transmitting the IU(s) in one or more frames over the first nexus. Following block 350, the method 300 proceeds to block 395 where the method ends.

[0045] FIG. 4 illustrates a flowchart of an example method 400 for transmitting commands to a non-volatile memory subsystem controller. The method 400 may be performed, by at least one component of the system 100 of FIG. 1 or system 200 of FIG. 2, for example, by the host unit 120 of FIG. 1 or the host 220 of FIG. 2. However, the method 400 is not limited to implementation with the system and devices illustrated in FIGs. 1 and 2, but may be applied in connection with any number of systems and devices for transmitting commands to a non-volatile memory subsystem controller. Alternatively, or in addition, at least one of the blocks of the method 400 may be implemented by a computing device having a processor, a memory, and I/O devices as illustrated below in FIG. 7, specifically programmed to perform the blocks of the method. Although any one of the elements in system 100, system 200, or in a similar system, may be configured to perform various blocks of the method 400, the method will now be described in terms of an example where blocks of the method are performed by a processor, such as processor 702 in FIG. 7. For instance, processor 702 may represent a core and/or a processor of a host unit, or a host device, as described in FIGs. 1 and 2 above. As used in connection with the description of FIG. 4, the term "processor" may also include multiple processors, hardware state machines, or hardware logic units, e.g., an application specific integrated circuit (ASIC), a programmable logic device (PLD), such as a field programmable gate array (FPGA), and so forth.

[0046] The method 400 begins in block 405 and proceeds to block 410. In block 410, the processor assigns at least a first nexus for ordered commands for accessing a non-volatile memory of a non-volatile memory subsystem. The first nexus may comprise one of plurality of nexuses that are available to connect a host device of the processor to the non-volatile memory subsystem. Each of the plurality of nexuses may comprise a different port pair comprising one port of the host device and one port of the non-volatile memory subsystem. In one example, ports of the non-volatile memory subsystem may be shared with other non-volatile memory subsystems of a same memory system.

[0047] In block 415, the processor assigns a second nexus to a plurality of nexuses for non-ordered commands. For instance, in one example, after assigning at least one nexus for ordered commands, any remaining nexuses between the host device and the non-volatile memory subsystem may be assigned to a plurality of nexuses for non-ordered commands. However, in one example, more than one nexus may be assigned for ordered commands. For instance, if the host device comprises multiple processors and/or multiple cores, each processor may be assigned a nexus for ordered commands, or several processors may share one nexus for ordered commands, while one or more other processors may share a different nexus for ordered commands. In one example, all or a portion of the remaining nexuses may then be collectively assigned to the multiple processors and/or cores for transporting non-ordered commands and responses thereto.

[0048] In block 420, the processor may set a first indicator in a first command to indicate that the first command is an ordered command. In one example, block 420 may comprise the same or similar operations as described above in connection with block 310 of the method 300.

[0049] In block 425, the processor may set a second indicator in a second command to indicate that the second command is an ordered command that is associated with the first command. In one example, block 425 may comprise the same or similar operations as described above in connection with block 320 of the method 300.

[0050] In block 430, the processor may place the first command and the second command in a first input/output (I/O) submission queue for ordered commands. In one example, the first I/O submission queue may reside on a host memory buffer of a host device of the processor, from where the processor may later send the commands to the non-volatile memory subsystem, and/or from where a nonvolatile memory subsystem controller of the non-volatile memory subsystem may retrieve the commands from the host memory buffer. However, in another example, the first I/O submission queue may reside on a controller memory buffer of the non-volatile memory subsystem and may be associated with the first nexus for ordered commands. For instance, all commands received via the first nexus for ordered commands may be placed in the first I/O submission queue in the controller memory buffer. In this regard, block 430 may be omitted from the performance of the method 400 in some examples of the present disclosure.

[0051] In block 435, the processor may place the third command in a second I/O submission queue for non-ordered commands. In one example, the second I/O submission queue may reside on a host memory buffer of a host device of the processor, from where the processor may later send the commands to the nonvolatile memory subsystem, and/or from where the non-volatile memory subsystem controller may retrieve the third command from the host memory buffer. In one example, the second I/O submission queue may be one of a plurality of I/O submission queues for non-ordered commands. Thus, in one example, the processor may select the second I/O submission queue from the plurality of I/O submission queues for non-ordered commands using any load distribution/load balancing algorithm, such as a round robin or weighted round robin algorithm. The plurality of I/O submission queues for non-ordered commands may be collectively associated with the plurality of nexuses for non-ordered commands that are available. For instance, the second nexus, or any one of the plurality of nexuses for non-ordered commands, may be used to transport a non-ordered command from the second I/O submission queue, or any other one of the plurality of I/O submission queues for non-ordered commands, to the non-volatile memory subsystem.

[0052] It should be noted that in another example, the second input/output submission queue may reside on a controller memory buffer of the non-volatile memory subsystem and may be associated with the first nexus for ordered commands. For instance, the third command may be received by the non-volatile memory subsystem via any nexus for non-ordered commands and may be placed in an input/output submission queue for non-ordered commands in the controller memory buffer. In such an example, the processor may include an indicator in a capsule of the third command to indicate which input/output submission queue should receive the third command. In this regard, it should also be noted that block 435 may be omitted from the performance of the method 400 in some examples of the present disclosure.

[0053] In block 440, the processor may transmit the first command and the second command to the non-volatile memory subsystem controller via the first nexus that is assigned for ordered commands. In one example, block 440 may comprise the same or similar operations as described above in connection with block 330 of the method 300. However, in another example, block 440 may include the processor notifying the non-volatile memory subsystem and/or the nonvolatile memory subsystem controller of commands that are in the first I/O submission queue awaiting execution, receiving a request from the non-volatile memory subsystem and/or the non-volatile memory subsystem controller for the commands, and transmitting the commands via the first nexus in response to the request. For instance, the processor may provide to the non-volatile memory subsystem an updated tail doorbell register associated with the first I/O submission queue when there are new commands added to the first I/O submission queue that are awaiting execution. The non-volatile memory subsystem controller may then retrieve the commands in the order in which the commands are stored in the first I/O submission queue, where the retrieval may comprise the processor of the host device encapsulating the commands in capsules, placing the capsules in one or more lUs, and sending the IU(s) in one or more frames over the nexus for ordered commands to the memory subsystem.

[0054] In block 445, the processor selects the second nexus (e.g., to transmit a third command comprising a non-ordered command). For instance, as mentioned above, the processor may assign a plurality of nexuses for non-ordered commands. Thus, when the processor has a non-ordered command for delivery to the non-volatile memory subsystem controller, the processor may select one of the available nexuses to transport the command. In one example, the processor may select the second nexus of the plurality of nexuses using any type of load- balancing/load-distribution algorithm. In one example, block 445 may be performed in response to receiving a request from the non-volatile memory subsystem controller to access one or more commands in the second I/O submission queue. In one example, block 445 may comprise the same or similar operations as described above in connection with block 340 of the method 300.

[0055] In block 450, the processor may transmit a third command comprising a non-ordered command to the non-volatile memory subsystem controller via the second nexus that is selected. In one example, block 450 may comprise the same or similar operations as described above in connection with block 350 of the method 300. However, in another example, block 450 may include the processor notifying the non-volatile memory subsystem controller of one or more commands (including at least the third command) that are in the second I/O submission queue awaiting execution, receiving a request from the non-volatile memory subsystem controller for the command(s), and transmitting the command(s) via the first nexus in response to the request. For instance, the processor may provide to the nonvolatile memory subsystem controller an updated tail doorbell register associated with the second I/O submission queue when the third command is added to the second I/O submission queue and awaiting execution. The non-volatile memory subsystem controller may then retrieve the third command from the second I/O submission queue, where the retrieval may comprise the processor of the host device encapsulating the third command in a capsule, placing the capsule in one or more lUs, and sending the IU(s) in one or more frames to the memory subsystem over the second nexus for non-ordered commands that has been selected.

[0056] In block 455, the processor may reassign the second nexus to ordered commands in response to an impairment of the first nexus. For example, a physical port up/down event, a failure of any intermediate device or link in the fabric, or other impairment may render the first nexus unusable or unsuitable for use in transporting communications between the host device and the non-volatile memory subsystem. In one example, the processor may maintain at least one nexus that is reserved for transporting ordered commands and responses thereto. As such, the processor may reassign the second nexus to ordered commands, while any remaining nexuses of the plurality of nexuses for non-ordered commands may continue to be used to load balance the transport of non-ordered commands and responses thereto. In one example, the second nexus may be selected by the processor for reassignment from among the plurality of nexuses for non-ordered commands. In one example, as part of the reassignment, the second nexus may also be associated with an I/O submission queue and an I/O completion queue for ordered commands. Thus, a host-subsystem session is not lost and does not need to be reestablished in the face of a failure of the first nexus. In one example, if the first nexus is repaired, ordered commands may be returned to the first nexus and the second nexus returned to the plurality of nexuses for non-ordered commands. However, in another example the second nexus that is reassigned may continue to be used to transport ordered commands, while the first nexus may be included in the plurality of nexuses for transporting non-ordered commands and responses thereto.

[0057] Following block 455, the method 400 proceeds to block 495 where the method ends. In addition, it should be noted that the method 400 may be modified or expanded in various examples, and that any one or more blocks of the method 400 may be omitted. For example, method 400 may be expanded to include receiving responses to the commands sent by the processor to the non-volatile memory subsystem controller. In one example, blocks 430 and 435 may be omitted. In another example, block 455 may be omitted. In still another example, method 400 may be modified to process additional ordered commands in the same sequence as the first command and the second command, additional non-ordered commands, and so forth.

[0058] FIG. 5 illustrates a flowchart of an example method 500 for processing commands at a non-volatile memory subsystem controller. The method 500 may be performed, by at least one component of the system 100 of FIG. 1 or system 200 of FIG. 2. For example, the method 500 may be performed by the memory subsystem 130 or the memory controller 135 of FIG. 1 , or the memory subsystem 230 or the memory controller 235 of FIG. 2. However, the method 500 is not limited to implementation with the system and devices illustrated in FIGs. 1 and 2, but may be applied in connection with any number of systems and devices for processing commands at a non-volatile memory subsystem controller. Alternatively, or in addition, at least one of the blocks of the method 500 may be implemented by a computing device having a processor, a memory, and I/O devices as illustrated below in FIG. 7, specifically programmed to perform the blocks of the method. Although any one of the elements in system 100, system 200, or in a similar system, may be configured to perform various blocks of the method 500, the method will now be described in terms of an example where blocks of the method are performed by a processor, such as processor 702 in FIG. 7. For instance, processor 702 may represent a non-volatile memory subsystem controller as described in FIGs. 1 and 2 above. As used in connection with the description of FIG. 5, the term "processor" may also include multiple processors, hardware state machines, or hardware logic units, e.g., an application specific integrated circuit (ASIC), a programmable logic device (PLD), such as a field programmable gate array (FPGA), and so forth.

[0059] The method 500 begins in block 505 and proceeds to block 510. In block 510, the processor may receive a first command and a second command from a host device via a first nexus for ordered commands. In this regard, the first command and the second command may both comprise ordered commands for interacting with a non-volatile memory of a non-volatile memory subsystem that is associated with the processor. In addition, the first command and the second command may be paired together, or may comprise commands that are part of a same sequence comprising more than two commands. For example, the first command may be to perform a test on a non-volatile memory or a portion thereof, while the second command may be to set a flag based upon a result of the test. In this case, the second command should follow the first command since the second command depends upon the result of the first command.

[0060] In block 520, the processor may determine a sequence for executing the first command and the second command based upon sequence information in a header of the first command and sequence information in a header of the second command. For instance, the host device may include a same sequence identifier and different sequence numbers in the respective headers of the first command and the second command. Thus, the processor may determine that the first command and the second command are part of the same sequence, and may also determine the relative order to execute the first command and the second command. In one example, the commands may be received out of order, and may therefore be rearranged by the processor at block 520 based upon the sequence numbers. For illustrative purposes, it may be determined that the first command is first in the sequence, while the second command is second in the sequence.

[0061] In block 530, the processor may receive a third command from the host device via a second nexus for non-ordered commands. In one example, the second nexus may comprise one of a plurality of nexuses between the host device and a non-volatile memory subsystem of the processor that are available for transporting non-ordered commands and responses thereto.

[0062] In block 540, the processor may execute the first command, the second command, and the third command, the first command and the second command being executed in the sequence that is determined at block 520. The commands may be selected by the processor for performance in different orders, so long as the first command and the second command remain back-to-back and are executed in the order determined at block 520. For example, the third command may be executed, followed the first command and the second command, or the first command may be executed and the second command may be executed, followed by the third command. Following block 540, the method 500 proceeds to block 595 where the method ends.

[0063] FIG. 6 illustrates a flowchart of an example method 600 for processing commands at a non-volatile memory subsystem controller. The method 600 may be performed, by at least one component of the system 100 of FIG. 1 or system 200 of FIG. 2. For example, the method 600 may be performed by the memory subsystem 130 or the memory controller 135 of FIG. 1 , or the memory subsystem 230 or the memory controller 235 of FIG. 2. However, the method 600 is not limited to implementation with the system and devices illustrated in FIGs. 1 and 2, but may be applied in connection with any number of systems and devices for processing commands at a non-volatile memory subsystem controller. Alternatively, or in addition, at least one of the blocks of the method 600 may be implemented by a computing device having a processor, a memory, and I/O devices as illustrated below in FIG. 7, specifically programmed to perform the blocks of the method. Although any one of the elements in system 100, system 200, or in a similar system, may be configured to perform various blocks of the method 600, the method will now be described in terms of an example where blocks of the method are performed by a processor, such as processor 702 in FIG. 7. For instance, processor 702 may represent a non-volatile memory subsystem controller as described in FIGs. 1 and 2 above. As used in connection with the description of FIG. 6, the term "processor" may also include multiple processors, hardware state machines, or hardware logic units, e.g., an application specific integrated circuit (ASIC), a programmable logic device (PLD), such as a field programmable gate array (FPGA), and so forth.

[0064] The method 600 begins in block 605 and proceeds to block 610. In block 610, the processor may receive a first command and a second command from a host device via a first nexus for ordered commands. In this regard, the first command and the second command may both comprise ordered commands for interacting with a non-volatile memory of a non-volatile memory subsystem that is associated with the processor. In addition, the first command and the second command may be paired together, or may comprise commands that are part of a same sequence comprising more than two commands. In one example, the processor may receive the first command and the second command from a first I/O submission queue for ordered commands of the host device via the first nexus. In one example, block 610 may comprise the same or similar operations as described above in connection with block 510 of the method 500.

[0065] In block 620, the processor may determine a sequence for executing the first command and the second command based upon sequence information in a header of the first command and sequence information in a header of the second command. In one example, block 620 may comprise the same or similar operations as described above in connection with block 520 of the method 500.

[0066] In block 630, the processor may receive a third command from the host device via a second nexus for non-ordered commands. In one example, the processor may receive the third command from a second I/O submission queue for non-ordered commands of the host device via the second nexus. In one example, block 630 may comprise the same or similar operations as described above in connection with block 530 of the method 500.

[0067] In block 640, the processor may receive a fourth command from the host device via a third nexus for non-ordered commands. For instance, the second and third nexuses may be included in a plurality of nexuses for transmitting non-ordered commands and responses thereto between the host device and a non-volatile memory subsystem associated with the processor. In addition, the second nexus and the third nexus may have been selected by the host device for transmitting the third command and the fourth command, respectively, according to a load balancing or load distribution arrangement. It should be noted that in some examples, block 640 may be omitted from the performance of the method 600.

[0068] In block 650, the processor may place commands in input/output (I/O) submission queues. For example, the non-volatile memory subsystem may include a controller memory buffer for use by the processor. The controller memory buffer may include at least one I/O submission queue for ordered commands and at least one I/O submission queue for non-ordered commands. Thus, at block 650, the processor may write the first command and the second command to a first I/O submission queue for ordered commands, and write the third command to a second I/O submission queue for non-ordered commands. In addition, in one example, the processor may write the fourth command to the same I/O submission queue as the third command, i.e., the second input/output submission queue. However, in another example, the processor may write the fourth command to a different I/O submission queue, e.g., a third I/O submission queue for non-ordered commands.

[0069] In block 660, the processor may determine an order to execute the first command, the second command, the third command, and the fourth command. The commands may be selected from the respective I/O submission queues according to any load balancing and/or load distribution algorithm, and may arranged in any order, so long as the first command and the second command remain in contiguous positions in the order and are arranged according to the sequence that is determined for the first command and the second command at block 620.

[0070] In block 670, the processor may execute the first command, the second command, and the third command in the order that is determined at block 660. In one example, the fourth command may also be executed in the order that is determined at block 660 (e.g., in examples where the method 600 is performed to include block 640). In one example, block 670 may comprise the same or similar operations as described above in connection with block 540 of the method 500.

[0071] In block 680, the processor may report a completion of the first command and a completion of the second command to a first I/O completion queue for the host device. In one example, the first I/O completion queue may be for responses associated with the execution of ordered commands. In one example, the first I/O completion queue may be stored in a controller memory buffer of the non-volatile memory subsystem associated with the processor. In another example, the first I/O completion queue may be stored in a host memory buffer of the host device. In this regard, the reporting of the completion of the first command and the second command may comprise transmitting responses over the first nexus that is reserved for ordered commands and responses thereto.

[0072] In block 690, the processor may report a completion of the third command to a second I/O completion queue for the host device. In one example, the second I/O completion queue may be for responses associated with the execution of non-ordered commands. In one example, the second I/O completion queue may be stored in a controller memory buffer of the non-volatile memory subsystem associated with the processor. In another example, the second I/O completion queue may be stored in a host memory buffer of the host device. In this regard, the reporting of the completion of the third command may comprise transmitting a response over the second nexus for non-ordered commands. However, in another example, any other nexus of a plurality of nexuses between the host device and the memory subsystem for non-ordered commands and responses thereto may be used to report the completion of the third command. Following block 690, the method 600 proceeds to block 695 where the method ends.

[0073] It should be noted that the method 600 may be modified or expanded in various examples, and that any one or more blocks of the method 600 may be omitted. For instance, as mentioned above, block 640 may be omitted, in which case blocks 660 and 670 may omit aspects relating to a fourth command. In another example, where block 640 is performed, the method 600 may be expanded to include the processor reporting a completion of the fourth command. In still another example, additional ordered commands in the same sequence as the first command and the second command may be received, scheduled for execution by the processor according to the sequence, and executed in the sequence/order that is determined. [0074] In addition, although not explicitly specified, at least one of the blocks, functions, or operations of the methods 300, 400, 500, and 600 described above may include storing, displaying, and/or outputting. In other words, any data, records, fields, and/or intermediate results discussed in the methods can be stored, displayed, and/or outputted to another device depending on the particular application. Furthermore, blocks, functions, or operations in FIGs. 3-6 that recite a determining operation, or involve a decision, do not necessarily imply that both branches of the determining operation are practiced. In other words, one of the branches of the determining operation can be deemed as optional.

[0075] FIG. 7 depicts an example high-level block diagram of a computing device suitable for use in performing the functions described herein. As depicted in FIG. 7, the computer 700 comprises a hardware processor element 702, e.g., a central processing unit (CPU), a microprocessor, or a multi-core processor, a memory 704, e.g., random access memory (RAM), a module 705 for transmitting commands to a non-volatile memory subsystem controller or for processing commands at a non-volatile memory subsystem controller, and various I/O devices 706, e.g., storage devices, including but not limited to, a tape drive, a floppy drive, a hard disk drive or a compact disk drive, a receiver, a transmitter, a speaker, a display, a speech synthesizer, an output port, an input port and a user input device, such as a keyboard, a keypad, a mouse, a microphone, and the like. Although one processor element is shown, it should be noted that the computer may employ a plurality of processor elements. Furthermore, although one computer is shown in the figure, if the method(s) as discussed above is implemented in a distributed or parallel manner for a particular illustrative example, i.e., the blocks of the above method(s) or the entire method(s) are implemented across multiple or parallel computers, then the computer of this figure is intended to represent each of those multiple computers.

[0076] It should be noted that the present disclosure can be implemented by machine readable instructions and/or in a combination of machine readable instructions and hardware, e.g., using application specific integrated circuits (ASIC), a programmable logic array (PLA), including a field-programmable gate array (FPGA), or a state machine deployed on a hardware device, a computer or any other hardware equivalents, e.g., computer readable instructions pertaining to the method(s) discussed above can be used to configure a hardware processor to perform the blocks, functions and/or operations of the above disclosed methods.

[0077] In one example, instructions and data for the present module or process 705 for transmitting commands to a non-volatile memory subsystem controller or for processing commands at a non-volatile memory subsystem controller, e.g., machine readable instructions, can be loaded into memory 704 and executed by hardware processor element 702 to implement the blocks, functions, or operations as discussed above in connection with the example methods 300-600. For instance, the module 705 may include a plurality of computer-readable components, including a set a first indicator component 710, a set a second indicator component 71 1 , a transmit first and second commands component 712, a select a second nexus component 713, a transmit a third command component 714, and an additional functions component 719. When executed by the hardware processor element 702 the set a first indicator component 710 may cause the hardware processor element 702 to set a first indicator in a first command to indicate that the first command is an ordered command, the set a second indicator component 71 1 may cause the hardware processor element 702 to set a second indicator in a second command to indicate that the second command is an ordered command that is associated with the first command, the transmit first and second commands component 712 may cause the hardware processor element 702 to transmit the first command and the second command to a non-volatile memory subsystem controller via a first nexus that is assigned for ordered commands, the select a second nexus component 713 may cause the hardware processor element 702 to select a second nexus of a plurality of nexuses that are assigned for non- ordered commands, and the transmit a third command component 714 may cause the hardware processor element 702 to transmit a third command comprising a non-ordered command to the non-volatile memory subsystem controller via the second nexus, e.g., as per the method 300 above. When executed by the hardware processor element 702, the additional functions component 719 may cause the hardware processor element 702 to perform additional functions, e.g., in connection with the method 400 as described above, and/or to perform any one or more additional functions as described in the present disclosure. The foregoing is just one example configuration of module 705 in accordance with the present disclosure.

[0078] For instance, in another example, instructions and data for the present module or process 705 for transmitting commands to a non-volatile memory subsystem controller or for processing commands at a non-volatile memory subsystem controller, e.g., machine readable instructions, can be loaded into memory 704 and executed by hardware processor element 702 to implement the blocks, functions, or operations as discussed above in connection with the example method 500. For instance, the module 705 may include a plurality of computer- readable components, including a receive first and second commands component 715, a determine a sequence component 716, a receive third command component 717, an execute commands component 718, and an additional functions component 719. When executed by the hardware processor element 702 the receive first and second commands component 715 may cause the hardware processor element 702 to receive a first command and a second command from a host device via a first nexus for ordered commands, the determine a sequence component 716 may cause the hardware processor element 702 to determine a sequence for executing the first command and the second command based upon sequence information in a header of the first command and sequence information in a header of the second command, the receive third command component 717 may cause the hardware processor element 702 to receive a third command from the host device via a second nexus for non-ordered commands, and the execute commands component 718 may cause the hardware processor element 702 to execute the first command, the second command, and the third command, the first command and the second command being executed in the sequence for executing the first command and the second command that is determined, e.g., as per the method 500 above. When executed by the hardware processor element 702, the additional functions component 719 may cause the hardware processor element 702 to perform additional functions, e.g., in connection with the method 600 as described above, and/or to perform any one or more additional functions as described in the present disclosure.

[0079] Furthermore, when a hardware processor executes instructions to perform "operations," this could include the hardware processor performing the operations directly and/or facilitating, directing, or cooperating with another hardware device or component, e.g., a co-processor and the like, to perform the operations.

[0080] The processor executing the machine readable instructions relating to the above described method(s) can be perceived as a programmed processor or a specialized processor. As such, the present module 705 for transmitting commands to a non-volatile memory subsystem controller or for processing commands at a non-volatile memory subsystem controller, including associated data structures, of the present disclosure can be stored on a tangible or physical (broadly non-transitory) computer-readable storage device or medium, e.g., volatile memory, non-volatile memory, ROM memory, RAM memory, magnetic or optical drive, device or diskette and the like. Furthermore, the computer-readable storage device may comprise any physical device or devices that provide the ability to store information such as data and/or instructions to be accessed by a processor or a computing device such as a computer or an application server.

[0081] It will be appreciated that variants of the above-disclosed and other features and functions, or alternatives thereof, may be combined into many other different systems or applications. Various presently unforeseen or unanticipated alternatives, modifications, or variations therein may be subsequently made, which are also intended to be encompassed by the following claims.

Claims

What is claimed is:

1 . A non-volatile memory subsystem controller to:

receive a first command and a second command from a host device via a first nexus for ordered commands;

determine a sequence for executing the first command and the second command based upon sequence information in a header of the first command and sequence information in a header of the second command;

receive a third command from the host device via a second nexus for non- ordered commands; and

execute the first command, the second command, and the third command, the first command and the second command being executed in the sequence for executing the first command and the second command that is determined.

2. The non-volatile memory subsystem controller of claim 1 , wherein the first command and the second command are received from a first input/output submission queue for ordered commands of the host device via the first nexus, wherein the third command is received from a second input/output submission queue for non-ordered commands of the host device via the second nexus.

3. The non-volatile memory subsystem controller of claim 1 , the non-volatile memory subsystem controller further to:

report a completion of the first command and a completion of the second command to a first input/output completion queue for the host device; and

report a completion of the third command to a second input/output completion queue for the host device.

4. The non-volatile memory subsystem controller of claim 3, wherein the first input/output completion queue and the second input/output completion queue are stored in a controller memory buffer of the non-volatile memory subsystem controller.

5. The non-volatile memory subsystem controller of claim 3, wherein the first input/output completion queue and the second input/output completion queue are stored in a host memory buffer of the host device, the non-volatile memory subsystem controller further to:

report the completion of the first command and the completion of the second command to the first input/output completion queue via the first nexus; and

report the completion of the third command to the second input/output completion queue via the second nexus.

6. The non-volatile memory subsystem controller of claim 1 , the non-volatile memory subsystem controller further to:

receive a fourth command from the host device via a third nexus for non- ordered commands; and

determine an order for executing the first command, the second command, the third command, and the fourth command, the first command and the second command being assigned contiguous positions in the order.

7. The non-volatile memory subsystem controller of claim 6, the non-volatile memory subsystem controller further to:

write the first command and the second command to a first input/output submission queue for ordered commands;

write the third command to a second input/output submission queue for non- ordered commands; and

write the fourth command to the second input/output submission queue or to a third input/output submission queue for non-ordered commands.

8. A system, comprising:

a host device, the host device comprising at least one port; a non-volatile memory subsystem, comprising:

at least two ports;

a non-volatile memory subsystem controller assigned to the host device; and

a non-volatile memory; and

a first nexus, the first nexus comprising a pair of one of the at least one port of the host device and one of the at least two ports of the non-volatile memory subsystem, the host device to assign the first nexus for ordered commands, the host device to assign at least a second nexus for non-ordered commands.

9. The system of claim 8, wherein the first nexus and the second nexus comprise different paths in a connectionless datagram fabric.

10. The system of claim 8, further comprising:

a second non-volatile memory subsystem controller, the second nonvolatile memory subsystem controller assigned to a second host device, the second non-volatile memory subsystem controller to receive commands from the second host device via the at least two ports, the non-volatile memory including a first namespace that is assigned to the non-volatile memory subsystem controller and to the host device and a second namespace that is assigned to the second non-volatile memory subsystem controller and to the second host device.

1 1 . A host device to access a non-volatile memory subsystem controller, the host device comprising:

a processor; and

a non-transitory computer-readable medium storing instructions which, when executed by the processor, cause the processor to:

set a first indicator in a first command to indicate that the first command is an ordered command;

set a second indicator in a second command to indicate that the second command is an ordered command that is associated with the first command;

transmit the first command and the second command to the nonvolatile memory subsystem controller via a first nexus that is assigned for ordered commands;

select a second nexus of a plurality of nexuses that are assigned for non-ordered commands; and

transmit a third command comprising a non-ordered command to the non-volatile memory subsystem controller via the second nexus.

12. The host device of claim 1 1 , wherein the instructions, when executed by the processor, further cause the processor to:

assign the first nexus for ordered commands; and

assign the second nexus of the plurality of nexuses for non-ordered commands.

13. The host device of claim 1 1 , wherein the instructions, when executed by the processor, further cause the processor to:

place the first command and the second command in a first input/output submission queue for ordered commands; and

place the third command in a second input/output submission queue for non-ordered commands.

14. The host device of claim 1 1 , wherein the instructions, when executed by the processor, further cause the processor to:

reassign the second nexus to ordered commands in response to an impairment of the first nexus.

15. The host device of claim 1 1 , wherein the first nexus comprises a first port of the host device and a first port of the non-volatile memory subsystem, and wherein the second nexus comprises one of:

the first port of the host device and a second port of the non-volatile memory subsystem;

a second port of the host device and the first port of the non-volatile memory subsystem; or

the second port of the host device and the second port of the non-volatile memory subsystem.