WO2016122514A1

WO2016122514A1 - Communications over a serial attached small computer interface (sas)

Info

Publication number: WO2016122514A1
Application number: PCT/US2015/013451
Authority: WO
Inventors: James Michael REUTER; Matt GATES
Original assignee: Hewlett Packard Enterprise Development LP
Current assignee: Hewlett Packard Enterprise Development LP
Priority date: 2015-01-29
Filing date: 2015-01-29
Publication date: 2016-08-04
Anticipated expiration: 2017-07-29

Abstract

In one example, disclosed is a system for communicating over a serial attached small computer system interface (SAS). The system includes a processor and a storage system, wherein the storage system comprises instructions configured to direct the processor to send a command data block from an initiator to a target over a SAS link. The command data block comprises a pull request instructing the target to send data to the initiator while the initiator is sending data to the target.

Description

COMMUNICATIONS OVER A

SERIAL ATTACHED SMALL COMPUTER INTERFACE (SAS)

BACKGROUND

[0001] Some storage array controllers achieve high availability by communicating and mirroring data, such as cached data, to a peer controller. In modern products, this communication is sometimes done using serial attached small computer system interface (SAS) interfaces that are already present for communication to the back end storage. SAS connection semantics cause the links to perform like half duplex links, which loses potential performance.

DESCRIPTION OF THE DRAWINGS

[0002] Certain exemplary embodiments are described in the following detailed description and in reference to the drawings, in which:

[0003] Fig. 1 is a block diagram of an example of a computing system using serial attached small computer system interface (SAS);

[0004] Fig. 2 is a block diagram of a protocol stack for SAS links;

[0005] Fig. 3 is a ladder diagram of an example of a communications protocol between an initiator and a target that may achieve full duplex communications; and

[0006] Fig. 4 is a non-transitory machine readable medium that includes code to direct a processor to carry out full duplex communications over a SAS link.

DETAILED DESCRIPTION

[0007] Examples described herein provide higher speed communications between units over a serial attached small computer system interface (SAS) link, for example, approaching full duplex communications. A SAS interface provides full duplex capability at the physical and link layers. However, the connection semantics for the higher level small computer system interface (SCSI) read and write

operations will typically result in half duplex performance. [0008] This is because connections are very short lived states between a target and an initiator for the purpose of transferring a request or data. Because requests are reads or writes, the data flows one way for that request. Because connections are exclusive, each end point and the path between those points effectively can transfer data one way at a time. The only time full duplex performance is achieved is when a connection between an initiator and a target has simultaneous outstanding reads and writes and is ready to transfer data both ways when a connection is established. Otherwise the bi-directional physical links are underutilized during connections, because data is only flowing one way and therefore one of the twisted pairs is idle.

[0009] Fig. 1 is a block diagram of an example of a computing system 100 using serial attached small computer system interface (SAS). However, the techniques are not limited to this configuration for the computing system 100, as any number of configurations can be used. For example, a large computing system 100 may often have many more hosts and disk arrays than shown in this illustration. The enterprise storage system 100 may be accessed from one or more host computers, such as host 1 102 and host 2 104. The host computers 102 and 104 may provide data, such as Web pages, database screens, and other services, to one or more client computers, such as client 1 106, client 2 108, and client 3 1 10, for example, over a network 1 12. The network 1 1 2 may be a local area network (LAN), wide area network (WAN), a storage area network (SAN), or other network, such as the Internet. The host computers 102 and 104 may each be coupled to a storage array controller, such as array controller 1 1 14 and array controller 2 1 16 for example, by one or more Fibre Channel (FC) links, such as FC links 1 18 and 120. Other links, such as PCIe, SAS, Ethernet, and other links, may be used to communicate between the host computers 1 02 and 104 and the array controllers 1 14 and 1 16.

[0010] The array controllers 1 14 and 1 16 may be coupled through SAS links, such as SAS link 122 and SAS link 1 24 to SAS expanders, such as SAS expander 1 126 and SAS expander 2 128. The SAS expanders 1 26 and 1 28 may link to disk arrays, for example, disk array 1 130, disk array 2 132, and disk array 3 134 through SAS links 136-142. Further, the SAS expanders 126 and 128 may be linked together through another SAS link, termed an intercontroller link or ICL 144 to allow communications between the array controllers 1 14 and 1 16. The ICL 144 may be through expanders, as shown, or may be directly between array controllers, for example, if the array controllers 1 14 and 1 16 have multiple SAS ports 145.

[0011 ] Each of the array controllers 1 14 and 1 16 may have a number of components to carry out the operations. For example a processor 146, memory 148 (such as RAM, nonvolatile RAM, and the like), and a storage unit 1 50 (such as a hard drive, optical drive, hard wired logic, and the like). Although the array controllers 1 14 and 1 16 are illustrated as computer systems, it may be noted that they could encompass any number of configurations, from the systems shown to fully hardwired logic controllers, and any number of combinations between. The memory 148 may hold a cache 152 that temporarily stores information being transferred between the host 102 or 104 and the disk arrays 130, 132, or 134. The storage unit 150, such as a flash memory, a programmable read only memory, a hard drive, and the like, may include instruction or code blocks to direct the processor 146 to carry out functions. These code blocks may include, for example, an O/S block 1 54 holding operational and boot up code, a SCSI stack 156, and a SCSI application 158.

[0012] In some array controllers, the traffic over the ICL 144 works by using SCSI operations that transfer data the same way as SCSI WRITE commands. Data transfers include metadata synchronization and mirroring of the cache 152 between array controllers 1 14 and 1 16. The cache mirroring can be high bandwidth traffic when the controller is serving high bandwidth write requests from its host. As described herein, array controllers that use inter-controller traffic often are limited to half duplex communications due to the nature of the link, which is dedicated to each communication after initiation, e.g., not allowing other communications to share the link before completion of the communication. This limits the performance of high bandwidth write traffic because the cache mirroring traffic and the disk I/O traffic are all flowing over the same links which causes the back-end SAS to saturate before other bottlenecks are encountered.

[0013] When data needs to be transferred to the peer controller the initiating controller issues a write command, which causes the SAS engines to make connections to issue the request, transfer the data to the target, and then transfer the completion response. The payload data always flows from the initiator to the target. If the peer controller wants to send data the other way, for example, when both controllers are handling loads with write traffic, it does the same thing, by issuing its own write requests which transfer the data over different connections. This never provides full-duplex behavior because the connections are exclusive in SAS, and initiator and target roles cannot be mixed within one connection.

[0014] Examples described herein may help overcome this issue by including a full duplex communications block 160 in the storage system 150. The full duplex communications block 160 provides a command sequence that can enable full duplex communications between the array controllers 1 14 and 1 16, for example, through the ICL 144 between the SAS expanders 1 26 and 128. The full duplex communications can be used for efficient synchronization of the contents of the buffer 152, among other things.

[0015] Fig. 2 is a block diagram of an example of a protocol stack 200 for SAS links. The protocol stack is present in each device using a SAS link, such as the array controllers 1 14 and 1 16, and SAS expanders 126 and 128 discussed with respect to Fig. 1 . This simplified overview of the protocol stack 200 is presented to place examples described herein in the proper context.

[0016] A physical layer 202 includes the driver circuitry to power the differential signals transmitted between devices. It also includes the wires, connectors, and other physical hardware that couples the devices together.

[0017] A PHY layer 204 handles the link initialization, bit rate negotiations with a connected device, as well as error encoding and error identification. The PHY layer 204 also adds control characters to assist in bit alignment of a bit stream.

[0018] A link layer 206 defines primitives, e.g., communication control symbols, adds or removes address frames, and handles connections. The link layer 206 may also use a link layer state machine to interface with a port layer 208 and the PHY layer 204 to handle identification and reset sequences. The link layer 206 also handles specific frame transmission and reception.

[0019] The port layer 208 includes a number of state machines running in parallel that interface between the state machines in the link layer 206 and state machines in a transport layer 210, such as a serial SAS Protocol (SSP) 212, a serial tunneling AT attachment (ATA) protocol (STP) 214, and a serial management protocol (SMP) 216 to manage port connections. The port layer 208 receives requests from the state machines in the transport layer 21 0 and from the management application layer 218 for connection management and frame transmission. The port layer 210 also sends requests to the link layer 206 for connection management and frame transmission.

[0020] The transport layer 210 contains state machines 21 2, 214, and 216 that interface between the port layer 208 and the management application layer 218. The transport layer 210 defines frame format, and construct and parse frame contents.

[0021] The management application layer 218 includes three functions that interface with respective state machines in the transport layer, a SCSI application layer 220, an ATA application layer 222, and a management application layer 224. The SCSI application layer 220 requests the processing of a SCSI command by invoking the SCSI transport protocol services.

[0022] Although most of the layers 202, 204, 206, 208, and 210 are capable of full duplex operations, when data is ready for transmission, the SCSI application layer 220 generally initiates a link with another unit for the transmission. However, the link is exclusive during the transmission. Accordingly, the link functions as a half duplex link. In examples described herein, a full duplex command module 226 is included as part of the SCSI application layer 220 to coordinate the commands that may be used to establish a full duplex link between SAS devices, as described further with respect to Fig. 3.

[0023] Fig. 3 is a ladder diagram of an example of a communications protocol 300 between an initiator 302 and a target 304 that may achieve full duplex

communications. Currently, the only way to get full duplex performance over SAS is when an initiator has outstanding read and write requests to the same target. In examples described herein, the communications protocol 300 includes the read type requests, for example, for cache mirroring over an ICL. The request uses a new type of "pull" request 306 that allows a controller that wants to send data to instruct the other side to issue a read for data to be transmitted from the target 304 to the initiator 302. [0024] The protocol begins when the initiator 302 sends 308 the target 304 a command data block (CDB) with a pull type request 306. The target 304 responds 310 with a transmit ready (XRDY) for scatter gather list (SGL) phase 312. The initiator 304 responds 314 by sending the SGL data, which may include metadata describing a list of memory buffers to be transferred in a memory copy operation.

[0025] The target 304 processes 318 the SGL data, then, acting as a second initiator, sends 320 a new CDB with the pull data 322. It can be noted that if the target 304 does not have data to share, the communication progresses as a normal SAS communication. Further, the amounts of data do not have to be the same, e.g., either the target or the initiator may send larger amounts of data than the

corresponding initiator or target. The initiator 302 responds 324 by sending the data 326 that was initially intended for transmission. Once the target 304 has

successfully received the data, it responds 328 with a completion message 330 that answers the initial CDB message 306, closing the communications.

[0026] In this way a controller, such as the initiator 302 or target 304, can issue writes to transfer data and also tell the other controller to issue reads. If both sides are doing this, they can then achieve the case where an initiating controller 302 has outstanding reads and writes to the target controller 304, resulting in full duplex performance. The special CDB with a pull request 306 that instructs the other node to read also avoids the issue of having to pre-allocate buffers for a request of unknown size. If the situation with concurrent reads and writes happens frequently enough, the performance of the SAS links used for ICL traffic can exceed half duplex performance.

[0027] Fig. 4 is a non-transitory machine readable medium 400 that includes instructions or code to direct a processor 402 to carry out full duplex communications over a SAS link. The processor may be coupled to the machine readable medium 400 over a bus 404, such as a PCIe bus, an interchip interconnection, or a proprietary bus, among others. The machine readable media can include read only memory (ROM), programmable read only memory (PROM), and similar types of ROM. Further, the machine readable memory may include a hard drive, an optical drive, a solid state disk drive, and other storage systems. The machine readable media may also include random access memory, such as dynamic RAM (DRAM), static RAM (SRAM) and the like. In some examples, the machine readable medium 400 may be hardwired logic that is integrated with the processor 402.

[0028] The machine readable medium 400 includes instructions or code or logic 406 to direct the processor 402 to implement commands that may achieve full duplex communications. This may be done, for example, through the use of the protocol described with respect to Fig. 3.

[0029] While the present techniques may be susceptible to various modifications and alternative forms, the exemplary examples discussed above have been shown only by way of example. It is to be understood that the technique is not intended to be limited to the particular examples disclosed herein. Indeed, the present techniques include all alternatives, modifications, and equivalents falling within the scope of the present techniques.

Claims

CLAIMS What is claimed is:

1 . A system for communicating over a serial attached small computer system interface (SAS), comprising:

a processor; and

a storage system, wherein the storage system comprises instructions

configured to direct the processor to send a command data block from an initiator to a target over a SAS link, wherein the command data block comprises a pull request instructing the target to send data to the initiator while the initiator is sending data to the target.

2. The system of claim 1 , wherein the storage system comprises instructions to direct the processor to, in response to receiving a symbol indicating that the target is ready for SGL phase, send SGL data from the initiator to the target.

3. The system of claim 1 , wherein the storage system comprises instructions to direct the processor to, in response to receiving a command data block comprising a pull request from the target, sending a command data block comprising pull data.

4. The system of claim 1 , wherein the storage system comprises instructions to direct the processor, in response to processing SGL data from an initiator, sending a command data block comprising pull data.

5. The system of claim 1 , wherein the storage system comprises hard wired logic.

6. The system of claim 1 , comprising a storage array controller, wherein the storage system comprises instructions to direct a processor to synchronize a cache or metadata, or both, with a second storage array controller over an SAS intercontroller link (ICL).

7. The system of claim 6, comprising at least two SAS expanders, wherein each of the SAS expanders has an ICL to another SAS expander.

8. The system of claim 3, comprising a storage array.

9. A method for communicating between storage array controllers over an intercontroller interlink, comprising:

sending a command data block with a request for pull data from an initiator controller to a target controller;

receiving from the target controller an indication that the target controller is ready for an SGL phase;

sending SGL data to the target controller for processing;

receiving from the target controller a command data block comprising the pull data;

sending a data block to the target controller; and

receiving a completion symbol from the target controller indicating that the communications are complete.

10. The method of claim 9, comprising receiving pull data from the target controller while sending the data block to the target controller.

1 1 . The method of claim 9, comprising sending the data block to the target controller without receiving pull data from the target controller.

12. The method of claim 9, comprising sending commands from the initiator controller to the target controller by through a serial attached small computer system interface (SAS) storage expander coupled with the initiator controller.

13. The method of claim 9, comprising synchronizing a cache in the initiator controller with a corresponding cache in the target controller.

14. A non-transitory machine readable medium comprising instructions to direct a processor to send a command data block with a request for pull data from an initiator to a target over a serial attached small computer system interface (SAS).

15. The non-transitory machine readable medium of claim 14, comprising instructions to direct a processor to:

receive from the target controller an indication that the target controller is ready for an SGL phase;

send SGL data to the target controller for processing;

receive from the target controller a command data block comprising the pull data;

send a data block to the target controller; and

receive a completion symbol from the target controller indicating that the

communications are complete.