WO2016182756A1

WO2016182756A1 - Accessing multiple storage devices from multiple hosts without remote direct memory access (rdma)

Info

Publication number: WO2016182756A1
Application number: PCT/US2016/029856
Authority: WO
Inventors: James R. Bergsten; Lawrence W. Lomelino; Christopher Christ; Steven R. Lahr
Original assignee: Apeiron Data Systems
Priority date: 2015-05-14
Filing date: 2016-04-28
Publication date: 2016-11-17
Also published as: EP3295321A4; EP3295321A1

Abstract

Various embodiments for implementing a method and apparatus for accessing multiple storage devices from multiple hosts without use of remote direct memory access (RDMA) as disclosed herein include: providing a data store switch fabric enabling data communications between a data storage access system and a plurality of compute nodes, each compute node having integrated compute capabilities, data storage, and a network interface controller (Host NIC); providing a plurality of physical data storage devices; providing a host bus adapter (HBA) in data communication with the plurality of physical data storage devices and the plurality of compute nodes via the data store switch fabric, the HBA including at least one submission queue and a corresponding shadow queue; receiving an input/output (I/O) request from the plurality of compute nodes; including an element of the I/O request to the at least one submission queue; and including additional information related to the element of the at least one submission queue to the corresponding shadow queue.

Description

APPLICATION FOR UTILITY PATENT

FOR

METHOD AND APPARATUS FOR ACCESSING MULTIPLE STORAGE DEVICES FROM MULTIPLE HOSTS WITHOUT USE OF REMOTE DIRECT MEMORY ACCESS (RDMA)

METHOD AND APPARATUS FOR

ACCESSING MULTIPLE STORAGE DEVICES FROM MULTIPLE HOSTS WITHOUT USE OF REMOTE DIRECT MEMORY ACCESS (RDMA)

PRIORITY PATENT APPLICATIONS

[0001] This application is a continuation-in-part (CIP) patent application drawing priority from non-provisional U.S. patent application no. 14/254,868; filed April 16, 2014; which draws priority from U.S. provisional patent application no. 61/812,916; filed April 17, 2013, and U.S. provisional patent application no. 61/812,927; filed April 17, 2013. The entire disclosure of the referenced patent applications is considered part of the disclosure of the present application and is hereby incorporated by reference herein in its entirety.

COPYRIGHT

[0002] A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever. The following notice applies to the software and data as described below and in the drawings that form a part of this document: Copyright 2013-2015 Apeiron Data Systems, Inc., All Rights Reserved.

TECHNICAL FIELD

[0003] The disclosed subject matter relates to the field of data access storage methods and systems.

BACKGROUND

[0004] The amount of data in our world has been exploding. All this data need to be stored and analyzed to extract value. The fundamental requirements for data storage and analysis to meet the rapid growth in data rates include:

1. Capacity - Seamlessly store and analyze peta-bytes of data;

2. Scalability - Add more compute and storage capacities as data storage requirements grow;

3. Accessibility - Maintain continuous access to stored data in the presence of hardware failures;

4. Performance - Increase performance as more resources are added incrementally; and

5. Cost - Maintain low total cost of ownership. [0005] However, conventional data storage architectures do not provide an efficient solution that addresses all of these requirements without any trade-offs. Additionally, current data storage architectures cannot provide access storage in a shared environment with a minimum of protocol overhead.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The various embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which:

[0007] Figures 1 through 3 illustrate a Clustered Direct Attach Storage (Clustered DAS) configuration of conventional systems;

[0008] Figures 4 and 5 illustrate a Network Attached Storage (NAS) or Storage Area

Network (SAN) configuration of conventional systems;

[0009] Figure 6 illustrates an example embodiment of a switched direct attached shared storage architecture;

[0010] Figure 7 illustrates the physical storage media assignment to compute nodes in an example embodiment;

1001 11 Figure 8 illustrates how each storage slice is the physical unit of abstraction that can be plugged into a storage media container in an example embodiment;

|00121 Figure 9 illustrates a procedure for assigning storage slices to compute nodes with

NVMe (non-volatile memory express) storage;

[0013] Figure 10 illustrates the process in an example embodiment for device management;

[0014] Figure 1 1 illustrates the procedure in an example embodiment for data flow from a compute node to one or more storage slices;

[0015] Figure 12 illustrates the procedure in an example embodiment for storage slice sharing:

[0016] Figure 13 illustrates the data flow in a sw itched DAS architecture of an example embodiment using Ethernet as the transport fabric protocol;

[0017] Figure 14 illustrates the encapsulation of an 10 operation into a standard Ethernet frame in an example embodiment;

[0018] Figures 15 and 16 illustrate an example embodiment for implementing instrumentation hooks to monitor, measure, and enforce performance metrics into the compute, memory, network and storage resources;

[0019] Figures 1 7 and 18 illustrate an example embodiment for continuous monitorin of the health of all resources to predict failures and proactively adjust/update the cluster resources; [0020] Figure 19 illustrates the standard NVM Express 1. 1 specification wherein an example embodi ment implements input output ( 10) acceleration by use of an Ethernet connection; 100211 Figure 20 illustrates a server to server configuration of the messaging protocol of an example embodiment;

[0022] Figure 2 1 illustrates the data flow for a sample message using the messaging protocol of an example embodiment;

[0023] Figure 22 show s the basic organization of the current flash media;

[0024] Figure 23 illustrates the object tag format for the object store of the example embodi ment;

[0025] Figure 24 shows a specific example of the conventional system shown in Figure 4. where storage is attached via Ethernet using conventional protocols;

[0026] Figure 25 illustrates how NVM Express devices are accessed when locally installed in a server;

[0027] Figure 26 illustrates a typical RDM A hardware and softw are stack required to implement remote access of NVM Express devices;

[0028] Figure 27 illustrates an embodiment of the data storage access system of the example embodiments described herein showing the savings in complexity to be gained by use of the example embodiments over the conventional implementation shown in Figure 26;

[0029] Figure 28 illustrates the configuration of queues in the host bus adapter (HBA) or host netw ork interface controller (NIC) in an example embodiment;

[0030] Figure 29 illustrates a detail of the configuration of queues in the host bus adapter

( HBA) or host network interface controller (NIC) in an example embodiment;

100311 Figure 30 illustrates an architectural view of the storage controller of an example embodiment in netw ork communication with a plurality of host server systems via a storage netw ork;

[0032] Figure 31 illustrates an example of a method for a host/server to communicate I/O requests to devices installed within the data storage access system of an example embodiment;

[0033] Figure 32 illustrates example contents of a single Shadow Queue Element in the data storage access system of an example embodiment;

[0034] Figures 33 and 34 illustrate example register sets of the data storage access system of an example embodiment used to set up and control the various request and completion queues as described herein;

100351 Figures 35 and 36 illustrate examples of how a host 1 0 request flow s through the data storage access system of an example embodiment; [0036] Figure 37 illustrates a node to node protocol in an example embodiment providin the ability for a plurality of data storage access systems to inter-communicate via unicast. multicast, or broadcast data transmissions using the queues described herein;

[0037] Figure 38 illustrates an example embodiment of a component of the data storage access system of an example embodiment as used within an existing host/server;

[0038] Figure 39 is a flow diagram illustrating the basic processing flow for a particular example embodiment of the data storage access system as described herein; and

[0039] Figure 40 shows a diagrammatic representation of a machine in the example form of a data processor within which a set of instructions, for causing the machine to perform any one or more of the methodologies described herein, may be executed.

DETAILED DESCRIPTION

[0040] In the following detailed description, a reference is made to the accompanying drawings that form a part hereof, and in which are shown, by way of illustration, specific embodiments in which the disclosed subject matter can be practiced. It is understood that other embodiments may be utilized and structural changes may be made without departing from the scope of the disclosed subject matter.

[0041] According to various example embodiments of the disclosed subject matter as described herein, there is provided a method and apparatus for accessing multiple storage devices from multiple hosts without use of RDMA. The various embodiments described herein provide a new data storage architecture to meet the above requirements to help enterprises extract value from the vast amounts of data they have been capturing. In today's market place, there are three markedly distinct solutions that try to address the above-listed requirements of growing needs of data storage and analysis. These three conventional solutions are listed below:

1. Cluster of nodes with integrated storage - In the storage industry parlance, this topology is often referred to as "Clustered Direct Attached Storage" (Clustered DAS or DAS) configuration;

2. Virtual Storage Area Network (VS AN); and

3. Shared storage connected over a network - In the storage industry parlance, this topology is often referred to as "Network Attached Storage" (NAS) or "Storage Area Networks" (SAN).

[0042] These three conventional solutions are each described in more detail in the following sections and illustrated in Figures 1 through 5.

Clustered DAS 100431 Figure 1 illustrates an example of the conventional Clustered DAS topology.

Clustered DAS is typically dedicated to a single server and is not sharable among multiple servers. Figure 2 illustrates a softw are representation of the Clustered DAS with a user-space distributed file system. Figure 3 illustrates a softw are representation of this Clustered DAS with a kernel- space distributed file system.

VSAN

100441 A virtual storage area netw ork (VSAN) is a collection of ports, from a set of connected sw itches that form a virtual storage fabric. In general, a VSAN enables management softw are to serve data storage on cluster nodes to other cluster nodes.

NAS

100451 Figure 4 illustrates a softw are representation of the NAS. SAN. Figure 5 illustrates an example of the conventional NAS. SA topology. NAS. SAN can be shared among several server applications.

100461 Each of the conventional data storage configurations described above are sub- optimal in addressing the grow ing data storage and analysis needs. The follow ing table summarizes the challenges with DAS and NAS. SAN architectures in comprehensively meeting the solution requirements.

Solution DAS NAS/SAN

Attribute

Capacity Peta-bytes of capacity can be Compute capacity can be built

built using building blocks using building blocks that have

that have well-defined well-defined compute compute processing and processing. This is usually

storage capacity. This is achieved with a 2U server with

usually achieved with a 2U 1 or 2 CPU sockets and 12

server with 1 or 2 CPU drives for storage, with drives

sockets and 12 drives for sparsely populated.

storage, with all drives

populated.

Scalability The challenge is that the While this topology allows

compute and storage compute and storage capacities

capacities can't grow to grow independently of one

independently of one another. another, the challenge is that

This limits the expansion the storage capacity will need

capabilities to meet differing to grow as a step function in

compute and storage increments of controller head

requirements of multiple capability to process data. This

workloads sharing the cluster, would result in and would result in utilization overprovisioning and

inefficiencies. utilization inefficiencies.

Accessibility When expensive NAND flash Storage media is usually tied to

based storage media is used one or two controller heads,

within a compute node, it's and in the event of cascading

rendered useless when the failures, the data could go out

compute node is down or of access until after a coarse

disconnected from the cluster granular recovery takes place.

due to some unrelated

hardware failures.

Performance Given that storage media is Multiple protocol translations

tied within a compute node are required as the data box, all remote access to it traverses from compute node to must go over a cluster the controller head to the network typically shared with physical storage media. This

other critical data traffic introduces unnecessary among distributed workloads. performance overhead in This impacts performance on network throughput and both data and storage traffic. compute capacity. This is

If physical resources are significantly exacerbated in

virtualized in an attempt to situations where the compute

address some of the cluster is running a distributed

accessibility issues discussed processing framework that has

above, it will result in built-in distributed storage

additional storage processing semantics. In such

overhead and increased deployments, some of the work network traffic. done in the controller head is

unwarranted.

Cost While acquisition costs for NAS/SAN solutions come with

this architecture could be the controller head where

cheap, they do run into other intelligent software is run to

operational costs, such as manage the physical storage

inability to power media. This usually increases

down/hibernate some unused the cost of the solution a lot

compute nodes because it will more than the cost of storage

take away the attached storage physical media. In

from the cluster. deployments, where there is

enough compute capacity

present to perform the data

analysis, costs to cover the

controller head become a

necessary overhead that can be eliminated.

[0047] While DAS and NAS/SAN architectures can be used to build a data storage solution, these architectures fail to efficiently address the exponential growth in data storage and analysis needs. As the table below illustrates, these conventional architectures do not provide an efficient or optimal solution that addresses all the requirements without any trade-offs. Solution Attribute DAS NAS/SAN

Capacity Yes Yes

Scalability No Maybe

Accessibility No No

Performance No No

Cost Yes No

Switched Direct Attached Shared Data Storage Architecture

10048] Referring now to Figure 6, the sw itched direct attached shared data storage architecture and system 100 of an example embodiment is show n. The various embodiments described herein detail a new data storage architecture. Sw itched Direct Attached Storage or Switched DAS. to implement scale out clusters that need both storage and compute capacities. As described herein, a cluster represents a cluster of nodes, wherein each node has integrated compute capabilities and data storage. To meet all the solution requirements of grow ing data and analysis, the architecture of the various embodiments described herein leverages among the follow ing features:

• Packet switching and routing features in storage media interface fabrics;

• Centralized physical storage media with native interface connectivity to the fabric;

• Native storage media interface protocols to avoid multiple protocol conversions; and

• Distributed storage processing software layer on compute nodes.

100491 The Sw itched DAS architecture of an example embodiment has the flexibility to adapt to numerous underlying storage media interface protocols, and can also be extended to other clustering interconnect technologies via protocol encapsulation. The various embodiments described herein can be implemented with the most popular and standards based native storage media protocols, such as: NVMe (NVM Express). SAS SATA. or SCSI over PCIe (SOP). NVM is an acronym for non-volatile memory, as used in SSDs. NVM Express is a specification for accessing solid-state drives ( SSDs) attached through the PCI Express (PCIe)bus. Peripheral

Component Interconnect Express (PCIe) is a high-speed serial computer expansion bus standard designed to replace older bus standards. Historically, most SSDs used buses, such as SATA ( Serial

ATA). SAS ( Serial Attached Small Computer System Interface - SCSI ), or Fibre Channel for interfacing with the rest of a computer system. SATA has been the most typical way for connecting

SSDs in personal computers; how ever. SAT A was designed for mechanical hard disk drives, and has become inadequate with SSDs. For example, unlike hard disk drives, some SSDs are limited by the maximum throughput of SATA. Serial Attached SCSI ( SAS) is a point-to-point serial protocol that moves data to and from computer storage devices such as hard drives and tape drives. In an example embodiment, a data store switch fabric is implemented using Ethernet protocol and Ethernet data encapsulation. The following sections detail the specific procedures used in an example embodiment for: physical storage media assignment to compute nodes; data flow to/from the compute nodes and storage slices; and sharin of storage media in a Switched DAS cluster via a data store sw itch fabric.

Storage Assignment

[0050] Referring now to Figures 6 and 7. the physical storage media assignment to compute nodes in an example embodiment is illustrated. Figures 6 and 7 illustrate the physical configuration of the system hardw are in an example embodiment. As show n in Figures 6 and 7, the plurality of compute nodes 150 can be interconnected with one or more data storage slices 1 71 of the physical storage media pool or storage media container 170 via a data store sw itch fabric 160. In an example embodiment, the compute nodes or servers 150 can also be in data communication with each other via a local area netw ork 165 as shown in Figure 6.

100511 As show n in Figure 8, each data storage slice 1 71 is the physical unit of abstraction that can be plugged into or otherw ise connected with a storage media container 170. To the data store sw itch fabric 160, each storage slice 1 7 1 can be associated with the storage controller 1 72 residin on or in data communication with the storage slice 1 7 1 .

100521 Figure 9 illustrates a procedure 801 for assignin storage slices to compute nodes with NVMe storage. The procedure includes a cluster manager that distributes storage slice resources by assignin them to one or multiple Virtual Devices or NVMe Logic Units (NLUN) on one or multiple compute nodes. Each compute node will have an NLUN that consists of physical storage on one or multiple storage slices. Any portion of a storage slice can be shared by one or multiple compute nodes ( processing block 810). In a particular embodiment, the storage slice, represented by a combination of NVMe storage devices and a corresponding storage controller, can be identified usin a media access control address ( MAC address). On each compute node, either at pow er up or on reset, the BIOS (basic input output system) on the compute node binds the NVMe virtual drive to the device driver runnin on the compute node (processin block 820). The local file system runnin on the compute node can create a file system on these virtual drives volumes ( processin block 830).

100531 Referrin again to Figure 7. a sw itched DAS architecture of an example embodiment allow s multiple compute nodes to have access to storage slices from di fferent storage containers to increase the data accessibility in the presence of hardw are failures. As an example. three compute nodes (902, 904, and 906) are shown in Figure 7. Each of these compute nodes can be assigned with storage slices (912, 914, and 916), respectively, from two di fferent storage containers 920 and 930.

[0054] Each of the storage containers 920 and 930 and compute nodes (902, 904, and 906) can be configured with the location of the physical hardware. Storage container to compute node assignment can use the physical location as required to manage the data accessibility in the presence of hardw are failures. The same architecture, implemented with an Ethernet infrastructure as described herein, can be extended to use protocol speci fic identifiers and assignment with S AS SAT A protocols connected over an SAS expander, and SOP protocol connected over a PCIe switch.

Device Management

100551 Figure 10 illustrates the process in an example embodiment for device management. A switched DAS storage system with a pool of readily available driver shelves allows the flexibility of removing and adding storage to the pool of drives. This type of system needs to track each drive as they get moved throughout the system and identify them as unique.

[0056] In an example embodiment as shown in Figure 10, when a new drive is added to a cluster or a cluster is created, a hash is calculated based on a unique device identifier ( I D). This hash is used to address into a device ID table. The table entry is marked as being occupied and the device I D is placed into the table. This is show n in Figure 10. The table has additional information along with the Device I D to identify the device location within the sw itched DAS storage network.

[0057] If a drive is removed and then added back to the storage pool in a different location, the hash is again calculated to address into the Device ID Table. This time, the entry of the table is found to not be empty and the Device ID matches. The physical information of the new location of the device is added to the table.

100581 When a drive that has otherw ise been functioning as part of the storage pool is removed, the management entity of the local storage controller will hash into the device I D table removing the special location of the device from the table, but leaving the Device I D information in the table so the device can be identified if the device is returned to the storage pool.

Data Flow

100591 Figure I I illustrates the procedure 1201 in an example embodiment for data flow from a compute node to one or more storage slices. In the procedure of the example embodiment, a file system or block access layer sends native storage commands through the disk device driver that is attached to a storage slice ( processing block 1210). The native storage command and results are encapsulated in a transport protocol (e.g.. Ethernet. PCIe, etc. ) per the respective protocols. The storage slice responds to the native storage command per native storage standards.

Sharing of Storage Media

100601 Figure 12 illustrates a procedure 1300 in an example embodiment for storage slice sharing. In the procedure of the example embodiment, the compute node w rites to the storage slice to which it is assigned (processing block 1305). On a PCIe fabric, a virtual function (VF) associated with the same physical function v irtual function ( PF VF) of the compute node is assigned to the remote compute node looking to share the data (processing block 13 1 ). Through a distributed softw are layer, the remote compute node is informed of the storage slice location, identity, offset, and length of the data ( processing block 1325). The remote compute node accesses the data. Once the data access is complete, the remote compute node informs the originating compute node of the task completion ( processing block 1335). The originating compute node reclaims control and continues w ith operations ( processing block 1345 ). On an Ethernet fabric using an NVMe tunnel, a virtual drive or NLU is used to distribute and share portions of the physical data devices or drives of multiple data storage slices ( processing block 1355 ). On an SAS fabric, a logical unit number (LUN) is used as a shared object between compute nodes ( processing block 1365).

100611 One of the key advantages of centralizing storage media is to enable dynamic sharing by cooperating compute nodes. The sw itched DAS architecture of the example embodiments enables this feature.

Switched DAS Using Ethernet

100621 Referring again to Figures 6 through 8 and 13 through 14. the example embodiment show s a basic data storage configuration that represents the common compute and storage interconnection scheme. The various example embodiments described herein use this basic topology and improve the way that data is moved through the system. The improvements lead to a drastic improvement in overall system performance and response time without impacting system reliabil ity and availability. The disclosed architecture reduces protocol layers in both the compute server and storage device end of the system.

100631 The architecture of the various example embodiments described herein eliminates complicated high latency I P ( Internet Protocol ) based storage protocol and its softw are based retries with long 10 ( input output) time-outs. These protocols are used to w ork around Ethernet^'s lossy nature to create a reliable storage protocol. 10064] The architecture of the various example embodiments described herein uses a data store switch fabric 160 to tunnel directly between nodes using server-based 10 protocols across the network, resulting in directly exposing high performance storage devices 1 7 1 to the netw ork. As a result, all the performance of the storage devices is made available to the network. This greatly benefits the compute server appl ications.

100651 Figure 1 3 illustrates a data flow 1301 in a switched DAS architecture of an example embodi ment using Ethernet as the transport fabric protocol. Referring to Figure 1 3. at the compute server 150 end. an 10 operation is initiated in the same manner as i f the storage device 1 7 1 were internal to the compute server 150. Compute node sends nati ve storage commands through the disk device driver, as i f the storage slice was directly attached ( processing block 1310). This 10 operation, data request, or native storage operation ( e.g.. commands, data, etc. ) gets encapsulated in an Ethernet frame ( processing block 1320). The Ethernet frame is then shipped via the data store switch fabric 160 to a storage device 1 7 1 at the other end of the network ( processing block 1330). At the storage end of the netw ork transaction, the Ethernet tunnel is undone, the Ethernet encapsulation is removed, leaving native storage operations, and the 10 protocol is passed to the storage device 1 7 1 as i the storage device 1 7 1 were connected via a direct method to the compute server 1 0 ( processing block 1340). The storage slice responds to the native storage command, as i f the compute node was directly attached ( processing block 1 350). As a result, the data store switch fabric 160 enables data communications betw een a plurality of compute nodes 150 and the pl urality of data storage devices 171 in a manner to emulate a di rect data connection. In an example embodi ment, the storage device 1 7 1 can be solid-state dri ve ( SSD). A solid-state drive ( SSD) is a type of data storage device, such as a flash memory device, which uses memory technology rather than conventional rotating media. The encapsulation of 10 operations into a standards based Layer 2 Ethernet frame is show n in Figure 14.

[0066] Referring to Figure 14. the encapsulation of an 10 operation into a standard

Ethernet frame is shown. The architecture of the example embodiment uses standard Ethernet protocol as an integral part of the storage system of a particular embodiment. As a result, it is extremely efficient and effective to use VLAN ( virtual local area netw ork) features to segregate and prioriti/e the storage traffic that is built with Ethernet as its core fabric. It will be apparent to those of ordinary skill in the art in view of the disclosure herein that many other alternati ve i mplementations can be used to segregate and prioriti/e storage traffic. The architecture of the example embodiment can utilize information available in the creation of the 10 traffic where the tunnel is constructed to decide how to prioriti/e or segment the Ethernet flow s. The architecture also provides a hardware-based packet loss detection and recovery feature. Moving the packet loss detection and recovery to a fast, close-to-the-netw ork mechanism improves the performance of the overall system over previous implementations.

Storage Processing on Application Servers, with External Sw itch DAS

[0067] Referring again to Figures 6 through 8, the example embodiment provides a very novel approach with significant benefits over today^'s storage architectures. Due to the high performance and small form factor of solid state memory devices currently on the market, old methods of external storage based on devices behind a single controller or banks of 10 controllers, typically Intel^® based motherboards, are too costly and woefully under provisioned.

[0068] This result of overpriced and underperforming external data storage solutions led to a transition in the data center. Compute users moved storage internal to the compute or application servers. This solved the cost and performance issues they were experiencing with external storage. It w orked great in smaller configurations. How ever, it is exceedingly difficult to support large compute environments with internal storage. Scaling storage independent of the compute environment is problematic. The density of the compute cluster is not optimal when placing storage in a server. Finally, the cost and performance of solid state devices is high enough that trapping devices in a single server, which is the least reliable portion of the system, is not cost effective and reduces overall system reliability.

100691 The data storage architecture of an example embodiment described herein moves the SAN/NAS type of storage processing softw are onto the compute nodes. This removes both cost from the system as well as performance bottlenecks of the external SAN/NAS or object storage architecture. However, the architecture of the example embodiments utilizes externally sw itched DAS storage that exposes the performance of the drivers directly to a storage netw ork. This allow s for SAN/NAS type reliability, manageability, and availability that internal storage cannot offer. Removing storage from the compute servers now allow s the compute environment and storage to scale independently. The removal of storage from the compute server allow s for a more dense performance point. The density of the distributed storage solution of the example embodiments is far greater than that of internal storage, thereby reducing both pow er and footprint of the implementation.

Platform Softw are Architecture

[0070] The various example embodiments provide technology and a softw are platform for: instrumentation hooks to monitor, measure, and enforce performance metrics into the compute, memory, network and storage resources; and continuous monitoring of the health of all resources to predict failures and proactively adjust/update the cluster resources. Details of the software platform in an example embodiment are provided below .

Instrumentation hooks to monitor, measure, and enforce performance metrics into the compute. memory, network and storage resources.

100711 Referring to Figures 15 and 16, a first step in an example embodiment is to perform resource awareness flow. This includes creating a catalog of available hardw are and their respective performance levels (e.g., flash devices or device types, number of NIC links per compute node, throughput and IOPS of storage devices, switch fabric infrastructure, connectivity, and timing, etc. ). A second step is to perform predictive Service Level Agreement ( SLA) requirement analysis. All resources that are required to run a job are virtuali/ed. namely Central Processing Unit (CPU ), memory, network, and storage. Jobs can be implemented as Hadoop jobs. Hadoop is a well-known open-source softw are framew ork from Apache Softw are Foundation for storage and large-scale processing of data-sets on clusters of commodity hardw are. Apache Hadoop is a registered trademark of the Apache Softw are Foundation. Platform softw are is made aw are of the performance capabilities such as throughput. IOPS ( input output operations per second), latency, number of queues, command queue-depth, etc. of all the underlying hardw are resources in the storage platform. The platform softw are will run matching algorithms to align the resource usage of a specific job against the hardw are capabilities, and assign virtuali/ed resources to meet a specific job. As cluster usage changes, the platform softw are continuously maps delivered SLAs against predicted SLAs. and adjusts predicted SLAs.

[0072] A job^'s execution time for a job "j" is predicted based on: Tj = f (# of phases in the job. # of datasets the job is using. # of sub-task datasets the job will be split into. # of processing units assigned for the job. # of memory bytes assigned for the job. the worst case time for each of the sub-tasks, the average delay between phases, the average network throughput, the average disk throughput, the average disk input output ( IO) operations, etc. ).

[0073] Referring now to Figure 15, an example embodiment illustrates a process 1500 to perform resource aw areness flow . For all the hardw are in the cluster: 1) cluster management applications are made aware of the raw performance capabilities of all hardware resources in the cluster (e.g.. number of NIC (netw ork interface controller) links per compute node; throughput and IOPS of underlying storage devices, switch fabric infrastructure, connectivity, and timing, etc.); 2) the cluster manager creates a catalog of available hardware and their respective performance levels (e.g., flash devices or device types, number of NIC links per compute node, throughput and IOPS of storage devices, switch fabric infrastructure, connectivity, and timing, etc.); and 3) the cluster manager creates and manages IO usage statistics (processing block 1510). 10074] Referring now to Figure 16, an example embodiment illustrates a process 1700 to perform predicti ve service level agreement requirement processing. In an example embodiment, a job is submitted into the cluster with job meta data (processing block 1710). The process can review and/or initialize statistics based on the job performance or the job profile (processing block 1720). The process can predict the expected time it would take for the job to complete on the cluster based on the job's statistics, available resources, and profiling results (processing block 1730). The process can match the job's statistics and profiling results against the hardware catalog performance metrics and provide an estimated amount of time to complete the job at the assigned priority level and an expected amount of standard deviation seen on the cluster (processing block 1740). As the job gets executed on the cluster, the process can monitor job progress and periodically assess the completion time and match it against the predicted job completion time. The process can adjust the resource assignment of the job to meet the predicted completion times. The process can warn an operator or a cluster management application of excessive delays (processing block 1750). For repetitive jobs, the process can store the job's resource requirements and track the job's actual execution time. The process can adjust the predicted time as the job gets executed and update statistics (processing block 1760).

Continuous monitoring of the health of all resources to predict failures and proactively

adjust/update the cluster resources

[0075] Referring to Figures 1 7 and 18, the platform softw are of an example embodiment continuously monitors the health of all critical hardware components across various compute nodes and storage containers. The system also performs resource monitoring to avoid failures. Platform software is made aware of the failure characteristics such as wear-levels of flash storage, failure ratings of power supplies, fans, network and storage errors, etc. of all the underlying hardw are resources in the storage platform. The platform softw are implements hooks to monitor the health of hardw are resources into the respective softw are control blocks. The platform softw are runs continuous fail ure models and proactively informs alerts an operator or a cluster management application to attend update the hardw are resource that is in question. When a change in resource is i mminent, the platform softw are proactively reduces the usage of affected hardw are, rebalances the storage, netw ork and compute tasks, and isolates the affected hardw are for quick and easy replacement.

[0076] Referring to Figure 18, an example embodiment illustrates a process 1800 to perform platform softw are resource monitoring for failure avoidance. In the process of an example embodi ment, the platform softw are periodically polls the health, usage, w ear-level of flash, error levels on NIC interfaces, and performance levels of all hardw are components ( processing block 1810). The process runs failure prediction analysis on components that are heavily used (processing block 1820). For components that are closer to failing based on a pre-configured probability and earlier than a pre-configured time limit - start the resource mitigation activity and don^'t take any new usage on the affected component( s) ( processin block 1830). After resource migration is complete, the process automatically marks the affected components as off-line ( processing block 1840). Then, the process automatically re-adj usts the projected completion ti mes for outstanding jobs ( processing block 1850) and generates alerts to an operator or a cluster management application for any needed corrective actions ( processing block 1860). In an alternative embodi ment, areas of the flash drives which are showing high levels of wearing ( or bad cell sites) can be used for the storage of lightly w ritten data ( e.g.. cold data storage). In this manner, the worn areas of the flash drives can still be used w ithout w asting storage.

Input Output ( 10) Acceleration Using an Ethernet Connection

[0077] Referring to Figure 1 9, the example embodi ment shows the standard NVM Express

1 . 1 speci fication. Step 2 of the 10 flow shown in Fig. 1 9 identi fies a host w rite of a doorbell . When this occurs, the Host NIC 1 56 (network interface controller show n in Figure 6) of an example embodi ment forw ards the doorbell down the Ethernet connection of the data store sw itch fabric 160 to the storage controller 1 72 as show n in Figures 6 and 8 w here the doorbell eventually gets passed to the storage device 1 7 1 ( e.g.. a flash drive or other SSD). At the same time, the Host NIC 1 56 acts on the doorbell and fetches the command from the Submission Queue as identi fied in step 3 of Figure 19. The Host N IC can start to process the command before the storage device has seen the command. The Host N IC 1 56 can send the relevant information across the data store switch fabric 160 ( e.g.. the Ethernet connection ) to the storage controller 1 72. When the storage device 1 7 1 sees the doorbell, the head information of the command has already been fetched and is either on the way or has arrived in the local packet buffer or the storage controller 1 72. This method of prefetching commands and data and overlapping processing operations effectively hides latency and improves performance of the 10 system. Additionally, by being 10 aware, the hardware can handle the lossy nature of Ethernet and more reliably handle packet drops.

Input Output ( 10) Virtualization Layer in an Ethernet Environment

[0078] Referring again to Figure 6, the example embodi ment shows the basic system interconnect w here a Host 150 w ith an Ethernet N IC 1 56 is connected via an Ethernet connection infrastructure of data store sw itch fabric 160, w hich is then connected to an Ethernet based storage controller 1 72. The storage controller 1 72 is connected to an SSD 1 7 1 . This is the basic physical configuration of the storage system of an example embodiment. The Host N IC 156 presents a virtual SSD to the server 150. The storage controller 1 72 presents a virtuali/ed root complex to the SSD 1 71 . As such, the Host NIC 156 presents an endpoint to the compute node 150. The storage protocol is tunneled across the Ethernet connection infrastructure. Tunneling the protocol limits the complexity, power and latency of the Host NIC 156 and storage controller 1 72. The virtuali/ation allows any host to be able to communicate to any number of storage controllers to utilize a portion of or the entire addressable space of the S SDs to which it is connected. Virtuali/ing the devices allows the example embodiments to use host resident storage management software 155 that can then implement features common to enterprise SAN and NAS systems at a much higher performance level, lower power level, and lower system cost.

Messaging Protocol

[0079] A low latency reliable secure messaging protocol is an important part of the data storage architecture described herein. The messaging protocol provided in an example embodiment uses the same connectivity infrastructure that customer 10 operations use. The architecture of the protocol permits a responding compute server to directly send indexes and meta data to the locations where a requesting compute server will use the data, eliminating any memory copies. This saves valuable system bandw idth as well as increasing storage software performance. The messaging protocol also reduces system response latencies. Performance is also optimized as hardware can snoop the message entries while moving the data to obtain information used to ensure the memory integrity of the system receiving the indexes and meta data, thereby eli minating another queue or table.

100801 Figure 20 illustrates a compute server to compute server configuration of the messaging protocol of an example embodiment. As described above, compute nodes or servers can be in data communication with each other via a local area netw ork 165. The messaging protocol of an example embodiment can be used to facilitate this data communication. As described herein, the term Initiator is used to identify the server that is sending a Request Message to get information from a Target server that sends a Response. As described herein, a response is a generic term for the data that is being used by the storage system softw are of an example embodiment. This data can include index data or other meta data or system status. In the example embodi ment, the messaging protocol described herein is a peer to peer (P2P) protocol. As a result, any server in the compute environment can and will be initiator and a target of the message passing protocol based on the needs of the systems at the time a conversation starts.

100811 Referring to Figure 2 1. the data flow 300 for a sample message using the messaging protocol of an example embodi ment is illustrated. The Initiator starts 301 a conversation by placing an entry into a work queue 320. The initiator then rings a doorbell telling the target a w ork queue entry is available. Next, the Target reads 302 the work queue entry. A side effect 303 of the work queue entry read moves check information into the Address Translation Unit (ATU ) 330 of the hardw are. The Target receives 304 the work queue entry, processes the work queue entry, and builds the appropriate response packet. The response packet is then sent 305 to the Initiator where the response packet is processed by the Address Translation Unit ( ATU) 330. I f there is no active check information matching an ATU 330 context for the response, then the message will be routed 306 to the Maintenance Queue 340 and a Completion message will be posted by hardware. I f the check information for the response matches an active ATU 330 context, then the response will be routed 307 to the appropriate system memory location in system memory 350. Multiple response messages can be sent 308 during one ATU 330 context depending on the rules set captured by the ATU 330 from the hardw are portion of the original request message entry. The ATU 330 has a context and is dynamic in nature. The ATU 330 can be opened and then closed as the message conversations start and then complete. At the end or completion of a conversation, a completion queue entry is written 309. Depending on the conversation, there could be multiple completion queue entries.

Feature Offloads in an Ethernet Environment

100821 Referring again to Figure 6, the example embodiment show s the basic structure of distributed storage netw ork connectivity in an example embodiment. The example embodiment utilizes this netw ork topology to implement storage features without impacting the compute servers and the links to the compute servers. Examples of these features include mirroring disks and building or rebuilding replicas of drives. Again, this is all done independently of the compute servers. This saves valuable bandw idth resources of the compute servers. These features also increase overall storage performance and efficiencies as well as low er the overall pow er of the storage implementation.

100831 Another class of offloads, which offload the processing burden of the compute cluster of servers, allow s the compute servers to write to a single data storage replica and have this storage device automatically move the updated data to a designated replica within the storage netw ork. Replication processing in an example embodiment is illustrated in Figure 30. This effectively eliminates the need to write to both the primary and the secondary storage device of a replica pair. A variation of the single write to a replica pair is to write two pieces of the updated information to each replica. The storage devices finish the transaction by sending the updated data they received to their mirrored device. This type of write method frees up bandw idth on specific links to the compute servers by allow ing each transaction to go dow n a different path to the network. In a particular embodi ment, multicast Ethernet packets can be used to send the same data to multiple destinations.

Storage Processing on Application Servers with External Sw itch DAS

100841 Figure 22 shows the basic organization of the current flash media. An enterprise class SSD is made up of many assembled chips of flash devices. The devices could be assemblies of multiple die in one package. Each die is made up of multiple blocks with many pages per block. The memory is address at a logical block boundary. Flash media is a media that does not allow di ect w rites. I f new data is to be written, a blank area must be found or an existing area must be erased. The unit of space that is bulk erased at one time is generally called the erase block. Because of this lack of direct write capability for this type of memory device, there is a management overhead. This management overhead includes managing the logic data blocks as virtual in that they don^'t exist in a speci fic physical location; but. over time are moved around the physical memory as various writes and reads occur to the die. Additionally, the media will wear out over time. Spare area is maintained to allow for user physical locations to fail and not lose user data.

100851 The organization of the blocks, pages, logical blocks, and erase blocks vary from generation to generation and vendor to vendor. The characteristics of the media will vary in an even greater manner as new non-volatile memory technologies appear.

100861 As described herein, an example embodi ment provides an 10 layer that virtual i/es the storage from the appl ication or operating system and then optimizes that storage to get the best performance out of the media, particularly flash memory devices. The example embodiment enables the i mplementation to avoid the performance pitfalls, which can occur when the media is not used opti mally.

[0087] With one interface, users can get the best out of flash memory devices from di fferent generations of flash memory devices, di fferent vendors, di fferent drives, and even di fferent non-volatile technology. This virtuali/ation softw are layer that is flash memory device aw are formats the physical media to optimize w rites so as to limit the need for the flash memory devices to perform garbage collection. This is done by ensuring all files or records are flash erase bank aligned and a multiple of the erase bank si/e. Additionally, block si/e is a multiple of the erase bank si/e.

[0088] The abil ity to format a drive and write records with an erase buffer in mind also help reduce the need for spare pages. This frees up the pages from the spare pool and makes the pages available to customer appl ications. The example embodiment increases the density of a current flash device due to the optimized usage of the device. This creates a more cost effective solution for customers.

Input Output ( 10) Performance Optimizations Based on Workload

100891 Today^'s storage slacks are developed to provide the optimal performance for an average 10 and storage w orkload the system will see. or the user can force the system to use preferred settings. Some systems will allow the user to characterize their w orkloads and then the user can set the systems to use a given set of settings.

[0090] The various embodiments of the data storage system described herein are designed to enable adjusting to the 10 traffic and storage characteristics as the traffic profile changes. The various embodiments can also be programmed to alert the operator or cluster management application when the traffic pattern is seen to cross preset limits. The various embodiments allow different segments of the storage to utilize completel different 10 and storage logical block settings to optimize performance.

100911 The feature of adjusting the configuration of the 10 stack and hardw are to the measured 10 & storage traffic is coupled with the knowledge of the flash media described above in connection w ith Figure 22. This feature of the various embodiments provides customers w ith the best possible performance for the jobs they are running as they run them. This feature also addresses multi-tenant environments being run on the cluster.

Flash Aw areness and Failure Avoidance

100921 The various embodiments described herein maintain real-time know ledge statistics of flash drives, which allows the system to avoid failures. Areas of the (lash drives which are showing high levels of w earing (or bad cell sites) can be avoided w hen w riting data. The cell use and the latency are monitored to determine w ear. To monitor wear, data can be re-allocated to alternate drives and the storage met a data maintained on the compute nodes can be updated. 100931 As individual flash drives near preset w ear leveling targets, data can be slotted to other drives and meta data updated. I f the user selects this feature, data can also be moved to alternate SSD^'s autonomously w hen these target thresholds are crossed. In addition, areas of the flash drives w hich are showing high levels of w earing (or bad cell sites) can be used for the storage of lightly w ritten data (e.g.. cold data storage). In this manner, the w orn areas of the flash drives can still be used w ithout w asting storage.

Storage Meta Data Structure 100941 Referring again to Figure 6, the example embodiment shows a basic compute environment where compute servers are attached to storage devices. Applications can run on the servers and the application data as well as operating data can reside on the storage devices. The environment enables object storage devices to perform at comparable or greater levels to compute servers with internal storage and vastly outperform other methods of external storage devices and storage systems, such as SAN and NAS storage as described above. This improved efficiency frees up the user to independently scale the compute and storage needs of their compute clusters without adversely impacting the performance. The distributed object store will have unmatched performance density for cluster based computing with the availability features of SAN or NAS storage.

100951 Figure 23 show s the object tag format for the object store of the example embodiment. The type field is used to define what fields are present in the rest of the tag as some files are optional and some fields can be duplicated. This is done to enable and disable storage of each object stored. The object source is a netw ork pointer to where the object resides in the network. This object source is generated to allow current commercial sw itches to locate the object source in an Ethernet netw ork with hardw are speed or the smallest possible latency. For a given IO command the object tag is used to move that 10 command to the correct location for the command to be processed. After an 10 command has been sent to the correct location for the 10 command to be processed, the object locater field is used to find the data object the command is processing or accessing. Finally the object feature field is used to track any special requirement or actions an object requires. It is also used to determine any special requirements of the object. Agents can use this field to make decisions or perform actions related to the object.

Uses Cases of the V arious Embodiments:

100961 The Sw itched DAS architecture of an example embodiment has a wide variety of use cases. The follow ing list presents a few of these use cases:

1. Using a distributed storage access layer across compute nodes - it could be used to build a scale-out cluster with centralized storage media to catering to Hadoop framework.

2. Using a distributed memory layer across compute nodes - it could be used to build a high- capacity shared memory running into 100's of terabytes (TB) and more.

3. Using a distributed block storage access layer across compute nodes - it could be used to provide a storage backend for RDBMS (relational database management system) applications addressing OLTP/OLAP (online transaction processing / online analytical processing) transactions. 4. Using a distributed object storage access layer across compute nodes - it could be used to build a scale-out cloud storage server.

5. Using a distributed storage access layer across compute nodes - it could be used to build a

VDI (virtual device interface) hosting server farm with integrated storage.

Accessing Multiple Storage Devices from Multiple Hosts Without Use of Remote Direct

Memory Access (RDMA)

Description of Conventional Technology

[0097] Figure 4 depicts conventional host server interconnections using existing protocols, such as Fibre Channel or serially attached Small Computer System Interface ( SCSI ). These protocols add significant overhead to each input output ( 1 0) operation, because the rotating physical media on the storage devices is slow. Historically, this was not a major issue as the devices were then sufficient to fulfill application needs.

100981 Figure 5 depicts similar interconnections using existing networks instead of protocol-specific interconnects, such as in Figure 4. While such implementations make the interconnects non-proprietary and less dependent on a specific protocol, additional netw orking overhead is introduced, so that the benefits of longer attachment distances, sharing, and use of existing network infrastructure can be lost due to the even slower access.

[0099] Figure 24 shows a specific example of the conventional system shown in Figure 4. where storage is attached via Ethernet using conventional protocols, such as Internet SCSI (iSCSI ). Fibre Channel over Ethernet ( FCoE). Advanced Technology Attachment ( ATA) over Ethernet ( AoE). and so forth.

100100| Recent technologies, such as FLASH (nonvolatile) memory, have resulted in storage devices at least 100 times faster than prior nonvolatile storage. However, in order to utilize the newer and faster speeds, the nonvolatile storage needs to be installed w ithin a server to take advantage of the server^'s internal bus speeds. Existing external connectivity and protocols are too slow to be used in such an implementation.

[00101] Figure 25 illustrates how NVM Express devices are accessed when locally installed in a server. NVM Express. NVMe, or Non-Volatile Memory Host Controller Interface S peci fi cat i on ( V M H CI ). is a specification for accessing solid-state drives (SSDs) attached through the PCI Express ( Peripheral Component Interconnect Express or PCIe) bus. "NVM" is an acronym for non-volatile memory, which is used in SSDs. Referring again to Figure 25, 1 0 operation requests, or "submissions.^"" are placed on producer queues. When data are transferred and the operation is completed, the results are posted to a "completion^"" queue. This protocol results in very fast, low latency operations, but does not lend itself to being used in a multi-server environment; because, the protocol is designed to be point-to-point solution. Also. PCIe has bus length, topology, reconfiguration, and sharing restrictions.

100102| Figure 26 illustrates a typical RDM A hardw are and software stack required to implement remote access of NVM Express devices. Note the large number o f layers of software in order to transport a request. The multiple layer overhead far exceeds the native device speeds.

Description of the Data Storage Access System of Example Embodiments

100103| One purpose of the data storage access system of various example embodiments is to allow a plurality of hosts servers to access a plurality of storage devices efficiently, while minimizing the hardware, firmware, softw are, and protocol overhead and cost. This results in the follow ing benefits:

• Devices perform at very near natively attached rates, so that applications perform with equivalent efficiency.

• Devices can shared amongst hosts servers, allow ing for more flexible configurations.

• Devices can be moved betw een servers with no hardw are changes, allow ing for fast server failure recovery times.

• Devices are accessed in a transparent fashion, allow ing for no change to the devices and little to no change to the host server or its applications and systems.

• Because of the simplicity of the design of the data storage access system of example embodiments, the implementation cost is nearly identical to that of individual devices spread amongst hosts servers.

• Because of the transparency and simplicity of the design, it is possible to relocate devices that are already installed in hosts servers to be moved into the data storage access system of example embodiments w ithout incurring the cost and time of data migration.

100104] Figure 27 illustrates an embodiment 2700 of the data storage access system of the example embodiments described herein show ing the savings in complexity to be gained by use of the example embodiments over the conventional implementation, for example, show n in Figure 26. Many layers of protocol and additional messages are no longer needed, resulting in much improved performance. In particular. Figure 27 illustrates an embodiment 2700 of the data storage access system, which includes a host system 2710 in data communication with a data storage controller system 27 12. The data communication betw een one or more host systems 2710 and the data storage controller system 2712 is provided by an NVMe tunnel 2714. In a particular embodiment, the NVMe tunnel 27 14 can effect the high-speed transfer of data to from the data storage controller system 27 12 using an Ethernet data transfer fabric. It will be apparent to those of ordinar skill in the art in view of the disclosure herein that other well-known data communication fabrics can also be used with the NVMe tunnel 27 14. In general, the NVMe tunnel 2714 provides a high-speed (e.g.. 40 Gigabit Ethernet) Layer 2 data conduit betw een the one or more host systems 271 0 and the data storage controller system 27 12. The details of an embodiment of the data storage access system 2700 and the NVMe tunnel 27 14 are provided below and in the referenced figures.

100105] Figure 28 illustrates the configuration of queues in the host bus adapter (HBA) or host network interface controller (N IC) in an example embodiment. Figure 28 also illustrates the positioning of the HBA or NIC between the network endpoint (e.g.. PCIe endpoint) and the data transmission fabric (e.g.. Ethernet).

100106| Figure 29 illustrates a detail of the configuration of queues in the host bus adapter ( HBA) or host netw ork interface controller (NIC) in an example embodiment. The set of queues of the HBA or Host NIC in an example embodiment includes a set of management queues and a set of data path queues. The management queues include a set of administrative submission queues, a set of administrative receive queues, and a set of administrative completion queues. The management queues enable the transfer of control or configuration messages betw een nodes (e.g.. servers hosts, storage controllers, or other fabric-connected components) w ithout interruption of the operational data flows transferred via the set of data path queues. The set of data path queues includes an Input Output ( I O) submission queue and a set of completion queues corresponding to each of a plurality of processing logic components or cores. As described in more detail below , the set of queues of the HBA or Host NIC in an example embodiment enable the high speed transfer of data across the NVMe tunnel to/from nodes on the other side of the data communication fabric. 100107| Figure 30 illustrates an architectural view of the storage controller 27 10 of an example embodiment in netw ork communication with a plurality of host server systems 27 10 via a storage netw ork 3010. As show n, the storage controller 27 12 in an example embodiment can be configured with sets of queues to handle the flow of data traffic betw een a node (e.g.. host server 2710) on the other side of the data communication fabric 3010 and a data storage repository 3012 (e.g.. NV e SSD Controller). The set of queues of the storage controller 27 12 in an example embodi ment includes a set of management queues and a set of data path queues. The management queues include a set of administrative submission queues and a set of administrative completion queues. The management queues enable the transfer of control or configuration messages betw een nodes (e.g.. servers hosts, storage controllers, or other fabric-connected components) and the data storage repository 3012 without interruption of the operational data flows transferred via the set of data path queues. The set of data path queues includes an Input Output ( 1 0) submission queue and a completion queue. Because the example embodiment can retain information from which a context can be inferred, a context cache is provided to cache this context information. As described in more detail below, the example embodiment can retain information needed to instruct the data storage controller of an example embodiment how to present the ending status of an operation. In other words, the context information can assist in defining the disposition of the request. In this example, the request disposition can represent the number or identifier of a completion queue to which the ending status is posted (e.g., Completion Queue Context). Optionally, a request may direct the data storage controller to post ending status as soon as the request is transmitted, for example, to signify that a stateless broadcast was sent (e.g., Submission Queue Context). The context information can be used to differentiate among a plurality of outbound data paths and correspondin outbound data path queue sets. As described in more detail herein, the set of queues of the storage controller 2712 in an example embodiment enable the high speed transfer of data across the NVMe tunnel between nodes on the other side of the data communication fabric 3010 and the data storage repository 3012.

100108] Figure 3 1 illustrates an example of a method for a host server to communicate 1 0 requests to devices installed within the data storage access system of an example embodiment. In a particular embodiment, the 1 0 requests are handled by a host bus adapter (HBA) or network interface controller ( NIC) on the host system. Requests to the data storage access system of the example embodiment can be placed on the first two Queues (SQO and SQ I ). one queue to transmit requests, the other queue to transmit completions. These requests take priority over any other requests, allow ing for a path to issue error-recovery directives, such as component resets or overall system configuration updates.

100109| Queue two (SQ2) is used to direct administrative requests to devices.

Administrative requests are used to control the physical aspects of a device, such as formatting its media, or to issue error-recovery commands, such as individual operation aborts or device resets. Administrative queue requests take priority over device 10 operations.

[00110] The remaining queues (SQ3 through SQ7) are used to issue application-related 1 0 operations (e.g.. reading and writing of application data). Multiple queues may exist to allow ordering of operations to devices or to alter the priority of queued operations.

1001 1 11 Each queue contains a variable number of elements, each of which represents a unit of work for the data storage access system of the example embodiment to perform. Each element can be comprised of the follow ing two parts:

• Shadow queue element, containing information over and above the request itself.

• Submission queue element, containing sufficient compatible information necessary to execute an individual operation. [00112] In one embodiment of the data storage access system, the submission queue contents exactly match those defined in the NVM Express Specification for NVM Express devices. The shadow and submission queues may be distinct regions in memoiy. or may be combined into a single element. How ever, the request receiving component of the data storage access system can receive both the shadow queue element and the submission queue element together. When transmitted across a netw ork, unused or reserved fields of the request may be omitted to save time and netw ork bandwidth.

[00113] Figure 32 illustrates example contents of a single Shadow Queue Element in the data storage access system of an example embodiment. This element includes:

• Information necessary to route the current request to a particular component of the data storage access system of an example embodiment, such as an individual device. In this example, a destination Ethernet Media Access Control (MAC) address or other network-specific addresses can be provided. The network-specific address can address a single component via a unicast, multiple components via a multicast, or all components via a broadcast. In a particular embodiment, a destination MAC address and a source MAC address can be provided. The destination MAC address can address a single component via a unicast, multiple components via a multicast, or all components via a broadcast. The source MAC address can be the sending port's MAC address. It will be apparent to those of ordinary skill in the art in view of the disclosure herein that the network-specific addresses can be Ethernet MAC addresses or other types of device addresses compatible with a particular data communication fabric.

• Information necessary to identify the class of request. In the example shown in Figure 32, the "Command" field designates the current request as a request submitted to the data storage access system, or to an individual device. In other words, the class of request field can override the implicit type by queue number.

• Information needed to instruct the data storage access system of an example

embodiment how to present the ending status of the operation. In other words, this element defines the disposition of the request. In this example, the request disposition can represent the number or identifier of a completion queue to which the ending status is posted (e.g., Completion Queue Context). Optionally, a request may direct the data storage access system to post ending status as soon as the request is transmitted, for example, to signify that a stateless broadcast was sent(e.g., Submission Queue Context).

• Information needed to prioritize the request and/or segment requests for security. In this example, optional Virtual Local Area Network (VLAN) information and/or Port / Routing Context can be provided, allowing the intervening network to order network traffic.

[00114] Generally, each issued request results in a completion event, which is placed into the next available slot in a completion queue. In this example embodiment of the data storage access system, the format of the completion queue can be identical to the format defined in the NVMe Specification. As such, the format is not discussed further here.

[00115] Figures 33 and 34 illustrate example register sets of the data storage access system of an example embodiment used to set up and control the various request and completion queues as described herein. In particular. Figure 33 illustrates the Submission Queue (SQ) Registers in the example embodiment. Figure 34 illustrates the Completion Queue (CQ) Registers in the example embodiment. Generally, these register sets, one per queue in the example embodiment, define the storage address of that particular queue, the queue length in number of elements, and "producer^" and "consumer^" queue pointers for the task adding requests to the queue (the "producer^") and the task servicing requests (the "consumer^"). Other fields in the register sets define the type of queue and provide other debugging and statistical information.

[00116] Figures 35 and 36 illustrate examples of how a host 1 0 request flows through the data storage access system of an example embodiment. In particular. Figure 35 illustrates a typical data read transaction flow in an example embodiment. Figure 36 illustrates a typical data write transaction flow in an example embodiment. In each example, a host server-specific tag is included with the transaction in order to identi fy which host server memory is used to transfer data betw een the host server and the device(s) being addressed.

[00117] Referring now to Figure 35. a typical data read transaction flow ( read data transferred from storage device to Host) in an example embodiment is illustrated. The basic sequence of processing operations for handling a read data transaction in an example embodiment is set forth below with the processing operation numbers listed below corresponding to the operation numbers shown in Figure 35:

1. Host writes a new queue index tail pointer to HBA Submission Queue Tail Doorbell register.

2. HBA generates a read request to access a Host memory Submission Queue Shadow entry using queue index head pointer and Submission Queue Shadow base address information.

3. Host returns the requested Submission Queue Shadow entry.

4. HBA generates a read request to access a Host memory Submission NVMe

command queue entry using queue index head pointer and Submission Queue base address information. 5. Host returns the requested Submission Queue NVMe command entry.

5a) The Shadow and NVMe command entries are used to generate a message that

contains the proper fabric information to reach the storage controller and drive.

5a.2) The message is encapsulated within a tunnel header and fabric (e.g., Ethernet)

header, and sent across the fabric (e.g., Ethernet).

5b) The storage controller receives the message and stores off the NVMe command entry and fabric information to be used later in the NVMe I/O.

5c) The storage controller tags the upper bits of the Physical Region Page (PRP)

addresses of the NVMe command with an I/O context tag and saves off the replaced bits. During the data phase of the I/O, the tag field of the address is used to determine to which I/O and host the data phase request belongs.

6. The storage controller writes NVMe Submission Queue doorbell register of the drive and the drive reads the local NVMe entry.

7. SSD returns data for the NVMe read.

7a) The tag field bits of the address are used to perform an I/O context lookup. The tag field of the address is restored to its original value (stored in step 5c) and the Transaction Layer Packet (TLP) is directed back to the requesting host based on the fabric information from the I/O context (stored in step 5b).

8. SSD writes NVMe completion.

9. The storage controller intercepts the NVMe completion and directs it back to the requesting host's proper completion queue based on the fabric information stored off in step 5b.

10. HBA writes a Message Signaled Interrupt (MSI-X) based on the completion shadow data.

11. Storage controller writes queue index head pointer to the drive Completion Queue Head Doorbell register.

12. Host writes queue index head pointer to the HBA Completion Queue Head Doorbell register. This completes the read data transaction.

[00118] Referring now to Figure 36, a typical data write transaction flow (written data transferred from Host to storage device) in an example embodiment is illustrated. The basic sequence of processin operations for handlin a write data transaction in an example embodiment is set forth below with the processing operation numbers listed below correspondin to the operation numbers shown in Figure 36:

1. Host writes new queue index tail pointer to HBA Submission Queue Tail Doorbell register. . HBA generates a read request to access a Host memory Submission Queue Shadow entry using queue index head pointer and Submission Queue Shadow base address information.

. Host returns the requested Submission Queue Shadow entry.

. HBA generates a read request to access a Host memory Submission NVMe

command queue entry using queue index head pointer and Submission Queue base address information.

. Host returns the requested Submission Queue NVMe command entry.

a) The Shadow and NVMe entries are used to generate a message that contains the proper fabric information to reach the storage controller and drive.

a.2) The message is encapsulated within a tunnel header and fabric (e.g., Ethernet) header, and sent across the fabric (e.g., Ethernet)

b) The storage controller receives the message and stores off the NVMe command entry and fabric information to be used later in the NVMe I/O.

c) The storage controller tags the upper bits of the Physical Region Page (PRP) addresses of the NVMe command with an I/O context tag and saves off the replaced bits. During the data phase of the I/O, the tag field of the address is used to determine to which I/O and host the data phase request belongs.

. The storage controller writes NVMe Submission Queue doorbell register of the drive and the drive reads the local NVMe entry.

. SSD generates Transaction Layer Packet (TLP) read requests for the NVMe write data.

a) The tag field bits of the address are used to perform an I/O context lookup. The tag field of the address is restored to its original value (stored in step 5 c) and the TLP is directed back to the requesting host based on the fabric information from the I/O context (stored in step 5 b).

b). The HBA receives the TLP read request and stores off fabric information to be used for the TLP read completion.

. Host returns TLP read completion data for the NVMe write. The HBA intercepts the TLP read completion and uses the information stored off in step 7b to direct the TLP back to the proper drive.

. SSD writes NVMe completion to the storage controller Completion queue.

0. The storage controller intercepts the NVMe completion and directs it back to the requesting host's proper completion queue based on the fabric information stored off in step 5b. 11. HBA writes a Message Signaled Interrupt (MSI-X) based on the completion shadow data.

12. Storage controller writes queue index head pointer to the drive Completion Queue Head Doorbell register.

13. Host writes queue index header pointer to the HBA Completion Queue Head

Doorbell register. This completes the write data transaction.

[00119] Figure 37 illustrates a node to node protocol in an example embodiment providing the ability for a pl urali y of data storage access systems to inter-communicate via unicast. multicast, or broadcast data transmissions using the queuing methodologies described herein. In this example embodiment, nodes can be servers hosts, storage controllers, or other fabric- connected components. Requests and completions can be submitted and posted, generally using the first two queues. This allows for in formation to be moved amongst the servers hosts, storage controllers, and other components connected together or sharing the same interconnection netw ork. The protocol is beneficial and useful for a variety of reasons, incl uding:

• Signaling changes in configuration or component status.

• Reporting unexpected conditions or error indications.

• Reporting environmental conditions, such as over-temperature or under-voltage.

• Retrieving/saving data sets of the data storage access system, such as configuration data, firmware, logs, debugging information, and other data sets for controlling the operation of the data storage access system of the example embodiments.

• Providing host/server driver, operating system, database, and/or application uses, such as semaphores.

• Providing shared high-speed memory access allowing components to write to and read from authorized locations in another component's storage.

100120] Figure 38 ill ustrates an example embodiment of a component of the data storage access system as used within an existing host server. In particular. Figure 38 illustrates the Host Server Bus Adapter component of an example embodi ment. This component i mplements the queues, tagging, and data transfer for host to array device and or host to host communications.

[00121] Figure 39 is a flow diagram ill ustrating the basic processing flow 401 for a particular embodi ment of a method for accessing multiple storage devices from multiple hosts without use of remote direct memory access (RDMA). As shown, an example embodiment includes: providing a data store switch fabric enabling data communications betw een a data storage access system and a plurality of compute nodes, each compute node having integrated compute capabilities, data storage, and a network interface controller (Host NIC) (processing block 410); providing a plurality of physical data storage devices (processing block 420); providing a host bus adapter (HBA) in data communication with the plurality of physical data storage devices and the plurality of compute nodes via the data store sw itch fabric, the HBA including at least one submission queue and a corresponding shadow queue (processing block 430); receiving an input output ( 1 0) request from the plurality of compute nodes ( processing block 440); including an element of the I/O request to the at least one submission queue ( processing block 450); and including additional information related to the element of the at least one submission queue to the corresponding shadow queue (processing block 460).

[00122] Figure 40 show s a diagrammatic representation of a machine in the example form of a mobile computing and or communication system 700 within which a set of instructions when executed and or processing logic when activated may cause the machine to perform any one or more of the methodologies described and or claimed herein. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g.. netw orked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) netw ork environment. The machine may be a server computer, a personal computer (PC), a laptop computer, a tablet computing system, a Personal Digital Assistant ( PDA), a cellular telephone, a smartphone. a web appliance, a set-top box (STB), a netw ork router, sw itch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) or activating processing logic that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine^" can also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions or processing logic to perform any one or more of the methodologies described and or claimed herein.

[00123] The example mobile computing and or communication system 700 includes a data processor 702 (e.g.. a System-on-a-Chip ( SoC). general processing core, graphics core, and optionally other processing logic) and a memor 704, which can communicate with each other via a bus or other data transfer system 706. The mobile computing and or communication system 700 may further include various input output ( I O) devices and or interfaces 710, such as a touchscreen display, an audio jack, and optionally a network interface 7 12. In an example embodiment, the netw ork interface 7 12 can include a standard wired netw ork interface, such as an Ethernet connection, or one or more radio transceivers configured for compatibility with any one or more standard w ireless and or cellular protocols or access technologies (e.g.. 2nd (2G). 2.5, 3rd ( 3G), 4th (4G) generation, and future generation radio access for cellular systems. Global System for Mobile communication (GSM), General Packet Radio Services (GPRS). Enhanced Data GSM Environment ( EDGE ). Wideband Code Division Multiple Access ( WC DMA). LTE. CDMA2000, WLAN, Wi reless Router (WR) mesh, and the like). Network interface 7 1 2 may also be configured for use with various other wired and or wireless communication protocols, includin TCP/IP, UDP, SIP, SMS, RTP, WAP, CDMA, TDMA, UMTS, UWB, W'i Fi. WiMax, Bluetooth, IEEE 802. 1 lx, and the like. In essence, network interface 7 1 2 may include or support virtually any wired and or wireless communication mechanisms by which information may travel betw een the mobile computing and/or communication system 700 and another computing or communication system via network 7 14. Sensor logic 720 provides the sensor hardware and or software to capture sensor input from a user action or system event that is used to assist i n the con figuration of the data storage system as described above.

[00124] The memory 704 can represent a machine-readable medium on which is stored one or more sets of instructions, software, firmware, or other processing logic ( e.g.. logic 708) embodying any one or more of the methodologies or functions described and or claimed herein. The logic 708, or a portion thereof, may also reside, completely or at least partially w ithin the processor 702 during execution thereof by the mobile computing and. or communication system 700. As such, the memory 704 and the processor 702 may also constitute machine-readable media. The logic 708, or a portion thereof, may also be configured as processing logic or logic, at least a portion of which is partially implemented in hardw are. The logic 708, or a portion thereof, may further be transmitted or recei ved over a netw ork 7 14 via the netw ork interface 7 1 2. While the machine-readable medium of an example embodi ment can be a single medi um, the term "machine- readable medium" should be taken to include a single non-transitory medium or multiple non- transitory media ( e.g.. a centralized or distributed database, and or associated caches and computing systems) that store the one or more sets of instructions. The term " machine-readable medium" can also be taken to include any non-transitory medium that is capable of storing, encoding or carrying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the various embodiments, or that is capable of storing, encoding or carrying data structures utilized by or associated with such a set of instructions. The term "machine-readable medium" can accordingly be taken to include, but not be l imited to, solid-state memories, optical media, and magnetic media.

100125| Applications that may include the apparatus and systems of various embodi ments broadly include a variety of electronic devices and computer systems. Some embodiments i mplement functions in two or more speci fic interconnected hardw are modules or devices with related control and data signals communicated betw een and through the modules, or as portions of an application-speci fic integrated ci rcuit. Thus, the example system is applicable to softw are, firmw are, and hardw are implementations. 100126| In example embodiments, a node configured by an application may constitute a

"module" that is configured and operates to perform certain operations as described herein. In other embodiments, the "module" may be implemented mechanically or electronically. For example, a module may comprise dedicated circuitry or logic that is permanently configured (e.g.. within a special-purpose processor) to perform certain operations. A module may also comprise programmable logic or circuitry (e.g., as encompassed within a general-purpose processor or other programmable processor) that is temporarily configured by software to perform certain operations. It will be appreciated that the decision to implement a module mechanically, in the dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations. Accordingly, the term "module" should be understood to encompass a functional entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired) or temporarily configured (e.g., programmed) to operate in a certain manner and/or to perform certain operations described herein.

[00127] While the machine-readable medium 704 or 708 is shown in an example embodiment to be a single medium, the term "machine-readable medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term "machine- readable medium" shall also be taken to include any non-transitory medium that is capable of storing, encoding or embodying a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies described herein. The term "machine-readable medium" shall accordingly be taken to include, but not be limited to, solid- state memories, optical media, and magnetic media.

[00128] As noted, the software and/or related data may be transmitted over a network using a transmission medium. The term "transmission medium" shall be taken to include any medium that is capable of storing, encoding or carrying instructions for transmission to and execution by the machine, and includes digital or analog communication signals or other intangible media to facilitate transmission and communication of such software and/or data.

[00129] The illustrations of embodiments described herein are intended to provide a general understanding of the structure of various embodiments, and they are not intended to serve as a complete description of all the elements and features of components and systems that might make use of the structures described herein. Many other embodiments will be apparent to those of ordinary skill in the art upon reviewing the description provided herein. Other embodiments may be utilized and derived, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. The figures herein are merely representational and may not be drawn to scale. Certain proportions thereof may be exaggerated, while others may be minimized. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

[00130] The description herein may include terms, such as "up", "down", "upper", "lower",

"first", "second", etc. that are used for descriptive purposes only and are not to be construed as limiting. The elements, materials, geometries, dimensions, and sequence of operations may all be varied to suit particular applications. Parts of some embodiments may be included in, or substituted for, those of other embodiments. While the foregoing examples of dimensions and ranges are considered typical, the various embodiments are not limited to such dimensions or ranges.

[00131] The Abstract is provided to allow the reader to quickly ascertain the nature and gist of the technical disclosure. The Abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims.

[00132] In the foregoing Detailed Description, various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments have more features than are expressly recited in each claim. The following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment.

[00133] Thus, as described herein, a method and apparatus for accessing multiple storage devices from multiple hosts without use of RDMA are disclosed. Although the disclosed subject matter has been described with reference to several example embodiments, it may be understood that the words that have been used are words of description and illustration, rather than words of limitation. Changes may be made within the purview of the appended claims, as presently stated and as amended, without departing from the scope and spirit of the disclosed subject matter in all its aspects. Although the disclosed subject matter has been described with reference to particular means, materials, and embodiments, the disclosed subject matter is not intended to be limited to the particulars disclosed; rather, the subject matter extends to all functionally equivalent structures, methods, and uses such as are within the scope of the appended claims.

Claims

Claims What is claimed is:

1. A data storage access system comprising:

a data store switch fabric enabling data communications between the data storage access system and a plurality of compute nodes, each compute node having integrated compute capabilities, data storage, and a network interface controller (Host NIC);

a plurality of physical data storage devices; and

a host bus adapter (HBA) in data communication with the plurality of physical data storage devices and the plurality of compute nodes via the data store switch fabric, the HBA to receive and process input output ( 1 0) requests from the plurality of compute nodes, the HBA including at least one submission queue and a corresponding shadow queue, the at least one submission queue including an element of an I/O request, the corresponding shadow queue including additional information related to the element of the at least one submission queue.

2. The system of claim 1 w herein the plurality of physical data storage devices is a plurality of solid-state drives (SSDs).

3. The system of claim 1 wherein the data store switch fabric is implemented using a protocol from the group consisting of: Ethernet. Non-Volatile Memory Express (NVM Express ), and Peripheral Component Interconnect Express (PCIe).

4. The system of claim 1 wherein the data store sw itch fabric enables data communications betw een the plurality of compute nodes and the plurality of physical data storage devices in a manner to emulate a direct data connection.

5. The system of claim 1 wherein the data storage access system includes a set of management queues and a set of data path queues.

6. The system of claim 1 wherein the data storage access system includes a context cache to cache context information to instruct the data storage access system how to present the ending status of an operation.

7. The system of claim 1 wherein the data storage access system includes at least one submission queue and a corresponding shadow queue, the shadow queue including information indicative of a destination Media Access Control (MAC) address.

8. The system of claim 1 wherein the data storage access system includes at least one submission queue and a corresponding shadow queue, the shadow queue including information indicative of a destination Media Access Control (MAC) address, which can address a single component via a unicast, multiple components via a multicast, or all components via a broadcast.

9. The system of claim 1 wherein the data storage access system includes at least one submission queue and a corresponding shadow queue, the shadow queue including information indicative of a priority of a request.

10. The system of claim 1 wherein the host bus adapter (HBA) is further configured for host to host data communications.

11. A method comprising:

providing a data store switch fabric enabling data communications between a data storage access system and a plurality of compute nodes, each compute node having integrated compute capabilities, data storage, and a network interface controller (Host NIC);

providing a plurality of physical data storage devices;

providin a host bus adapter ( HBA) in data communication with the plurality of physical data storage devices and the plurality of compute nodes via the data store switch fabric, the HBA including at least one submission queue and a corresponding shadow queue; receiving an input output ( 1 0) request from the plurality of compute nodes;

including an element of the I/O request to the at least one submission queue; and including additional information related to the element of the at least one submission queue to the corresponding shadow queue.

12. The method of claim 1 1 wherein the plurality of physical data storage devices is a plurality of solid-state drives (SSDs).

13. The method of claim 1 1 wherein the data store switch fabric is implemented using a protocol from the group consisting of: Ethernet. Non-Volatile Memory Express (NVM Express), and Peripheral Component Interconnect Express (PCIe).

14. The method of claim 11 wherein the data store switch fabric enables data communications between the plurality of compute nodes and the plurality of physical data storage devices in a manner to emulate a direct data connection.

15. The method of claim 11 wherein the data storage access system includes a set of

management queues and a set of data path queues.

16. The method of claim 11 wherein the data storage access system includes a context cache to cache context information to instruct the data storage access system how to present the ending status of an operation.

17. The method of claim 11 wherein the data storage access system includes at least one

submission queue and a corresponding shadow queue, the shadow queue including information indicative of a destination Media Access Control (MAC) address.

18. The method of claim 11 wherein the data storage access system includes at least one

submission queue and a corresponding shadow queue, the shadow queue including information indicative of a destination Media Access Control (MAC) address, which can address a single component via a unicast, multiple components via a multicast, or all components via a broadcast.

19. The method of claim 11 wherein the data storage access system includes at least one

submission queue and a corresponding shadow queue, the shadow queue including information indicative of a priority of a request.

20. The method of claim 1 1 wherein the host bus adapter (HBA) is further configured for host to host data communications.