TWI734895B - Method of aggregating storage, method of nvme-of ssd capacity aggregation and aggregated ethernet ssd group - Google Patents

Abstract

In a method of storage aggregation for NVMe Over Fabrics devices, the method includes: identifying an aggregation group as an aggregated Ethernet SSD comprising a plurality of NVMe-oF SSDs; selecting one of the NVMe-oF SSDs of the aggregation group as a primary NVMe-oF SSD; selecting others of the NVMe-oF SSDs of the aggregation group as secondary NVMe-oF SSDs; and initializing a Map Allocation Table in the primary NVMe-oF SSD with a processor for managing the NVMe-oF SSDs.

Description

Aggregate storage method, cross-structure high-speed non-volatile memory solid Capacity aggregation method for state hard disk and aggregated Ethernet solid state hard disk group [Cross references to related applications]

This application claims the U.S. Provisional Patent Application No. 62/480,113 filed in the U.S. Patent and Trademark Office on March 31, 2017 and the U.S. Provisional Patent Application No. 62/483,913 filed in the U.S. Patent and Trademark Office on April 10, 2017. Priority of the number, all the disclosed contents of the two provisional patent applications are incorporated into this case for reference.

Some embodiments of this disclosure generally relate to a system and method for aggregating multiple memory hard disks (such as Ethernet solid state hard disks) into a single large logical capacity for the host.

Solid State Drive (SSD) has undergone rapid development and has now become the first choice for storage components for modern information technology infrastructure, thus replacing traditional hard disk drives (HDD). The latency of the solid state drive is extremely low, and Provide high data read/write throughput and reliable data storage capacity.

Cross-structure high-speed non-volatile memory (Non-Volatile-Memory-express-over-Fabric, NVMe-oF) is an emerging technology that enables hundreds or thousands of cross-structure high-speed non-volatile memory devices (such as cross- Structure High-speed non-volatile memory solid state drives) are connected to each other through a network structure (such as Infinite Bandwidth (IB), Fibre Channel (FC), Ethernet). The cross-structure high-speed non-volatile memory communication protocol can be used to implement remote direct attached storage (Direct Attach Storage, rDAS). In this way, a large number of solid state drives can be connected to the remote host. The cross-structure high-speed non-volatile memory communication protocol uses the Remote Direct Memory Access (RDMA) communication protocol to provide reliable high-speed non-volatile memory commands, data and responses. The transmission protocols that provide remote direct memory access services include iWARP, RoCE v1, and RoCE v2.

The cross-structure high-speed non-volatile memory interface allows a large number of solid state drives to connect to remote hosts. In each cross-structure high-speed non-volatile memory solid state drive, the driver is usually executed on the remote host as usual. For some applications, the storage capacity provided by a single solid-state drive may not be enough.

Some embodiments of the present disclosure provide a method for aggregating multiple solid-state hard disks, which enables the host to treat these solid-state hard disks as a single large-capacity logical volume, and provides a network structure for achieving this method.

According to some embodiments, in a storage aggregation method for a cross-structure high-speed non-volatile memory device, the above-mentioned method includes: identifying the aggregation group as an aggregation group including a plurality of cross-structure high-speed non-volatile memory solid state drives. Ethernet solid state drive; select one of the cross-structure high-speed non-volatile memory solid state drives in the above aggregation group as the main cross-structure high-speed non-volatile memory solid state drive; select the other in the above aggregation group Cross-structure high-speed non-volatile memory solid state drive as a secondary cross-structure high-speed non-volatile memory solid state drive; and initialize the mapping configuration table in the main cross-structure high-speed non-volatile memory solid state drive with a processor , In order to manage the cross-structure high-speed non-volatile memory solid state drive.

According to some exemplary embodiments, the initialization of the mapping configuration table by the processor is performed under the guidance of the storage administrator connected to the aggregation group.

According to some exemplary embodiments, for each cross-structure high-speed non-volatile memory solid state drive in the aforementioned aggregation group, the above mapping configuration table includes the capacity of the cross-structure high-speed non-volatile memory solid state drive, and the cross-structure high-speed non-volatile memory solid state drive. The address of the non-volatile memory solid state drive and the remaining capacity of the cross-structure high-speed non-volatile memory solid state drive.

According to some example embodiments, the above-mentioned method further includes providing the above-mentioned address of the main cross-structure high-speed non-volatile memory solid state drive to the user application, so as to enable data transmission between the aggregation group and the user application .

According to some example embodiments, the above method further includes: receiving an Admin command from a host connected to the aggregation group on the main cross-structure high-speed non-volatile memory solid state drive; determining whether the data corresponding to the above management command is Store only Exist on the above-mentioned main cross-structure high-speed non-volatile memory solid-state drive, or stored on one or more secondary cross-structure high-speed non-volatile memory solid-state hard drives; when the above data is stored in one or more secondary When the cross-structure high-speed non-volatile memory solid state drive is used, the above management commands are divided into one or more management sub-commands corresponding to one or more secondary cross-structure high-speed non-volatile memory solid state drives; Data transfer to the host; receive sub-commands from one or more secondary cross-structure high-speed non-volatile memory solid state drives to complete the project; and create and transfer the completed project from the above-mentioned main cross-structure high-speed non-volatile memory solid state drive To the host.

According to some example embodiments, the above method further includes: receiving one or One management sub-command of multiple management sub-commands; according to the above-mentioned management sub-command, determine whether to transfer the above-mentioned data from the corresponding secondary cross-structure high-speed non-volatile memory solid state drive to the above-mentioned main cross-structure high-speed non-volatile memory Solid-state hard drive; establish and complete the project; and transfer the above-mentioned completed project to the above-mentioned main cross-structure high-speed non-volatile memory solid-state drive.

According to some example embodiments, the above method further includes: receiving a command to create a namespace (Namespace) or a command to delete a namespace on the above-mentioned main cross-structure high-speed non-volatile memory solid state drive; refer to the above-mentioned main cross-structure high-speed non-volatile memory The above-mentioned mapping configuration table of the non-volatile memory solid state drive; when the above command is to create a namespace, in the above-mentioned main cross-structure high-speed non-volatile memory solid state drive and/or in one or more secondary cross-structure high-speed Non-volatile memory solid state hard Configure the capacity in the disk, or when the above command deletes the namespace, recover the main cross-structure high-speed non-volatile memory solid state drive and/or one or more secondary cross-structure high-speed non-volatile memory solid state drives Correspond to one; and update the above mapping configuration table.

According to some embodiments, the above method further includes: receiving a read/write (Read/Write) command on the above-mentioned main cross-structure high-speed non-volatile memory solid state drive; querying the above-mentioned main cross-structure high-speed non-volatile memory solid state drive The mapping configuration table of the hard disk; create one or more read/write subcommands; send the above one or more read/write subcommands to one or more secondary cross-structure high-speed non-volatile memories respectively Solid state drive; according to the above read/write command, between the host and the above-mentioned main cross-structure high-speed non-volatile memory solid state drive and/or one or more secondary cross-structure high-speed non-volatile memory solid state drives Transfer data between; and transfer the above-mentioned data to the host after the completion.

According to some embodiments, the above method further includes: receiving the above one or the above on a corresponding secondary cross-structure high-speed non-volatile memory solid state drive of the one or more secondary cross-structure high-speed non-volatile memory solid state drives. One of the multiple read/write sub-commands; captures the transmission information corresponding to the read/write sub-command; from the above-mentioned corresponding secondary cross-structure high-speed non-volatile memory solid state hard disk The disk sends a read/write request to the host; and after the data transmission corresponding to the above-mentioned read/write subcommand is completed, the completed item is sent to the main cross-structure high-speed non-volatile memory solid state drive.

According to some example embodiments, a cross-structure high-speed non-volatile memory A capacity aggregation method for cross-structure high-speed non-volatile memory solid state drives in an Ethernet solid-state drive group. The above method includes: identifying a plurality of cross-structure high-speed non-volatile memory solid state drives in the aggregation group ; Designate one of the cross-structure high-speed non-volatile memory solid state drives as the primary cross-structure high-speed non-volatile memory solid state drive; and assign the remaining cross-structure high-speed non-volatile memory solid state drives as the secondary Cross-structure high-speed non-volatile memory solid state drives, among which the only cross-structure high-speed non-volatile memory solid state drive visible in the host's host driver is the main cross-structure high-speed non-volatile memory solid state drive.

According to some embodiments, the above method further includes maintaining the mapping configuration table of the main cross-structure high-speed non-volatile memory solid state drive according to the command received from the host computer for the main cross-structure high-speed non-volatile memory solid state drive, wherein The mapping configuration table indicates the logical block address of the main cross-structure high-speed non-volatile memory solid state drive and one or more secondary cross-structure high-speed non-volatile memory solid state drives in the aggregation group. , LBA) space.

According to some embodiments, the above method further includes initializing the mapping configuration table with the processor, and assigning it as one of the main cross-structure high-speed non-volatile memory solid state drives according to the designation To set the aggregation group.

According to some embodiments, the above method further includes gathering the secondary cross-structure high-speed non-volatile memory solid state drive and the above-mentioned main cross-structure high-speed non-volatile memory according to the command received from the host computer. The capacity of the volatile memory solid state drive makes the multiple cross-structures of the above-mentioned aggregation group high-speed The non-volatile memory solid state drive appears to the host as a single aggregate logical capacity.

According to some embodiments, the above method further includes: using the above-mentioned main cross-structure high-speed non-volatile memory solid state drive to allocate capacity to one or more secondary cross-structure high-speed non-volatile memory solid state drives; and using the above Mainly across the structure of high-speed non-volatile memory solid state hard disks to record the allocated capacity and the associated and mapped logical block address range in the mapping configuration table.

According to some embodiments, the above-mentioned method further includes using the above-mentioned main cross-structure high-speed non-volatile memory solid state drive to over-allocate the secondary cross-structure high-speed non-volatile memory solid state drive and the above-mentioned main cross-structure high-speed non-volatile memory solid state drive. The aggregate capacity of the hard disk.

According to some embodiments, the above method further includes: receiving a command from the host on the main cross-structure high-speed non-volatile memory solid state drive; dividing the command into a plurality of sub-commands, and each sub-command corresponds to a corresponding sub-command. Each of the cross-structure high-speed non-volatile memory solid state drives; and the sub-commands from the above-mentioned main cross-structure high-speed non-volatile memory solid state drive to the corresponding secondary cross-structure high-speed non-volatile memory Solid state drive.

According to some embodiments, the above method further includes directly transmitting data from the corresponding secondary cross-structure high-speed non-volatile memory solid state drive to the host based on individual sub-commands.

According to some embodiments, the above-mentioned method further includes: receiving on the corresponding secondary cross-structure high-speed non-volatile memory solid state drive from the above-mentioned main cross-structure high-speed Individual sub-commands of the non-volatile memory solid state drive; execute the task corresponding to the individual sub-command; and after the task is completed, send the individual sub-command from the corresponding secondary cross-structure high-speed non-volatile memory solid state drive to complete the project to the above Mainly cross-structure high-speed non-volatile memory solid state drives.

According to some embodiments, the above method further includes: using the above-mentioned main cross-structure high-speed non-volatile memory solid state drive to maintain the sub-command content table; in the above-mentioned main cross-structure high-speed non-volatile memory solid state drive, receiving data from the secondary The sub-command of the structured high-speed non-volatile memory solid state drive completes the project; and the project is completed according to the received sub-command, and the execution of the sub-command is tracked by the above-mentioned main cross-structured high-speed non-volatile memory solid state drive.

According to some example embodiments, an aggregated Ethernet solid-state drive group is provided, including: an Ethernet solid-state drive base; an Ethernet switch on the Ethernet solid-state drive base to communicate with Host driver communication; processor, coupled to the Ethernet switch; Peripheral Component Interconnect Express (PCIe) switch, coupled to the baseboard management controller; multiple cross-structure high-speed non-volatile Memory solid state drives, including: main cross-structure high-speed non-volatile memory solid state hard drives; and multiple secondary cross-structure high-speed non-volatile memory solid state drives, including Ethernet switches and shortcut peripherals The private communication channel of the interconnected standard switch is connected to the main cross-structure high-speed non-volatile memory solid state drive, of which only the above-mentioned main cross-structure high-speed non-volatile memory solid state drive is visible to the host, and Among them, the above-mentioned baseboard management controller is used to determine which of the above-mentioned main components of the cross-structure high-speed non-volatile memory solid-state hard disk is initially determined. Structure high-speed non-volatile memory solid state drive.

Based on the above, because there is only a single primary Ethernet SSD for processing all cross-structure high-speed non-volatile memory communication protocols, it also tracks whether all related subcommands through the secondary Ethernet SSD have been completed , And ensure that the secondary Ethernet SSD is invisible to the host, so the aggregated group of the Ethernet SSD is displayed to the host as a single large logical capacity.

100, 500: system architecture

110: Ethernet solid state drive

110s: Secondary Ethernet SSD

110p: Main Ethernet solid state drive

120: Ethernet Solid State Drive Dock

130: private communication channel

132: Subcommand

134: Complete the project

135: control plane

140: Fast Peripheral Interconnection Standard Bus

150: baseboard management controller

160: Ethernet switch

170: Machine cross-structure high-speed non-volatile memory driver

180: Command

182: Complete the project

185: Information

190: host

230: Ethernet rack

240: rack top switch

300: mapping configuration table

311: Capacity (hundred megabytes)

312: Transmission address (MAC/IP)

313: Ethernet SSD Index

314: Mapping address namespace logical block address range

400, 700, 800, 900, 1000, 1100, 1200: flow chart

530: Ethernet communication channel

600: form

610: Command Mark

620: Subcommand#

630: Cumulative error status

640: Command ID

S410, S420, S430, S440: steps

S710, S720, S730, S740, S750, S760, S770, S780, S790: steps

S810, S820, S830, S840, S811, S821, S831, S841: steps

S910, S920, S930, S940, S950, S960, S970, S980, S990: steps

S1010, S1020, S1030, S1040, S1050: steps

S1110, S1120, S1130, S1140, S1150: steps

S1210, S1220, S1230, S1240, S1250, S1260, S1270: steps

In order to make the present invention more comprehensible, the following specific examples are given in conjunction with the accompanying drawings to describe in detail as follows. In the accompanying drawings: A cross-structure high-speed non-volatile memory of multiple clustered Ethernet solid-state drives in an Ethernet solid-state drive chassis 100 block diagram of the system architecture of Ethernet solid-state drive (Ethernet SSD, eSSD) storage .

FIG. 2 is a block diagram showing several Ethernet SSD bases shown in FIG. 1 connected to each other on an Ethernet SSD rack according to an embodiment of the present disclosure.

FIG. 3 is an example of a map allocation table (Map Allocation Table) maintained by a main Ethernet solid state drive according to an embodiment of the disclosure.

FIG. 4 is a flowchart showing the initialization of the mapping configuration table according to an embodiment of the present disclosure.

FIG. 5 is a diagram showing a plurality of aggregated Ethernet solid state drives in a plurality of Ethernet solid state drive bays located in a plurality of racks according to an embodiment of the present disclosure; A block diagram of the system architecture for cross-structure high-speed non-volatile memory Ethernet solid-state hard disk storage.

Figure 6 shows an example command content of a table.

FIG. 7 is a flowchart showing the main Ethernet SSD processing management (Admin) command according to an embodiment of the disclosure.

FIG. 8 is a flowchart of executing a Namespace Create and Delete command according to an embodiment of the disclosure.

FIG. 9 is a flowchart illustrating a read/write (Read/Write) command executed under the control of a main Ethernet solid state drive according to an embodiment of the present disclosure.

FIG. 10 is a flowchart showing the execution and management sub-commands of the secondary Ethernet SSD according to an embodiment of the disclosure.

FIG. 11 is a flowchart showing the execution of read/write subcommands by the secondary Ethernet solid-state drive according to an embodiment of the disclosure.

FIG. 12 is a flowchart of data transmission and sub-command completion synchronization in a secondary Ethernet solid-state drive according to an embodiment of the disclosure.

By referring to the detailed description of the embodiments and the accompanying drawings below, it is easier to understand the inventive concept features and implementation methods of the present invention. The embodiments of the present invention will be described below in conjunction with the accompanying drawings, and the same reference numerals represent the same components throughout the text. However, the present invention can be exemplified in many different forms and should not be regarded as limited to the embodiments described herein. To be precise, these embodiments are provided as examples, in order to make this disclosure more thorough and complete, and to provide common knowledge in the field. The author fully conveys the views and characteristics of this disclosure. Accordingly, for a person with ordinary knowledge in the field, procedures, elements, and techniques that are not necessary for a complete understanding of the viewpoints and features of the present invention may not be explained. Unless otherwise specified, in the accompanying drawings and the description, the same symbols are used to refer to similar elements, so the description will not be repeated. In the drawings, for clarity, the relative dimensions of components, layers, and regions can be enlarged.

In the following description, for explanatory purposes, several specific details are listed to provide a comprehensive understanding of the various embodiments. However, it is obvious that various embodiments can be implemented without these specific details or providing one or more equivalent configurations. In other cases, the known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the various embodiments.

It should be understood that although terms such as "first", "second", and "third" may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, Regions, layers and/or sections should not be limited by these terms. These terms are only used to distinguish individual elements, components, regions, layers or sections. Therefore, without departing from the spirit and scope of the present invention, the first element, member, region, layer or section discussed below may be referred to as a second element, member, region, layer or section.

In this article, for the convenience of explanation, you can use, for example, "to...below", "below", "lower", "under )”, “above”, “upper” and other spatially relative terms to describe the relationship between one element or feature shown in the diagram relative to another (other) element or feature . It should be understood that these spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientations shown in the drawings. For example, if you flip The device in the drawing is described as being "below" or "below" or "below" other elements, and will be oriented "above" other elements or features at this time. Therefore, the terms "below" and "below" can include both above and below orientations. The device may also have other orientations (for example, a 90-degree rotation or other orientations), and the spatial relativity description used herein should be interpretable accordingly.

It should be understood that when an element, layer, region or member is referred to as being "on", "connected to" or "coupled to" another element, layer, region or member, the element , Layer, region or member can be directly located on, directly "connected to" or directly "coupled to" said another element, layer, region or member or there may be in between. Other one or more elements, layers, regions or components. In addition, it should also be understood that when an element or layer is referred to as being "between" two elements or layers, the element or layer is the only element or layer between the two elements or layers, or there may be other one or Multiple elements or layers.

For the purpose of this disclosure, "at least one of X, Y, and Z" and "select at least one from the group consisting of X, Y, and Z" should be understood as being limited to X, limited to Y, and limited to Z. Or any combination of two or more of X, Y, Z, such as XYZ, XYY, YZ, ZZ. The same numbers refer to the same elements throughout the text. The term "and/or" as used herein includes any and all combinations of one or more of the related listed items.

In the following example, the x-axis, y-axis, and z-axis are not limited to the three axes of the Cartesian coordinate system, but can be explained in a broader sense. For example, the x-axis, y-axis, and z-axis may be perpendicular to each other, or may also represent different directions that are not perpendicular to each other.

The terminology used herein is only used to describe specific embodiments, not to limit the present invention. Unless the context clearly indicates otherwise, the singular form "一 (a, an)" used herein is intended to also include the plural form. It should be understood that when the term "comprises, comprising, includes, including" is used in this specification, it is used to indicate the features, integers, steps, operations, elements and/or components but does not exclude one or more The existence or addition of other features, integers, steps, operations, elements and/or components. The term "and/or" as used herein includes any and all combinations of one or more of the related listed items. When "at least one of" appears before the element list, the expression is intended to modify the description of the entire element list, and is not used to modify a single element in the list.

As used herein, the terms "approximately", "approximately" and other similar terms are used as approximate terms rather than terms used to express degree, and these terms are intended to describe measurement values or calculations known to those with ordinary knowledge in the field. The inherent deviation in the value. Furthermore, the use of "可" when describing embodiments of the present invention means "one or more embodiments of the present invention." As the terms used in this article, "use", "using" and "used" can be regarded as the same as the terms "utilize", "utilizing" and " "Utilized" is synonymous. The term "exemplary" is intended to refer to examples or illustrations.

When a certain embodiment can be implemented in a different way, a specific program sequence different from the described sequence can be executed. For example, two programs described one after another may be executed substantially at the same time, or may also be executed in an order opposite to the described order.

At the same time, any numerical range disclosed and/or quoted here refers to a sub-range of the same exact numerical value included in the inclusive quoted range. For example, the range of "1.0 to 10.0" refers to all sub-ranges that include (and include) all quoted minimum values of 1.0 and quoted maximum values of 10.0. That is, any range between the minimum value 1.0 or more and the maximum value 10.0 or less, such as 2.4 to 7.6. Any maximum numerical limit quoted herein means including all lower numerical limits covered, and any minimum numerical limit quoted in the text means including all higher numerical limits covered. Accordingly, the applicant reserves the right to amend this application, including the scope of the patent of this application, so that it can be clearly quoted to cover sub-scopes within the scope expressly quoted in this article.

The various embodiments are described here, and reference is made to schematically illustrate the embodiment and/or the cross-sectional schematic diagram of the intermediate structure. Therefore, it can be expected that changes in the shape of the drawing due to, for example, manufacturing technology and/or tolerances will occur. Therefore, the disclosed embodiments should not be regarded as limited to the specific shapes shown herein, but include, for example, shape changes caused by manufacturing. For example, an implanted area described as a rectangle usually has rounded or curved features and/or implant concentration gradients at its edges, rather than just a binary change from the implanted area to the non-implanted area. Similarly, the buried area formed by implantation can cause a cloth value to be generated in the area between the buried area and the surface produced by the implantation. Therefore, the area shown in the drawings is schematic in nature, and its shape is not used to show the actual shape of the device area, and is not used to limit the content.

The electronic devices and/or any other related devices or components described herein according to the embodiments of the present invention can utilize any suitable hardware and firmware (such as special application products). Body circuit), software or a combination of hardware, firmware and software. For example, the various components of these devices can be formed on one integrated circuit (IC) chip or on different integrated circuit chips. Furthermore, the various components of these devices can be implemented on flexible printed circuit film, tape carrier package (TCP), printed circuit board (PCB), or formed on a substrate superior. In addition, the various components of these devices can be programs or threads that execute on one or more processors, one or more computing devices, and execute computer program instructions, and can also be integrated with other system components to execute the instructions described in this article. The functions described. Computer program instructions are stored in memory and can be executed in a computing device using standard memory devices (such as random access memory (RAM)). The computer program instructions can also be stored in other non-transitory computer-readable media, such as optical disc drives, flash drives, and so on. Without departing from the spirit and scope of the exemplary embodiments of the present invention, those skilled in the art should know that the functions of different computing devices can be combined or integrated into a single computing device, or the functions of a specific computing device can be allocated To one or more other computing devices.

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those with ordinary knowledge in the field to which the present invention belongs. It should be further understood that terms such as those defined in general dictionaries should be interpreted as having meanings consistent with their meanings in the context of related technologies and/or this specification, and should not be interpreted as idealized or over-formalized The meaning of, unless clearly defined otherwise.

FIG. 1 is a diagram showing a method for including a single Ethernet network according to an embodiment of the present disclosure A block diagram of a cross-structure high-speed non-volatile memory Ethernet solid-state drive (Ethernet SSD, eSSD) storage system architecture block diagram of several clustered Ethernet solid-state drives 110 in the solid-state drive base 120. FIG. 2 is a block diagram illustrating a method of connecting a plurality of Ethernet SSD bases 120 shown in FIG. 1 to each other in an Ethernet SSD rack according to an embodiment of the present disclosure.

As mentioned above, the cross-structure high-speed non-volatile memory interface allows a large number of solid state hard disks 111 to be connected to the remote host 190. In each cross-structure high-speed non-volatile memory solid state drive 110, the driver program is executed on the remote host 190, for example. However, for some applications, the storage capacity provided by a single solid-state drive 110 is insufficient. This type of application is more suitable for a single logical volume with hundreds of megabytes (Terabyte, TB). Accordingly, this type of application program may be more suitable for the embodiments provided in this disclosure. Because the disclosed embodiment provides a large number of individual solid state drives 110, these solid state drives 110 are collectively aggregated into an "aggregation group" and appear as a single logical volume for applications.

For example, 24 16TB solid state drives can be displayed as a single logical 384TB hard drive. Some examples of solid-state drive 110 applications that require mass aggregation include large-scale data mining and analysis, petrochemistry, gas and energy exploration, experimental particle physics, and pharmaceutical drug development. These paradigms require High Performance Computing (HPC), which also requires a large amount of storage capacity and high performance.

Although it is possible to have the solid state drive 110 under the system software layer, and Provide a single logical, expandable volume, but such system software is usually very complex and sophisticated. Such software may require a large number of cross-structure high-speed non-volatile memory drivers to be executed on, for example, the host 190, and therefore consume system resources, such as memory, central processing unit (CPU) cycles, and power. The target solution may use an x86 server or a Redundant Array of Independent Disk-on-Chip (ROC) system to provide a large-capacity single logical volume. However, such solutions are usually complex, expensive, and have a negative impact on performance and energy usage. For example, according to the embodiment of the present invention, the energy consumed by the use of the central processing unit to receive and transmit data is to use a Direct Memory Access (DMA) engine and an Application-Specific Integrated Circuit (Application-Specific Integrated Circuit). Circuit, ASIC), etc. consume several times the energy.

Accordingly, the embodiments of the present invention provide methods and methods that can gather multiple Ethernet solid state drives 110 in an Ethernet cross-structure high-speed non-volatile memory solid state drive in an efficient and cost-effective manner. Architecture.

Please refer to Figure 1, the Ethernet solid state drive 110 is assigned one of the two roles (for example, assigned by the storage administrator), so each Ethernet solid state drive 110 can be used as the main Ethernet Network Solid State Drive (P-eSSD) 110p or Secondary Ethernet Solid State Drive (S-eSSD) 110s. A single primary Ethernet solid state in a single base 120 (or multiple bases 120 in a given rack, or multiple bases 120 in multiple racks 230 distributed over a wide area) The hard disk 110p and a collection of several secondary Ethernet solid-state hard disks 110s together provide remote The flash memory capacity required by the host 190 is used as a single logical hard disk. The Ethernet solid state drive base 120 includes an Ethernet solid state drive 110 and a processor (such as a baseboard management controller (BMC) device 150), and an Ethernet switch 160 for connecting external devices. Although the Ethernet SSD chassis 120 described in the following embodiments is used to refer to a group of cross-structure high-speed non-volatile memory devices, other embodiments of the present invention can also be applied to any other multiple devices. A cross-structure high-speed non-volatile memory device, regardless of the physical enclosure (such as a base, a rack, or a box enclosure) of these devices. Furthermore, although the Ethernet solid state drive 110 is used to describe the cross-structure high-speed non-volatile memory device in the following embodiments, other cross-structure high-speed non-volatile memory devices can also be applied to the present invention.的其实施例。 Its embodiment.

Accordingly, through the corresponding Ethernet switch 160, it is possible to traverse multiple bases 120 in one rack 230, and across multiple racks 230 including multiple bases 120 to gather the Ethernet solid state. Hard 110.

The main Ethernet solid state drive 110p is the only Ethernet solid state drive 110 visible to the remote host cross-structure high-speed non-volatile memory driver 170, so the terminal cross-structure high-speed non-volatile memory communication protocol. The primary Ethernet solid state drive 110p represents itself and all the remaining secondary Ethernet solid state drives 110s in the same aggregation group, which appears to the remote host 190 as a single large aggregate logical capacity. The main Ethernet solid state drive 110p receives all input/output (input/output, I/O) commands 180 from the remote host's cross-structure high-speed non-volatile memory driver 170, and provides command responses (such as completing a project) ) 182 to the remote host 190.

The main Ethernet SSD 110p also maintains the Map Allocation Table (MAT). The mapping configuration table indicates that in the same Ethernet SSD 110 aggregation group, in the main Ethernet The logical block address (LBA) space is divided between the solid state drive 110p and part or all of the secondary Ethernet solid state drive 110s. When the main Ethernet solid state drive 110p receives the input/output command 180, the main Ethernet solid state drive 110p first queries the mapping configuration table (for example, the mapping configuration table 300 of FIG. 3 described below), To determine which Ethernet SSD 110 (primary Ethernet SSD 110p, one or more secondary Ethernet SSD 110s or a combination of the two) can satisfy the input/output command 180 . According to the mapping configuration table, the primary Ethernet SSD 110p then sends the cross-structure high-speed non-volatile memory input/output subcommand 132 to the appropriate secondary Ethernet SSD 110s after appropriate modifications. Collection.

In order to send the subcommand 132, after the power is connected, the main Ethernet solid-state drive 110p will be connected to the Peripheral Component Interconnect Express (PCIe) bus 140 and with various secondary Ethernet A private Ethernet-remote direct memory access (Ethernet-RDMA) connection (or an appropriate dedicated communication channel) 130 is established on the control plane 135 of the solid state drive 110s. The private Queue-Pair (QP) communication channel 130 is used by the primary Ethernet SSD 110p to send input/output commands (such as subcommands 132) to the secondary Ethernet SSD 110s , And used to receive completed items 134 from the secondary Ethernet SSD 110s. The private communication channel 130 may be an Ethernet network, and can allow data to be transmitted through the Ethernet switch 160. However, private communication The channel 130 can also be a channel established based on the Quick Peripheral Interconnection Standard, so that data can be transmitted through the Quick Peripheral Interconnection Standard Exchanger. That is, all the Ethernet solid state drives 110 may use two or more communication modes to communicate with each other. For example, an Ethernet channel is usually used for data transmission, a standard channel for quick peripheral interconnection can be used for management, and any one of the channels can be used as a private communication channel 130.

The secondary Ethernet solid state drive 110s is a general cross-structure high-speed non-volatile memory solid state drive 110, which is only used to send data to the remote host 190 through the cross-structure high-speed non-volatile memory communication protocol Or receive data from the remote host 190. The data transmission with the remote host 190 is directly accomplished by using the remote direct memory access read service and the remote direct memory access write service. The secondary Ethernet SSD 110s receives commands (such as subcommand 132) from the primary Ethernet SSD 110p, but does not directly receive commands from the remote host cross-structure high-speed non-volatile memory driver 170 . The secondary Ethernet solid state drive 110s sends the subcommand completion item 134 to the primary Ethernet solid state drive 110p to indicate that the subcommand 132 is completed, but not to the remote host 190.

The main Ethernet solid-state drive 110p handles all cross-structure high-speed non-volatile memory communication protocol terminals, handles all host commands and completion queues (such as submission queue/completion queue (Submission Queue/Completion Queue, SQ/ CQ)), and it is visible to the remote host cross-structure high-speed non-volatile memory driver 170 running on the remote host launcher. When the remote host driver 170 sends a cross-structure high-speed non-volatile memory management (Admin) command 180 or enters / Output command 180, command 180 is sent to the main Ethernet solid state drive 110p, and all management commands 180 are executed by the main Ethernet solid state drive 110p. However, the input/output command 180 can be transmitted between multiple Ethernet solid state drives 110.

The main Ethernet solid state drive 110p can also perform its own data transmission and sharing according to the input/output command 180. The primary Ethernet SSD 110p then waits for the completion of all sub-commands and items 134 have arrived (for example, from the collection of the secondary Ethernet SSD 110s) private communication channel 130 before sending the corresponding original command The 180 command completes the item 182 to the remote host 190.

The main Ethernet solid-state drive 110p also maintains the "command content" of each command executed in the command content table (for example, see FIG. 6). The main Ethernet SSD 110p uses this command content to track the execution status of the subcommand 132, the data transmission status, and any error status. When all the subcommands 132 are completed, the command response/completion item 182 will be provided to the remote host 190, and the command content table will be de-configured.

Please refer to FIG. 2, a plurality of Ethernet solid-state hard disk bases 120 can be connected together in the Ethernet solid-state disk rack 230, wherein the top-of-rack (TOR) switch 240 is It is used to enable the connectivity between the multiple bases 120 in the common rack 230. Similarly, multiple racks 230 located in different geographical locations can also be connected to each other through individual rack top switches 240, which are directly connected to each other or connected to each other through an external switch. These Ethernet racks 230 can be located in the same data center building, or they can be distributed in different geographic areas Domain.

As mentioned above, the mechanism provided by the embodiments of the present disclosure can aggregate multiple Ethernet cross-structure high-speed non-volatile memory solid state drives (Ethernet solid state drive) 110 to display as a single large-capacity cross-structure High-speed non-volatile memory solid state drive. The Ethernet SSD 110 can be located in a single base 120, in multiple bases 120 of a single rack 230, or even distributed on multiple Ethernet racks 230, and each Ethernet network Each rack 230 has a plurality of bases 120. One of the Ethernet solid state drives 110 is assigned the role of the primary Ethernet solid state drive (P-eSSD) 110p. The other Ethernet solid state drives 110 are assigned the role of secondary Ethernet solid state drives (S-eSSD) 110s. Although the secondary Ethernet solid-state drive 110s and the remote host initiator perform direct data transfer, the secondary Ethernet solid-state drive 110s receives the subcommand 132 from the primary Ethernet solid-state drive 110p and completes The subcommand 132 and the completion item 134 of the subcommand 132 are returned to the main Ethernet solid state drive 110p. Accordingly, in this embodiment, without sacrificing any storage bandwidth, the capacity can be aggregated to act as an effective single Ethernet solid state drive.

FIG. 3 is an example of a "mapping configuration table" 300 maintained by a main Ethernet solid-state drive 110p according to an embodiment of the present disclosure. 4 is a flowchart 400 showing the initialization of the mapping configuration table 300 according to an embodiment of the disclosure.

3 and 4, as described above, this embodiment uses two types of Ethernet SSD 110 (for example, primary Ethernet SSD 110p and secondary Ethernet SSD 110s ). Primary Ethernet SSD 110p and secondary B Both of the Ethernet SSD 110s use a cross-structure high-speed non-volatile memory communication protocol to provide storage services to the host 190. The main Ethernet SSD 110p maintenance table (for example, the mapping configuration table (MAT) 300), this table contains the secondary Ethernet SSD 110s in the aggregation group of the Ethernet SSD 110 For the host 190, the aggregation group of the Ethernet solid-state drive 110 is a single logical volume.

The mapping configuration table 300 can be initialized as the main Ethernet solid state drive 110p in the same chassis 120 via the baseboard management controller 150. The baseboard management controller 150 can manage the Ethernet base 120 and components, such as the Ethernet switch 160 and the Ethernet solid state drive 110. The baseboard management controller 150 has a standard interface for quick peripheral interconnection and a system management bus (SMBus) interface for system management purposes. At the same time, the baseboard management controller 150 decides to gather those Ethernet SSDs 110 (S410) (for example, under the instruction of the storage manager). When the Ethernet SSD 110 to be aggregated is determined, the baseboard management controller 150 can configure the Ethernet switch 160.

The baseboard management controller 150 initializes the three left columns of the mapping configuration table 300 under the guidance of the storage manager. The baseboard management controller 150 and the storage manager can see and know that all the Ethernet solid state drives 110 in the aggregation group/storage system exist. The known content includes the capacity 311 and address position 312 of each Ethernet SSD 110. The storage administrator can determine which secondary Ethernet SSDs 110s are necessary to form "aggregated Ethernet SSDs" (for example, aggregate groups). The baseboard management controller 150 and the storage administrator will notify or provide the main Ethernet fixed The network address of the state hard disk 110p is given to the user. In this way, the user application corresponding to the remote host's cross-structure high-speed non-volatile memory driver 170 will know where to find the clustered Ethernet Road solid state drive. The baseboard management controller 150 and the storage manager can also select or designate one of the Ethernet SSDs 110 as the main Ethernet SSD 110p (S420), or after the original assignment is made, based on One or more different reasons to change the Ethernet solid state drive 110 as the main Ethernet solid state drive 110p. Next, the baseboard management controller 150 can program the primary and secondary modes of the aggregation group (S430).

The main Ethernet solid state drive 110p can keep a copy of the mapping configuration table 300 on the baseboard management controller 150, and the mapping configuration table 300 stored on the baseboard management controller 150 can be updated regularly. In some embodiments, only the main Ethernet solid state drive 110p includes the official mapping configuration table 300, and the Ethernet solid state drive index 313 of "0" represents the main Ethernet solid state drive 110p. The remaining Ethernet SSD index values are relative to the respective secondary Ethernet SSD 110s.

The main Ethernet SSD 110p terminal host driver 170 is a cross-structure high-speed non-volatile memory communication protocol, and executes all commands 180 issued by the host driver 170. When the command 180 of the host 190 is completed, the main Ethernet solid-state drive 110p uses the "Completion Entry" 182 format to return the completion entry 182 to the host driver 170. Regarding the host command 180, the remote host cross-structure high-speed non-volatile memory driver 170 is completely unaware of the existence of the secondary Ethernet SSD 110s. The main Ethernet solid state drive 110p also maintains The guard submits the queue (SQ), and submits the order completion to the completion queue (CQ).

The main Ethernet SSD 110p updates and maintains the three columns on the right side of the mapping configuration table 300. When the remote host cross-architecture high-speed non-volatile memory driver 170 creates a "Namespace", some flash capacity will be allocated to the namespace. The namespace logical block address range 314 is mapped to the set of Ethernet SSD 110 and recorded in the mapping configuration table 300 maintained by the main Ethernet SSD 110p. The details of this procedure will be explained as follows with reference to FIG. 8.

The main Ethernet SSD 110p can also perform certain initialization procedures. Once the mapping configuration table 300 is initialized in the main Ethernet solid state drive 110p (S440), the main Ethernet solid state drive 110p knows which Ethernet solid state drive 110 is the corresponding secondary Ethernet solid state drive 110s. The primary Ethernet solid state drive 110p then sets up a communication channel 130 with each secondary Ethernet solid state drive 110s in the aggregation group. The communication channel 130 can pass through the Ethernet interface through the Ethernet switch 160 in the base 120, or through the Express Peripheral Interconnection Standard Interface through the Express Peripheral Interconnection Standard Switch in the base 120. If one of the secondary Ethernet SSD 110s is located in a different base 120 in the same rack 230, the communication channel 130 is established through the top-of-rack switch 240. The communication channel 130 in a given base 120 may also be a standard bus 140 via a quick peripheral interconnection.

FIG. 5 is a diagram showing a plurality of gatherings in multiple Ethernet solid-state drive bays 120 that are respectively located in multiple racks 230, including data traveling back and forth according to an embodiment of the present disclosure. A block diagram of the system architecture 500 of the Ethernet solid state drive 110 for cross-structure high-speed non-volatile memory Ethernet solid state drive (eSSD) storage.

Please refer to Figure 5, the primary Ethernet solid state drive 110p can be connected to the secondary Ethernet solid state drive 110s located in the wide area network (Wide Area Network, WAN) to establish an Ethernet through an external network switch and router. Network communication channel 530. The private Ethernet communication channel 530 can be a remote direct memory access queue pair (QP) or can be a proprietary method. The Ethernet communication channel 530 is used to exchange subcommands 132 and associated completions.

FIG. 6 shows an example command content of a table 600.

Please refer to Figure 6, each word command 132 has a command identification (ID) 640 and carries a "command tag" (Command Tag) 610, so when the main Ethernet solid state drive 110p receives the completion item 134, it can use these Complete item 134 to trace the original order 180. When tracing the completion item 134 of the sub-command 132, the "# of Sub-commands" field is decremented, and the received error status is latched with the current status. When the “sub-command #” field is zero, the corresponding command 180 (that is, the upper-level command of the sub-command 132) is completed, and the main Ethernet solid state drive 110p can send the completed item 182 back to the remote host 190. At this time, the main Ethernet solid state drive 110p uses the accumulated error status 630 to generate a completed item, and puts the completed item into the associated completion queue.

FIG. 7 is a flowchart 700 illustrating the processing of the Admin command 180 by the main Ethernet solid-state drive 110p according to an embodiment of the present disclosure.

Please refer to Figure 1 and Figure 7, as mentioned above, each of the main Ethernet solid state hardware The disk 110p maintenance command is submitted to the queue. When the main Ethernet solid state drive 110p receives the executable command 180 (S710), the main Ethernet solid state drive 110p arbitrates and submits the queue, and then selects the command 180 to be executed. The main Ethernet solid state drive 110p executes all high-speed non-volatile memory commands (such as management commands and input/output commands) 180. In other words, although the secondary Ethernet solid-state drive 110s can directly send data to the host 190, the secondary Ethernet solid-state drive 110s will not directly receive commands from the host 190, nor will they send it directly. Finish to host 190. The management commands executed by the primary Ethernet solid-state drive 110p do not require any communication with the secondary Ethernet solid-state drive 110s.

After receiving the command 180, the main Ethernet solid state drive 110P determines the location of the data and determines whether all the data has been accessed (S720). If the main Ethernet solid state drive 110p has all the data, the main Ethernet solid state drive 110p transmits the data to the host 190 (S770).

If the main Ethernet solid state drive 110p determines that it does not have all the data (S720), the main Ethernet solid state drive 110p then consults the mapping configuration table 300 to determine the location of the required data (S730). Once the set of related Ethernet solid state drives 110 is identified, the main Ethernet solid state drive 110p proceeds to the execution of the command 180. When the main Ethernet solid state drive 110p itself has all the required data, the main Ethernet solid state drive 110p transmits data 185. However, when the requested data is scattered in a collection of the primary Ethernet solid-state drive 110p and/or the secondary Ethernet solid-state drive 110s, the primary Ethernet solid-state drive 110p divides the original command 180 into appropriate The number of subcommands 132 (S740). Subcommand 132 The number corresponds to the number of Ethernet SSDs 110 on which the requested data is distributed. Each subcommand 132 corresponds to the requested data part possessed by each Ethernet solid-state drive 110.

The main Ethernet solid state drive 110p will have the appropriate start logical block address (Start LBA, SLBA), the number of blocks (Number of Blocks, NLB), and multiple remote scatter/gather lists (Scatter/Gather Lists, SGLs) are placed in the subcommand 132. The remote distribution/recycling list includes addresses, keys, and the size of the transmission buffer on the remote host 190. The primary Ethernet solid state drive 110p then uses the private queue pair communication channel 130 to send those sub-commands 132 to the respective secondary Ethernet solid state drive 110s (S750) in the command splitting process. And it waits to receive the completed item 134 from each of the individual secondary Ethernet SSD 110s (S760). According to this, the original command 180 is divided into sub-commands 132, and the appropriate Ethernet solid state drive 110 can perform data transmission in parallel, so that the data can be transmitted to the host 190 (S770).

The main Ethernet solid state drive 110p creates command content for the command 180 being executed. Please refer to the description of FIG. 6. The command content is used to track the execution status of the subcommand 132 and any intermediate error status of the subcommand 132 (S780). Once the main Ethernet solid state drive 110p confirms that the management command is completed, the main Ethernet solid state drive 110p transmits the completed item to the host 190 (S790).

Accordingly, the main Ethernet solid state drive 110p receives (S710) and executes all management commands 180 issued by the remote host cross-structure high-speed non-volatile memory driver 170. The main Ethernet SSD 110p may complete multiple or all Partial management command 180 (for example, when the main Ethernet solid-state drive 110p judges (S720) that it alone has all the necessary information to complete the management command). In some cases, the primary Ethernet SSD 110p can retrieve certain information from the secondary Ethernet SSD 110s before completing the management command 180. If the primary Ethernet solid-state drive 110p wants to find some non-user data from the secondary Ethernet solid-state drive 110s, the primary Ethernet solid-state drive 110p will create and send management subcommands 132 to each Other secondary Ethernet solid state drives 110s. The secondary Ethernet solid state drive 110s can use the private communication channel 130 between the two to return any required data and sub-command completion items 134 to the primary Ethernet solid state drive 110p.

FIG. 8 shows a flowchart 800 of executing a Namespace Create and Delete command according to an embodiment of the disclosure.

Referring to FIG. 8, the main Ethernet solid state drive 110p can receive and execute a Namespace Create command (S810) and/or a Delete Admin command (S811). When the main Ethernet solid-state drive 110p receives the namespace creation command (S810), the main Ethernet solid-state drive 110p can query the mapping configuration table 300 (S820), and configure the appropriate from the total available pool Capacity (S830). The newly created namespace can have a flash capacity entirely from the primary Ethernet SSD 110p or entirely from some secondary Ethernet SSD 110s, or the newly created namespace can have a flash capacity from the primary Ethernet The flash capacity of any combination of network solid state drive 110p and secondary Ethernet solid state drive 110s. The main Ethernet solid state drive 110p can then record the allocated capacity and the associated corresponding logical block address range in the mapping configuration table 300 (S840).

When the main Ethernet solid state drive 110p receives the Namespace Delete command (S811) to delete the namespace, the main Ethernet solid state drive 110p queries the mapping configuration table 300 (S821) and retrieves the corresponding (S831), de-allocate the relevant capacity (S841), and update the mapping configuration table 300 accordingly.

Regarding the namespace creation/delete commands executed by the secondary Ethernet SSD 110s, the secondary Ethernet SSD 110s will not directly receive the namespace creation/delete commands. Under normal circumstances, the secondary Ethernet SSD 110s will contain a single namespace representing the total capacity. At an appropriate time, the primary Ethernet solid state drive 110p can send a namespace creation command or a namespace deletion command to the secondary Ethernet solid state drive 110s as a subcommand 132. The secondary Ethernet solid-state drive 110s then executes those commands separately, and returns the corresponding completion item 134 to the primary Ethernet solid-state drive 110p. Such a process is the same as any process of receiving management subcommands from the main Ethernet SSD 110p.

FIG. 9 is a flowchart 900 illustrating a read/write (Read/Write) command executed under the control of the main Ethernet solid-state drive 110p according to an embodiment of the present disclosure.

Please refer to FIG. 9, the main Ethernet solid state drive 110p can receive and execute all input/output commands 180, including read and write commands 180. When the main Ethernet solid state drive 110p receives the read/write command 180 (S910), the main Ethernet solid state drive 110p first queries the mapping configuration table 300 (S920). From the mapping configuration table 300, the main Ethernet SSD 110p identifies where the relevant user data is A collection of Ethernet SSD 110.

As shown in FIG. 6, the main Ethernet solid state drive 110p then creates the command content for the original command (S930). In this way, the main Ethernet solid state drive 110p can track the execution status of the subcommand 132. The primary Ethernet solid state drive 110p then creates a corresponding read/write subcommand 132 (S940), and sends the appropriate subcommand 132 to the appropriate secondary Ethernet solid state drive 110s (S950). The primary Ethernet solid state drive 110p also provides all necessary transmission network related information (such as address) to the secondary Ethernet solid state drive 110s. As part of the subcommand 132, the secondary Ethernet SSD 110s receives the remote host 190 scatter/recycle list containing the remote buffer address, size, and security key.

The data transmission field in the subcommand 132 is appropriately modified to the correct displacement. The secondary Ethernet SSD 110s directly sends data to the buffer (S960) of the remote host 190. When completed, the secondary Ethernet SSD 110s is sent to the primary Ethernet SSD 110p (instead of directly transmitting to the host 190). If necessary, the main Ethernet solid state drive 110p will perform its own data transmission and sharing to the remote host 190 (S960).

Furthermore, each secondary Ethernet solid state drive 110s receives enough information from the primary Ethernet solid state drive 110p to perform any direct data transmission to the remote host 190 (S960). In the cross-structure high-speed non-volatile memory communication protocol, the remote direct memory access transmission service (remote direct memory access read and remote direct memory access write) is used to transfer data from the secondary The Ethernet SSD 110s is transmitted to the remote host 190. The remote host 190 may need to support remote The Shared Receive Queue (SRQ) function of the direct memory access protocol allows multiple secondary Ethernet SSDs 110s to transmit data to the remote host 190 (S960). The remote direct memory access protocol can be implemented on Ethernet/IP/TCP (iWARP), Ethernet/InfiniBand (RoCE v1) or Ethernet/IP/UDP (RoCE v2). In terms of communication between the secondary Ethernet solid state drive 110s and the remote host 190, the secondary Ethernet solid state drive 110s strictly implements data transmission only with the remote host 190 (S960). That is, only remote direct memory access-reading and remote direct memory access-write operations are performed between the secondary Ethernet solid-state drive 110s and the remote host 190 (S960). The subcommand completion item 134 and any non-user data transmission are executed using remote direct memory access transfer operations or other proprietary communication protocols and the main Ethernet solid state drive 110p.

When all the sub-commands are completed (S970), as instructed when the main Ethernet solid-state drive 110p receives all the sub-commands to complete the project, the main Ethernet solid-state drive 110p creates a complete project 182 for the original command 180, And the completion item 182 is transmitted to the appropriate completion queue on the host 190 (S980). The main Ethernet solid state drive 110p then cancels the configuration of the command content (S990).

FIG. 10 is a flowchart 1000 showing the execution of management sub-commands for the secondary Ethernet solid-state drive 110s according to an embodiment of the disclosure.

Referring to FIG. 10, in this embodiment, no secondary Ethernet SSD 110s has directly received the management command 180 or any command from the cross-structure high-speed non-volatile memory driver 170 of the host 190. Instead, it is the primary Ethernet SSD 110p, and only when necessary, sends the management subcommand 132 to the secondary B Ethernet Solid State Drive 110s (S1010). The secondary Ethernet solid-state drive 110s then determines whether any data transmission is required (S1020), and performs any required data transmission with the primary Ethernet solid-state drive 110p on the private communication channel 130 ( S1030). The secondary Ethernet solid state drive 110s is then created (S1040) and the completed item 134 is sent to the primary Ethernet solid state drive 110p (S1050). In other embodiments, the secondary Ethernet solid state drive 110s can use remote direct memory access transfer operations to transmit data and complete items 134 to the primary Ethernet solid state drive 110p.

FIG. 11 is a flowchart 1100 of a read/write command executed by the Ethernet solid-state drive 110s once according to an embodiment of the disclosure.

Referring to FIG. 11, the main task of the secondary Ethernet SSD 110s is to execute read or write word commands 132 (for example, instead of management word commands 132). That is, the secondary Ethernet solid-state drive 110s mainly executes the movement of data to or from the remote host 190 without performing other aspects of communication protocol processing. When the secondary Ethernet solid state drive 110s receives a read/write command (S1110), the secondary Ethernet solid state drive 110s uses the received transmission network information (S1120) to send the remote The direct memory access read or remote direct memory access write request is sent to the remote host 190 (S1130). As part of the subcommand 132, the secondary Ethernet SSD 110s receives the details of the remote buffer address/displacement, size, and security key. When the necessary data transmission is completed (S1140), the secondary Ethernet solid state drive 110s sends a completion item 134 with an appropriate error status to the primary Ethernet solid state drive 110p (S1150).

FIG. 12 is a flowchart 1200 of data transmission and sub-command completion synchronization in the secondary Ethernet solid-state drive 110s according to an embodiment of the present disclosure.

Please refer to Figure 12, in a given host command 180, although the primary Ethernet SSD 110p will complete the item 182 to the host 192, the set of secondary Ethernet SSD 110s can send data to Host 190. After all the data is sent to the remote host 190, the completion item 182 of the given command 180 can be provided to the host. This is because if the completion item 182 of the command 180 is sent to the host 190 one step earlier than the related data, it may cause failure. Defined behavior/error.

As described above, the main Ethernet solid state drive 110p receives the read/write command 180 from the host 190 (S1210), and divides the command 180 into several sub-commands 132 (S1220). Next, each of the secondary Ethernet solid state drives 110s respectively receiving the subcommand 132 from the primary Ethernet solid state drive 110p sends data to the host 190 (S1250). After it is determined that the data transmission is completed (S1260), the secondary Ethernet solid-state drive 110s sends the completed item 134 to the primary Ethernet solid-state drive 110p (S1270). Next, after receiving all the sub-commands from each related secondary Ethernet SSD 110s to complete the item 134 (S1230), the primary Ethernet SSD 110p sends the completed item 182 to the host 190 (S1240) .

Because the aggregated Ethernet solid-state drive data transmission and completion project announcement are scattered on the collection of Ethernet solid-state drives 110, and data and completion synchronization should be achieved. Generally speaking, when a single Ethernet solid state drive 110 performs the two phases of command execution (data transmission + completion announcement), these problems will not occur. However, clustered Ethernet SSDs do not work this way. Therefore, the main Ethernet The solid state drive 110p must wait for all the subcommands from the respective secondary Ethernet solid state drive 110s to complete the item 134 before it can announce the completion item 182 of the command 180. Furthermore, the secondary Ethernet solid state drive 110s must ensure that all data transmissions have been completely and reliably completed before the completion item 134 of the subcommand 132 can be transmitted to the primary Ethernet solid state drive 110p. Such a two-stage synchronization process ensures that at any time, the integrity of the cross-structure high-speed non-volatile memory communication protocol is achievable in the clustered Ethernet solid-state drive.

To sum up, because only a single main Ethernet solid state drive is visible to the host, and this main Ethernet solid state drive performs all the processing operations of the cross-structure high-speed non-volatile memory communication protocol At the same time, it is necessary to track whether the related subcommands issued by all secondary Ethernet SSDs have been completed. Therefore, the aggregation group of Ethernet SSDs is displayed to the host as a single large logical capacity.

100: System Architecture

110s: Secondary Ethernet SSD

110p: Main Ethernet solid state drive

120: Ethernet Solid State Drive Dock

130: private communication channel

132: Subcommand

134: Complete the project

135: control plane

140: Fast Peripheral Interconnection Standard Bus

150: baseboard management controller

160: Ethernet switch

170: Machine cross-structure high-speed non-volatile memory driver

180: Command

182: Complete the project

185: Information

190: host

Claims (20)

A storage aggregation method for a cross-structure high-speed non-volatile memory device, the method comprising: recognizing an aggregation group as an aggregated Ethernet solid-state drive including a plurality of cross-structure high-speed non-volatile memory solid-state drives Disk; select one of the cross-structure high-speed non-volatile memory solid state drives in the aggregation group as the main cross-structure high-speed non-volatile memory solid state drive; select the aggregation group The other ones of the cross-structure high-speed non-volatile memory solid state hard disks are used as secondary cross-structure high-speed non-volatile memory solid state hard disks; and the main cross-structure high-speed non-volatile memory solid state hard disks The Map Allocation Table is initialized to manage the cross-structure high-speed non-volatile memory solid state drive; among them, only the main cross-structure high-speed non-volatile memory solid state drive is visible to the host And wherein the baseboard management controller is used to determine at the beginning of the cross-structure high-speed non-volatile memory solid state drive, which is the main cross-structure high-speed non-volatile memory solid state drive. The method according to the first item of the scope of patent application, wherein the initialization of the mapping configuration table by the processor is performed under the guidance of a storage administrator connected to the aggregation group. The method described in item 1 of the scope of patent application, wherein for each cross-structure high-speed non-volatile memory solid state drive of the aggregation group, the mapping configuration table includes the cross-structure high-speed non-volatile memory The capacity of the solid state drive, the address of the cross-structure high-speed non-volatile memory solid state drive, and the remaining capacity of the cross-structure high-speed non-volatile memory solid state drive. As the method described in item 3 of the scope of patent application, the method further includes providing the address of the main cross-structure high-speed non-volatile memory solid state drive to the user application program, so that the clustering Data can be transferred between the group and the user application. According to the method described in item 1 of the scope of patent application, the method further includes: receiving management from the host connected to the aggregation group on the main cross-structure high-speed non-volatile memory solid state drive ( Admin) command; determine whether the data corresponding to the management command is only stored on the main cross-structure high-speed non-volatile memory solid state drive, or stored in one or more of the secondary cross-structure high-speed non-volatile memory When the data is stored on the one or more secondary cross-structure high-speed non-volatile memory solid-state hard disks, the management commands are divided into corresponding ones respectively Said one or more of said one or more of said secondary cross-structure high-speed non-volatile memory solid state hard disk one or more management sub-commands; transmitting said data to said host; receiving said one or more of said one or more Describe the sub-commands that cross the structure of the high-speed non-volatile memory solid state drive to complete the project; and Create and transfer completed items from the main cross-structure high-speed non-volatile memory solid state drive to the host. As the method described in item 5 of the scope of the patent application, the method further includes: connecting the one or more of the secondary cross-structure high-speed non-volatile memory solid state hard disks to the corresponding secondary cross-structure high-speed non-volatile memory A management sub-command of the one or more management sub-commands is received on the volatile memory solid state disk; according to the management sub-command, it is determined whether the data is to be transferred from the corresponding secondary cross-structure high-speed non-volatile The memory solid state drive is transferred to the main cross-structure high-speed non-volatile memory solid state drive; the completed project is created; and the completed project is transferred to the main cross-structure high-speed non-volatile memory solid state drive. As the method described in item 1 of the scope of patent application, the method further includes: receiving a command to create a namespace (Namespace) or a command to delete a namespace on the main cross-structure high-speed non-volatile memory solid state drive; The mapping configuration table of the main cross-structure high-speed non-volatile memory solid state drive; when the command is to create the namespace, in the main cross-structure high-speed non-volatile memory solid state drive and / Or allocate capacity in one or more of the secondary cross-structure high-speed non-volatile memory solid state hard disks, or when the command is to delete the namespace, retract the primary cross-structure high-speed non-volatile memory Memory solid state drive and/or said one or more of said secondary cross-structure high-speed non-volatile memory The corresponding one of the solid-state hard disk; and updating the mapping configuration table. As the method described in item 1 of the scope of the patent application, the method further includes: receiving a Read/Write command on the main cross-structure high-speed non-volatile memory solid state drive; querying the The mapping configuration table of the main cross-structure high-speed non-volatile memory solid state hard disk; establish one or more read/write subcommands; transmit the one or more read/write subcommands to One or more of the secondary cross-structure high-speed non-volatile memory solid state hard disks; according to the read/write command, the host and the main cross-structure high-speed non-volatile memory solid state hard disk And/or the one or more of the secondary cross-structure high-speed non-volatile memory solid state hard disks; and the data is transmitted to the host after the transmission is completed. As the method described in item 8 of the scope of the patent application, the method further includes: in one or more of the secondary cross-structure high-speed non-volatile memory solid state hard disks corresponding to the secondary cross-structure high-speed non-volatile A read/write sub-command of the one or more read/write sub-commands is received on the memory solid state disk; the transmission information corresponding to the one read/write sub-command is retrieved; The corresponding secondary cross-structure high-speed non-volatile memory solid state drive sends a read/write request to the host; and after the data transmission corresponding to the one read/write subcommand is completed, the The completed item is transferred to the main cross-structure high-speed non-volatile memory solid state drive. A capacity aggregation method for a cross-structure high-speed non-volatile memory solid state drive in a cross-structure high-speed non-volatile memory Ethernet solid-state drive group, the method comprising: identifying a plurality of spans of the aggregation group Structure high-speed non-volatile memory solid state drive; assigning one of the cross-structure high-speed non-volatile memory solid state hard disks as the main cross-structure high-speed non-volatile memory solid state drive; and assigning the cross-structure high-speed The remainder of the non-volatile memory solid state drive is designated as the secondary cross-structure high-speed non-volatile memory solid state drive, and the only cross-structure high-speed non-volatile memory solid state drive visible to the host's host driver is said The main cross-structure high-speed non-volatile memory solid state drive, where only the main cross-structure high-speed non-volatile memory solid state drive is visible to the host, and where the baseboard management controller is used to determine at the beginning Among the cross-structure high-speed non-volatile memory solid state hard disks, which one is the main cross-structure high-speed non-volatile memory solid state hard disk. According to the method described in item 10 of the scope of patent application, the method further includes maintaining the main cross-structure high-speed non-volatile memory according to the command received from the host by the main cross-structure high-speed non-volatile memory solid state drive A mapping configuration table of a memory solid state drive, wherein the mapping configuration table indicates that the main cross-structure high-speed non-volatile memory solid state drive in the aggregation group and one or more of the secondary Cross structure high The logical block address (LBA) space is divided between fast non-volatile memory solid-state drives. According to the method described in item 10 of the scope of patent application, the method further includes initializing the mapping configuration table with a processor to use the cross-structure high-speed non-volatile memory solid state drive as the main cross-structure high-speed One of the non-volatile memory solid-state drives is assigned to set the aggregation group. According to the method described in item 10 of the scope of patent application, the method further includes aggregating the secondary cross-structure high-speed non-volatile memory according to the command received from the host by the main cross-structure high-speed non-volatile memory solid state drive. The capacities of the volatile memory solid state drive and the main cross-structure high-speed non-volatile memory solid state drive are such that a plurality of the cross-structure high-speed non-volatile memory solid state drives in the aggregation group are paired with the The host appears as a single aggregate logical capacity. As the method described in item 10 of the scope of patent application, the method further includes: using the main cross-structure high-speed non-volatile memory solid state drive to allocate capacity to one or more of the secondary cross-structure high-speed non-volatile memory Volatile memory solid state drive; and using the main cross-structure high-speed non-volatile memory solid state drive to record the allocated capacity and the associated mapping logical block address range in the mapping configuration table . As the method described in item 10 of the scope of patent application, the method further includes over-allocating the secondary cross-structure high-speed non-volatile memory solid state drive with the main cross-structure high-speed non-volatile memory solid state drive and The aggregate capacity of the main cross-structure high-speed non-volatile memory solid state hard disk. As the method described in item 10 of the scope of patent application, the method further includes: receiving a command from the host on the main cross-structure high-speed non-volatile memory solid state drive; dividing the command into a plurality of sub Command and each sub-command corresponds to the corresponding one of the secondary cross-structure high-speed non-volatile memory solid state hard disk; The hard disk is transferred to the corresponding secondary cross-structure high-speed non-volatile memory solid state hard disk. According to the method described in item 16 of the scope of the patent application, the method further includes directly transmitting data from the corresponding secondary cross-structure high-speed non-volatile memory solid state drive to the host based on the individual sub-commands. According to the method described in item 16 of the scope of patent application, the method further includes: receiving from the main cross-structure high-speed non-volatile memory on the corresponding secondary cross-structure high-speed non-volatile memory solid-state hard disk The individual sub-commands of the solid state drive; execute the task corresponding to the individual sub-command; and after the task is completed, send the individual sub-command from the corresponding secondary cross-structure high-speed non-volatile memory solid state drive The subcommand completes the project to the main cross-structure high-speed non-volatile memory solid state drive. As the method described in item 18 of the scope of patent application, the method further includes: using the main cross-structure high-speed non-volatile memory solid state drive to maintain a sub-command content table; Receive the sub-command completion item from the secondary cross-structure high-speed non-volatile memory solid state drive on the main cross-structure high-speed non-volatile memory solid state drive; and complete the item according to the received sub-command Item, the main cross-structure high-speed non-volatile memory solid state drive is used to track the execution of the sub-command. An aggregated Ethernet solid-state hard disk group, including: an Ethernet solid-state hard disk base; an Ethernet switch, which is located on the Ethernet solid-state hard disk base, and is used to communicate with the host of the host Driver communication; processor, coupled to the Ethernet switch; Peripheral Component Interconnect Express (PCIe) switch, coupled to the baseboard management controller; and multiple cross-structure high-speed non Volatile memory solid state hard disks, including: main cross-structure high-speed non-volatile memory solid state hard disks; and multiple secondary cross-structure high-speed non-volatile memory solid state hard disks, by including the Ethernet switch The private communication channel of the fast peripheral interconnect standard switch is connected to the main cross-structure high-speed non-volatile memory solid state drive, and only the main cross-structure high-speed non-volatile memory solid state drive is connected to the The host is visible, and wherein the baseboard management controller is used to determine at the beginning of the cross-structure high-speed non-volatile memory solid state drive, which is the main cross-structure high-speed non-volatile memory Sexual memory solid state drive.

Images