JP6796589B2 - Storage system architecture - Google Patents

Description

Traditional hard disk drives (HDDs), writable CDs (compact discs) or writable DVDs (digital diversity discs) drives, commonly known as spin media, and tape drives for storing large amounts of data. Solid state drives (SSDs) that augment or replace are currently used in solid state memories such as flash. Flash and other solid-state memories have different properties than spin media. In addition, many solid-state drives are designed to meet hard disk drive standards for compatibility reasons, which incorporates or provides improved features that take advantage of the unique perspectives of flash and other solid-state memories. Make it difficult. Spin media has limited flexibility or versatility of connection communication paths between storage units or storage nodes in conventional storage arrays.

The present embodiment is obtained in such a situation.

In certain embodiments, a storage system is provided. The storage system comprises a plurality of storage units, each of the plurality of storage units having a storage memory for user data, further comprising a plurality of storage nodes, and each of the plurality of storage nodes. It is configured to own a portion of the user data. Further, the storage system includes a first path for connecting the plurality of storage units, and each of the plurality of storage units goes through the first path without assistance from the plurality of storage nodes, and at least one other of the plurality of storage units. Allow communication with the storage unit.

In certain embodiments, storage clusters are provided. The storage cluster comprises a single chassis with multiple slots, each slot being configured to accommodate a storage node or storage unit, each of which may occupy one or more of the plurality of slots. it can. A storage cluster comprises multiple storage units in a single chassis, each of which has solid state storage memory for user data. The storage cluster has a first path for connecting a plurality of storage units in a single chassis, and from the first storage unit among the plurality of storage units to the second storage unit among the plurality of storage units via the first path. Make sure there are no storage nodes that intervene or support the communication.

In certain embodiments, a single chassis storage system is provided. The storage system comprises a plurality of storage units in a single chassis, each of which has a solid state storage memory for user data. Further, the storage system includes a first path in a single chassis, and each of the plurality of storage units is coupled so as to communicate with at least one other storage unit among the plurality of storage units via the first path. Will be done. Also, a storage system comprises a plurality of storage nodes in a single chassis, each of which has a subset of a plurality of storage units and owns a portion of user data. Multiple storage nodes are not involved in direct communication between storage units. Further, the storage system includes a second path for connecting a plurality of storage nodes in a single chassis.

Other aspects and effects of this embodiment will become apparent from the following detailed description relating to the accompanying drawings which illustrate the principles of the embodiments described herein.

The embodiments described herein and their effects will be best understood from the following description with reference to the accompanying drawings. These drawings do not limit any modification of the embodiments and details made by those skilled in the art to the embodiments described herein without departing from the spirit and scope of the embodiments described herein.

FIG. 5 is a perspective view of a storage cluster with a large number of storage nodes and internal storage coupled to each storage node to form network-attached storage according to an embodiment. FIG. 5 is a system diagram of an enterprise computing system that can use one or more of the storage clusters of FIG. 1 as storage resources in certain embodiments. FIG. 5 is a block diagram showing a large number of storage nodes with different capacities and non-volatile solid-state storage suitable for use in the storage cluster of FIG. 1 according to certain embodiments. It is a block diagram which shows the interconnection switch which connects a large number of storage nodes by a certain embodiment. It is a multi-level block diagram which shows the content of a storage node and the content of one non-volatile solid state storage unit according to an embodiment. FIG. 5 is a block diagram of yet another embodiment of a storage cluster having connections between and within storage nodes and storage units as an example according to one embodiment. It is a modification of the connection in the storage cluster of FIG. 6A according to an embodiment. FIG. 6 is a block diagram of yet another embodiment of a storage cluster of FIGS. 1-5 suitable for data storage or a combination of data storage and computing according to one embodiment. FIG. 5 is a block diagram of yet another embodiment of the connection within the storage cluster of FIGS. 1-5 with a switch according to one embodiment. A variant of the connection within the storage cluster of FIG. 8A with a switch that couples the storage units according to one embodiment is shown. It is a block diagram of an example of the architecture of a computer node coupled together with respect to a storage cluster according to an embodiment. FIG. 5 is a block diagram of yet another embodiment of the storage cluster of FIGS. 1-5 with the computer node of FIG. 9A according to one embodiment. In a block diagram of a variant of a storage cluster with computer nodes in FIG. 9B according to an embodiment, a chassis, a storage node, a storage unit, and a computer node, all of which are combined together as one or more storage clusters in many chassis. It is a figure which shows the modification of the connection inside and between chassis. It is a flowchart of the storage cluster operation method which can be carried out in the embodiment of a storage cluster, a storage node and / or a non-volatile solid state storage or a storage unit by an embodiment, or by an embodiment. It is a figure which shows the normative computing device which embodies the embodiment described here.

The following embodiments describe storage clusters that store user data, such as user data originating from one or more users, or client systems, or other sources outside the storage cluster. Storage clusters use erase coding and redundant copies of metadata to distribute user data across storage nodes housed within the chassis. Erasing coding refers to a method of data protection or reconstruction in which data is stored across different sets of locations, such as disks, storage nodes, or geographic locations. Flash memory is a type of solid state memory integrated into this embodiment, but this embodiment extends to other types of solid state memory or other storage media, including non-solid state memory. be able to. Storage location and workload control is distributed across the storage locations of clustered peer-to-peer systems. All tasks such as arbitrating communication between different storage nodes, detecting when storage nodes become unavailable, and balancing I / O (inputs and outputs) across different storage nodes are handled on a distributed basis. Is done. In certain embodiments, the data is laid out or distributed across a large number of storage nodes with data fragments or stripes that support data recovery. Data ownership can be redesignated within the cluster regardless of input and output patterns. This architecture, detailed below, allows the storage nodes in the cluster to fail while the system is still running. This is because the data can be reconstructed from other storage nodes and therefore remains available for input and output behavior. In various embodiments, the storage node is also referred to as a cluster node, blade, or server.

The storage cluster is housed in a chassis, i.e., in an enclosure that houses one or more storage nodes. A mechanism for supplying power to each storage node, such as a distribution bus, and a communication mechanism, for example, a communication bus that enables communication between storage nodes, are included within the chassis. According to certain embodiments, the storage cluster can run as an independent system in one location. In certain embodiments, the chassis houses at least two instances of both power distribution and communication buses that are independently enabled or disabled. The internal communication bus is an Ethernet bus, but other technologies such as Peripheral Component Interconnect (PCI) Express, InfiniBand, etc. are equally suitable. The chassis provides a port for an external communication bus to enable communication between multiple chassis, either directly or through a switch and with the client system. External communication uses technologies such as Ethernet (registered trademark), InfiniBand, and fiber channels. In certain embodiments, the external communication bus uses different communication bus technologies for chassis-to-chassis and client-to-client communication. When a switch is deployed within or between chassis, the switch acts as a transformant between a number of protocols or technologies. When multiple chassis are connected to define a storage cluster, the storage cluster can be a network file system (NFS), a regular Internet file system (CIFS), a small computer system interface (SCSI) or a hypertext transfer protocol (SCSI). It can be accessed by the client using an exclusive or standard interface such as HTTP). Conversion from the client protocol takes place within the switch, chassis external communication or within each storage node.

Each storage node is one or more storage servers, and each storage server is connected to one or more non-volatile solid-state memory units called storage units. In one embodiment, each storage node includes a single storage server between one or eight non-volatile solid-state memory units, but is not limited to this example. The storage server includes a processor, dynamic random access memory (DRAM), and an interface for power distribution of the Internet communication bus and each power bus. Within the storage node, the interface and storage unit share a communication bus, eg, in some embodiments, PCI Express. The non-volatile solid-state memory unit either directly accesses the internal communication bus interface via the storage node communication bus or requires the storage node to access the bus interface. The non-volatile solid state memory unit includes an embedded central processing unit (CPU), a solid state storage controller, and, in certain embodiments, an amount of solid state mass storage, eg 2-32 terabytes (TB). Embedded volatile storage media such as DRAM and energy storage devices are included in the non-volatile solid state memory unit. In certain embodiments, the energy storage device is a capacitor, supercapacitor or battery capable of transferring a subset of DRAM content to a stable storage medium in the event of a power outage. In certain embodiments, the non-volatile solid-state memory unit is configured with storage class memory, eg, a phase change or magnetoresistive random access memory (MRAM) that replaces DRAM and allows a reduced power holding device.

One of the many features of storage nodes and non-volatile solid-state storage is the ability to a priori reconstruct data in storage clusters. The storage node and non-volatile solid-state storage determine when the storage node or non-volatile solid-state storage of the storage cluster is unreachable, regardless of whether there is an attempt to read data about the storage node or non-volatile solid-state storage. Can be done. Storage nodes and non-volatile solid-state storage then work together to recover and reconstruct data at least partially in new locations. This constitutes a forward-looking rebuild in that the system rebuilds the data without waiting for the data to be needed for read access initiated by the client system using the storage cluster. These and other details of the storage memory and its operation are described below.

FIG. 1 is a perspective view of a storage cluster 160 with a large number of storage nodes 150 and internal solid-state memory coupled to each storage node to form a network-attached storage or storage area network according to certain embodiments. A network-attached storage, storage area network or storage cluster or other storage memory comprises one or more storage clusters 160, each with one or more storage nodes 150, of physical components and the amount of storage memory provided thereby. Both flexible and reconfigurable sequences can be provided. The storage cluster 160 is designed to fit racks, and one or more racks can be configured and exist as storage memory as needed. The storage cluster 160 includes a chassis 138 with a large number of slots 142. It is clear that chassis 138 is also referred to as a housing, enclosure, or rack unit. In one embodiment, the chassis 138 has 14 slots 142, but other numbers of slots are readily devised. For example, in some embodiments, there are 4 slots, 8 slots, 16 slots, 32 slots, or a number of other slots. In certain embodiments, each slot 142 can accommodate one storage node 150. Chassis 138 includes flaps 148 that can be used to mount chassis 138 in a rack. Fan 144 provides air circulation to cool the storage node 150 and its components, but other cooling components may be used, or embodiments without cooling components can be devised. The switch fabric 146 joins the storage nodes 150 in chassis 138 together and joins the network for communication to memory. In the embodiment shown in FIG. 1, the left slot 142 of the switch fabric 146 and the fan 144 is shown occupied by the storage node 150, whereas the right slot 142 of the switch fabric 146 and the fan 144 is for purposes of illustration. It is empty and can be used to insert the storage node 150. This configuration is only an example, and in various other arrays, one or more storage nodes 150 occupy slot 142. The array of storage nodes does not have to be continuous or adjacent in some embodiments. The storage node 150 is a hot plug type, which means that the storage node 150 can be inserted into or removed from slot 142 of chassis 138 without stopping or powering down the system. When the storage node 150 is inserted into or removed from slot 142, the system automatically reconfigures to recognize and adapt to the change. In certain embodiments, the reconfiguration comprises restoring redundancy and / or rebalancing the data or load.

Each storage node 150 can have a large number of components. In the embodiment shown here, the storage node 150 comprises a CPU 156, a printed circuit board 158 in which a processor, a memory 154 coupled to the CPU 156, and a non-volatile solid state storage 152 coupled to the CPU 156 reside. In yet another embodiment, other mounting and / or components can be used. The memory 154 has instructions executed by the CPU 156 and / or data manipulated by the CPU 156. As described in detail below, the non-volatile solid state storage 152 includes flash, or, in yet another embodiment, other types of solid state memory.

FIG. 2 is a system diagram of an enterprise computing system 102 that can use one or more of the storage nodes, storage clusters, and / or non-volatile solid-state storage of FIG. 1 as storage resources 108. For example, the flash storage 128 of FIG. 2 may, in certain embodiments, integrate the storage node, storage cluster, and / or non-volatile solid-state storage of FIG. The enterprise computing system 102 has a storage resource 108 including a processing resource 104, a network resource 106, and a flash storage 128. The flash storage 128 includes a flash controller 130 and a flash memory 132. In various embodiments, the flash storage 128 may include one or more storage nodes or storage clusters, along with a flash controller 130 including a CPU and a flash memory 132 including non-volatile solid state storage of the storage nodes. In certain embodiments, the flash memory 132 may include different types of flash memory or the same type of flash memory. The enterprise computing system 102 exhibits a suitable environment for deploying the flash storage 128, which may be used for other larger or smaller computing systems or devices, or for fewer or additional. It can be used as a variant of the enterprise computing system 102 with resources. The enterprise computing system 102 is coupled to a network 140 such as the Internet to provide or use services. For example, the enterprise computing system 102 can provide cloud services, physical computing resources, or virtual computing services.

In the enterprise computing system 102, various resources are arranged and managed by various controllers. The processing controller 110 manages a processing resource 104 including a processor 116 and a random access memory (RAM) 118. The network controller 112 manages a network resource 106 that includes a router 120, a switch 122, and a server 124. The storage controller 114 manages a storage resource 108 including a hard drive 126 and a flash storage 128. These embodiments may include other types of processing resources, network resources and storage resources. In one embodiment, the flash storage 128 completely replaces the hard drive 126. The enterprise computing system 102 can provide or allocate various resources as physical computing resources or, in a variant, as virtual computing resources supported by physical computing resources. For example, various resources can be implemented using one or more servers running the software. Files or other objects, or other forms of data, are stored in storage resource 108.

In various embodiments, the enterprise computing system 102 comprises a large number of racks in which storage clusters reside, which are located in a single physical location, such as a cluster or server farm. In other embodiments, multiple racks are located in multiple physical locations such as various networked cities, states or countries. Each rack, each storage cluster, each storage node and each non-volatile solid-state storage is individually configured with each amount of storage space, which can then be reconfigured independently of anything else. Therefore, storage capacity can be flexibly added, upgraded, deducted, recovered and / or reconfigured in each of the non-volatile solid state storages. As mentioned above, each storage node can implement one or more servers in certain embodiments.

FIG. 3 is a block diagram showing a large number of storage nodes 150 and non-volatile solid state storage 152 with different capacities suitable for use in the chassis of FIG. Each storage node 150 has one or more units of non-volatile solid state storage 152. Each non-volatile solid-state storage 152 may include, in certain embodiments, different capacities from the storage node 150 or another non-volatile solid-state storage 152 at the other storage node 150. Alternatively, all non-volatile solid-state storage 152 in the storage node or multiple storage nodes may have the same capacity, or may have the same and / or different capacity combinations. This flexibility is shown in FIG. 3, which has one storage node 150 with mixed non-volatile solid-state storage 152 with 4, 8 and 32 TB capacity, each with 32 TB capacity non-volatile solid state storage 152. An example of yet another storage node having another storage node 150 and each having a non-volatile solid state storage 152 having a capacity of 8 TB is shown. Yet another variety of combinations and capacities are readily devised according to the teachings described herein. In the context of clustering, eg, clustered storage for forming a storage cluster, the storage node is or includes non-volatile solid state storage 152. The non-volatile solid-state storage 152 is a convenient clustering point as the non-volatile solid-state storage 152 includes a non-volatile random access memory (NVRAM) component, as further described below.

With reference to FIGS. 1 and 3, the storage cluster 160 is expandable, which means that storage capacity with non-uniform storage size is easily added, as described above. One or more storage nodes 150 can be plugged in and removed from each chassis, and in certain embodiments, the storage cluster is self-configuring. The plug-in storage node 150 can have different sizes, whether it is installed in the chassis at the time of delivery or added later. For example, in some embodiments, the storage node 150 can have multiples of 4TB, such as 8TB, 12TB, 16TB, 32TB, and so on. In yet another embodiment, the storage node 150 may be another storage amount or multiple of capacity. The storage capacity of each storage node 150 is broadcast, which influences the determination of how to strip the data. For maximum storage efficiency, in one embodiment, the stripes can be self-configured as widely as possible, which is a continuous loss of up to one or two non-volatile solid-state storage units 152 or storage nodes 150 in the chassis. It receives a predetermined condition of operation.

FIG. 4 is a block diagram showing a communication interconnect 170 and a distribution bus 172 that connect a large number of storage nodes 150. Returning to FIG. 1, the communication interconnect 170 is, in certain embodiments, included in or implemented in switch fabric 146. Where a large number of storage clusters 160 occupy the rack, the communication interconnect 170 is, in certain embodiments, included or implemented at the top of the rack switch. As shown in FIG. 4, the storage cluster 160 is housed in a single chassis 138. The external port 176 is coupled to the storage node 150 through the communication interconnect 170, while the external port 174 is directly connected to the storage node. The external power port 178 is coupled to the distribution bus 172. Storage node 150 includes non-volatile solid-state storage 152 in varying amounts and different capacities, as described with reference to FIG. In addition, one or more storage nodes 150 are calculation-only storage nodes, as shown in FIG. The compute-only storage node 150 either performs the compute functions of the storage cluster 160 or acts as a compute node to perform compute functions and / or execute applications, which, in various embodiments, is the storage cluster. Uses user data destined for, stored in, or retrieved from 160 non-volatile solid-state storage 152. Authority 168 is implemented in the non-volatile solid-state storage 152, for example, as a list or other data structure stored in memory. In certain embodiments, the authority is stored in the non-volatile solid-state storage 152 and is supported by software running on the controller or other processor of the non-volatile solid-state storage 152. In yet another embodiment, authority 168 is implemented at storage node 150, for example as a list or other data structure stored in memory 154, and is supported by software running on CPU 156 of storage node 150. .. Authority 168, in certain embodiments, controls how and where data is stored in the non-volatile solid-state storage 152. This control helps determine which type of erase coding skim is applied to the data and which storage node 150 has which part of the data. Each authority 168 is designated as non-volatile solid state storage 152. In various embodiments, each authority controls a range of inode numbers, segment numbers, or other data identifiers specified in the data by the file system, by the storage node 150, or by the non-volatile solid-state storage 152. To do.

Each data fragment and each metadata fragment, in certain embodiments, has redundancy in the system. In addition, each data fragment and each metadata fragment has an owner referred to as an authority or, in some variants, a ward, an authority being a group or set of words. For example, if a storage node fails to reach its authority, there is a successful plan on how to find that data or its metadata. In various embodiments, there are redundant copies of authority 168. Authority 168, in certain embodiments, has a relationship with storage node 150 and non-volatile solid-state storage 152. Each authority 168 covering a range of data segment numbers or other data identifiers (and thus, in embodiments where the authority is a group of words, each group of words) is designated for a particular non-volatile solid-state storage 152. .. In certain embodiments, the authority 168 for all such ranges is distributed across the non-volatile solid-state storage 152 of the storage cluster. Each storage node 150 has a network port that provides access to the non-volatile solid state storage 152 of that storage node 150. The data is stored in the segment associated with the segment number, which in certain embodiments is an indirection to the configuration of a RAID (redundant array of independent disks) stripes. Therefore, the designation and use of authority 168 (and words, if applicable) establishes an indirect reference to the data. Indirect reference, according to some embodiments, is also referred to as the ability to reference data indirectly, in this case via authority 168. A segment identifies a set of non-volatile solid-state storage 152 and is a location identifier for the set of non-volatile solid-state storage 152 containing data. In some embodiments, the location identifier is an offset to the device and is sequentially reused by a number of segments. In other embodiments, the location identifier is unique to a particular segment and is never reused. The offset in the non-volatile solid state storage 152 is applied to the position data for writing to or reading from the non-volatile solid state storage 152 (in the form of RAID stripes). Data is striped across a number of units of non-volatile solid-state storage 152, which may or may not include non-volatile solid-state storage 152 having authority 168 for a particular data segment. ..

For example, if the location of a particular segment of data changes during data movement or data reconstruction, the authority 168 for that data segment is assigned to the non-volatile solid-state storage 152 or storage node 150 having that authority 168. Must be discussed at. In these embodiments, a hash value for a data segment is calculated or an inode number or data segment number is applied to position a particular piece of data. The output of this operation refers to the non-volatile solid state storage 152 having authority 168 for that particular piece of data. In one embodiment, there are two stages to this operation. The first step maps an entity identifier (ID), such as a segment number, inode number, or directory number, to an authority identifier. This mapping involves calculations such as hashes or bitmasks. The second step maps the authority identifier to a particular non-volatile solid state storage 152, which is done through explicit mapping. The operation is repeatable, and when the calculation is performed, the result of the calculation repeatedly and reliably points to a particular non-volatile solid state storage 152 having authority 168. The operation includes a reachable storage node as input. If the set of reachable non-volatile solid-state storage units changes, the optimal set changes. In certain embodiments, the persisted value is the current specification (which is always true), and the calculated value is the target specification that the cluster attempts to reconfigure. This calculation is used to determine the optimal non-volatile solid-state storage 152 for authority in the presence of a set of non-volatile solid-state storage 152 that is reachable and also constitutes a raster. This calculation also determines an ordered set of peer non-volatile solid-state storage 152 that records authority for mapping of non-volatile solid-state storage, even if the specified non-volatile solid-state storage is unreachable. Be able to determine authority. If a particular authority 168 is not available in certain embodiments, a copy or substitute authority 168 is discussed.

Referring to FIGS. 1 to 4, two of the many tasks of the CPU 156 at the storage node 150 are disassembling the write data and reassembling the read data. When the system determines that the data should be written, the authority 168 for that data is positioned as described above. When the segment ID of the data has already been determined, the write request is transferred from the segment to the non-volatile solid-state storage 152 currently determined to be the host of the determined authority 168. The host CPU 156 of the storage node 150 in which the non-volatile solid-state storage 152 and the corresponding authority 168 are present disassembles or shares the data and transmits the data to various non-volatile solid-state storage 152. The transmitted data is written as a data stripe according to the erasure code scheme. In some embodiments it is required to pull the data, and in other embodiments the data is pushed. Conversely, when the data is read, the authority 168 for the segment ID containing the data is positioned as described above. The host CPU 156 of the storage node 150 in which the non-volatile solid-state storage 152 and the corresponding authority 168 are present requests data from the non-volatile solid-state storage pointed out by the authority and the corresponding storage node. In certain embodiments, the data is read from flash storage as data stripes. The host CPU 156 of the storage node 150 then reassembles the read data, corrects any errors (if any) according to the appropriate erase code scheme, and transfers the reassembled data to the network. In yet another embodiment, some or all of these tasks can be handled in the non-volatile solid state storage 152. In one embodiment, the segment host requests the data to be sent to the storage node 150 by requesting the page from the storage and then sending the data to the storage node that made the original request.

In some systems, such as UNIX-style file systems, data is handled by index nodes or inodes, which identify data structures that represent objects in the file system. The object is, for example, a file or directory. Metadata accompanies an object, among other attributes, as attributes such as authorization data and generation timestamps. Segment numbers can be specified for all or part of such objects in the file system. In other systems, data segments are treated with a segment number specified somewhere. Descriptionally, the unit of distribution is an entity, and an entity is a file, directory or segment. That is, an entity is a unit of data or metadata stored by a storage system. Entities are grouped into sets called authorities. Each authority has an authority owner, which is a storage node that has the exclusive right to update an entity in the authority. In other words, the storage node contains an authority, and the authority then contains an entity.

According to one embodiment, a segment is a logical container for data. The segment is the address space between the media address space and the physical flash location, i.e. the data segment number is in this address space. The segment also contains metadata, which allows data redundancy to be restored without high-level software (rewriting to a different flash location or device). In one embodiment, the internal format of the segment includes client data and media mapping to determine the location of that data. Each data segment is protected, for example, from memory and other defects by appropriately dividing the segment into a large number of data and parity fragments (shards). The data and parity fragments are distributed, or striped, across the non-volatile solid-state storage 152 coupled to the host CPU 156 (see FIG. 5) according to the erase code scheme. The use of the term segment, in certain embodiments, refers to the container and its location in the address space of the segment. The use of the term stripe refers to the same set of fragments as a segment, and according to certain embodiments, includes how the fragments are distributed with redundancy or parity information.

A series of address space translations are performed throughout the storage system. At the top is a directory input (filename) that links to the inode. The inode refers to the medium address space where the data is logically stored. Media addresses are mapped through a set of indirect media to offload large files or to perform data services such as copy exclusion or snapshots. Media addresses are mapped through a set of indirect media to offload large files or to perform data services such as copy exclusion or snapshots. The segment address is then translated into a physical flash position. The physical flash location, according to certain embodiments, has an address range limited by the amount of flash in the system. The media and segment addresses are logical containers, and in some embodiments, 128-bit or higher identifiers are used to make them virtually infinite, but perhaps reuse is longer than the expected life of the system. It is calculated. Addresses from the logical container are, in some embodiments, assigned in a hierarchy form. Initially, each non-volatile solid-state storage 152 is designated with a range of address spaces. Within this specification, the non-volatile solid-state storage 152 can be assigned an address without synchronizing with other non-volatile solid-state storage 152.

Data and metadata are stored by a set of basic storage layouts optimized for changing workload patterns and storage devices. These layouts combine a number of redundancy schemes, compression formats, and indexing algorithms. Some of these layouts store information about the authority and authority master, while others store file metadata and file data. Redundancy schemes include error correction codes that allow decay bits in a single storage device (such as NAND flash chips), erase codes that allow defects in many storage nodes, and replication that allows data center or area defects. Includes scheme. In certain embodiments, a low density parity check (LDPC) code is used within a single storage unit. In certain embodiments, Reed-Solomon encoding is used within the storage cluster and mirroring is used within the storage grid. Metadata is stored using an ordered log-structured index (such as a log-structured merger tree), and large data is not stored in the log-structured layout.

In order to maintain consistency across multiple copies of an entity, storage nodes implicitly agree on two things through computation: (1) an authority containing an entity, and (2) a storage node containing an authority. Specifying an entity to an authority can be done by specifying the entity to the authority in a pseudo-random manner, dividing the entity into ranges based on externally generated keys, or putting a single entity in each authority. .. Examples of pseudo-random schemes are linear hashing, and the replication underscalable hashing (RUSH) family of hashes, which includes controlled replication underscalable hashing (CRUSH). In certain embodiments, the pseudo-random designation is only used to specify the authority of a node as the set of nodes changes. The set of authorities does not change, so in these embodiments subjective functions apply. Some placement schemes automatically place the authority on the storage node, while other placement schemes rely on explicitly mapping the authority to the storage node. In some embodiments, a pseudo-random scheme is used to map from each authority to a set of candidate authority owners. The pseudo-random data distribution feature associated with CRUSH specifies an authority for a storage node and produces a list of where the authority is specified. Each storage node has a copy of the pseudo-random data distribution function, arrives at the same calculation for distribution, and then finds or positions the authority. Each of the pseudo-random schemes, in certain embodiments, requires an reachable set of storage nodes as input to conclude the same target node. When an entity is placed in the authority, it is stored in the physical device so that the expected defects do not lead to unexpected data loss. In one embodiment, the rebalancing algorithm attempts to store copies of all entities within the same layout authority and in the same machine set.

Possible defects include, for example, equipment defects, stolen machines, data center fires, and regional catastrophes, such as nuclear or regional accidents. Different defects result in different levels of tolerable data loss. In some embodiments, the stolen storage node does not affect the security or reliability of the system, but depending on the system configuration, a regional accident may result in no data loss or lost updates of seconds or minutes. Or cause complete data loss.

In these embodiments, the placement of data for storage redundancy is independent of the placement of authorities for data consistency. In certain embodiments, the storage node containing the authority does not include persistent storage. Rather, the storage node is connected to a non-volatile solid-state storage unit that does not contain authority. The communication interconnect between the storage node and the non-volatile solid-state storage unit consists of a number of communication technologies and has non-uniform performance and defect tolerance characteristics. In one embodiment, as described above, the non-volatile solid-state storage unit is connected to the storage node via PCI Express, the storage nodes are connected together in a single chassis using an Ethernet backplane, and The chassis are connected together to form a storage cluster. In some embodiments, the storage cluster is connected to the client using Ethernet or fiber channels. When many storage clusters are configured in a storage grid, many storage clusters connect using the Internet or other long-distance network links, such as "metroscale" or private links that do not cross the Internet. Will be done.

The authority owner has the exclusive right to move an entity from one non-volatile solid state storage unit to another and modify the entity to add and remove copies of the entity. This allows to maintain the redundancy of the underlying data. If the authority owner fails, is closed, or is overloaded, the authority is migrated to a new storage node. Temporary flaws are not trivial to ensure that all non-defective machines agree on a new authority position. Ambiguities caused by temporary defects can be disambiguated by consensus protocols such as Paxos, hot-war failover schemes, or through manual intervention by remote system administrators, or by local hardware administrators (eg, physics of defective machines from a cluster). It can be achieved automatically (by removing it or pressing a button on the defective machine). In some embodiments, a consensus protocol is used, and failover is automatic. If too many defects or copying accidents occur within too short a period of time, the system, according to certain embodiments, goes into self-preservation mode and suspends copying and data movement activities until the administrator intervenes. ..

When an authority is transferred between storage nodes and the authority owner updates an entity in the authority, the system transfers a message between the storage node and the non-volatile solid-state storage unit. With respect to persistent messages, messages with different purposes are of different types. Based on the type of message, the system maintains different order and durability guarantees. When a persistent message is processed, the message is temporarily stored in a number of durable and non-durable storage hardware technologies. In certain embodiments, the messages are stored in RAM, NVRAM and NAND flash devices, and various protocols are used to make efficient use of each storage medium. Latency-sensitive client requests are persisted in copy NVRAM and then NAND, while background rebalancing operations are persisted directly in NAND.

Persistent messages are persistently stored before being copied. This allows the system to continue to serve client requests despite defects and component replacements. Many hardware components include unique identifiers that appear to be system administrators, manufacturers, hardware supply chains, and ongoing monitoring quality control infrastructure, but applications running at the top of the infrastructure address address the address. Virtualize. Their virtualized addresses do not change over the life of the storage system, despite component defects and replacements. This allows each component of the storage system to be replaced over time without reconfiguring or interrupting client request processing.

In some embodiments, the virtualized address is stored with sufficient redundancy. The continuous monitoring system correlates the hardware and software state with the hardware identifier. This makes it possible to detect and predict failures due to defective components and manufacturing details. Surveillance systems also allow, in certain embodiments, a priori transfer of authorities and entities from affected devices by removing components from critical paths before failure occurs.

FIG. 5 is a multi-level block diagram showing the contents of the storage node 150 and the contents of the non-volatile solid state storage 152 of the storage node 150. In certain embodiments, data is communicated to and from storage node 150 by network interface controller (NIC) 202. Each storage node 150 has a CPU 156 and one or more non-volatile solid-state storages 152, as described above. Moving down one level in FIG. 5, each non-volatile solid state storage 152 has relatively fast non-volatile solid state memory, such as non-volatile random access memory (NVRAM) 204 and flash memory 206. In certain embodiments, NVRAM 204 is a component (DRAM, MRAM, PCM) that does not require a program / erase cycle and is a memory that is supported so that the memory is written significantly more often than it is read. Moving down another level in FIG. 5, NVRAM 204 is implemented in one embodiment as a fast volatile memory such as dynamic random access memory (DRAM) 216 backed up by energy storage 218. The energy storage 218 supplies sufficient power to keep the DRAM 216 energized long enough to transfer content to the flash memory 206 in the event of a power outage. In certain embodiments, the energy storage 218 is a capacitor, supercapacitor, battery or other device that provides adequate energy to transfer the contents of the DRAM 216 to a stable storage medium in the event of a power outage. The flash memory 206 is implemented as a number of flash dies 222, also referred to as a package of flash dies 222 or an array of flash dies 222. The flash die 222 can be packaged in many ways, eg, with one die per package, or with multiple dies per package (ie, a multi-chip package), in a hybrid package, bare on a printed circuit board or other substrate. It is clear that it can be packaged as a die, a capsule die, and so on. In the embodiments shown herein, the non-volatile solid state storage 152 has a controller 212 or other processor and an input / output (I / O) port 210 coupled to the controller 212. The I / O port 210 is coupled to the CPU 156 and / or network interface controller 202 of the flash storage node 150. The flash input / output (I / O) port 220 is coupled to the flash die 222, and the direct memory access unit (DMA) 214 is coupled to the controller 212, DRAM 216 and flash die 222. In the embodiments shown herein, the I / O port 210, the controller 212, the DMA unit 214, and the flash I / O port 220 are implemented in a programmable logic device (PLD) 208, such as a field programmable gate array (FPGA). .. In this embodiment, each flash die 222 has a page organized as 16 kB (kilobytes) page 224 and a register 226 through which data is written to or read from the flash die 222. In yet another embodiment, other types of solid state memory are used in place of or in addition to the flash memory shown within the flash die 222.

FIG. 6A is a block diagram of a further embodiment of the storage cluster 160 of FIGS. 1-5. In this embodiment, components are provided on chassis 138, such as chassis 138 with a large number of slots as shown in FIG. Power supply 606 with distribution bus 172 (as seen in FIG. 2) powers various components within chassis 138. Two storage nodes 150 are shown coupled in one embodiment to a path 604 such as a network switch 620. Yet another route is easily devised. Route 604 couples the storage nodes 150 to each other and also to the network outside the chassis 138 so that it can be connected to an external device, system or network.

A large number of storage units 152 are coupled to each other and to the storage node 150 by a network switch 620 that couples the storage nodes 150 or another route 602 that is separate from the other route 604. In one embodiment, the path 602 connecting the storage unit 152 and the storage node 150 is a PCI Express Bus (PCIe), but other buses, networks and various other couplings can be used. In one embodiment, a transparent bridge is provided to connect the storage node 150 to the path 602, eg, the PCI Express Bus.

Each storage node 150 has two ports 608,610 to connect to the two paths 602,604. One of the ports 610 of each storage node 150 is coupled to one of the routes 604, and the other port 608 of each storage node 150 is coupled to the other route 602.

In certain embodiments, each of the storage nodes 150 performs a computational function as a compute node. For example, the storage node 150 can execute one or more applications. Further, the storage node 150 communicates with the storage unit 152 via the path 602 and writes and reads user data (eg, using an erase code) as described above with reference to FIGS. 1 to 3. As another example, a storage node 150 that executes one or more applications uses user data, generates user data for storage in storage unit 152, reads user data from storage unit 152, processes it, and processes it. And so on. Even if one of the storage units 152 is lost, or in some embodiments, two of the storage units 152 are lost, the storage node 150 and / or the remaining storage unit 152 can still read the user data.

In certain embodiments, the erase code function is performed almost or completely in the storage unit 152, which frees up the computing power of the storage node 150. This allows the storage node 150 to focus more on the task of the compute node, such as running one or more applications. In certain embodiments, the erase code function is performed almost or completely on the storage node 150. This allows the storage node 150 to focus more on the mission of the storage node. In certain embodiments, the erase code function is shared across the storage node 150 and the storage unit 152. This allows the compute bandwidth available to the storage node 150 to be shared between the compute node mission and the storage node mission.

With the two paths 602, 604 being separate from each other, a number of effects become apparent. Neither of the routes 602 and 604 is a bottleneck that occurs when there is only one route that connects the storage node 150 and the storage unit 152 to each other and to the external network. With only one route, the hostile can gain direct access to the storage unit 152 without having to travel through the storage node 150. With the two paths 602, 604 present, the storage nodes 150 can be coupled to each other through one path 604, for example for multi-process applications or interprocessor communication. The other path 602 can be used by each of the storage nodes 150 for data access in the storage unit 152. Therefore, the architecture shown in FIG. 6A supports a variety of storage and computing functions as well as scenarios. In particular, one embodiment shown in FIG. 6A is a storage and computing system in a single chassis 138. The processing power in the form of one or more storage nodes 150, and the storage capacity in the form of one or more storage units 152, can be easily added to chassis 138 when storage and / or calculations require changes. it can.

FIG. 6B is a modification of the storage cluster 160 of FIG. 6A. In this modification, the path 612 has a portion specific to the storage unit 152 included in each storage node 150. In one embodiment, the route 612 is implemented as a PCI Express bus that connects the storage unit 152 and the storage node 150 together. That is, in one embodiment, the storage node 150 and the storage unit 152 in one blade share a PCI Express bus. The PCI Express Bus is blade-specific and is not directly coupled to the PCI Express Bus of another blade. Therefore, the storage unit 152 of a blade can communicate with each other and with the storage node 150 of that blade. Communication from the storage unit 152 or storage node 150 of one blade to the storage node 150 or storage unit 152 of another blade is performed via a network switch 620, for example, a path 614.

FIG. 7 is a block diagram of yet another embodiment of the storage cluster 160 of FIGS. 1-5, which is suitable for data storage or a combination of data storage and computation. The variant of FIG. 7 has all storage units 152 coupled together by a first path 616, which is a bus, network, or hardwired mesh, among other possibilities. One storage node 150 is coupled to each of the two storage units 152. A separate storage unit 152 is coupled to each of the two further separate storage units 152. The coupling from the storage node 150 to the storage unit 152 indicates a second path 618.

FIG. 8A is a block diagram of yet another embodiment of the storage cluster 160 of FIGS. 1-5 with a switch 620. One switch 620 couples all storage nodes 150 together. Another switch 620 also couples all storage nodes 150 together. In this embodiment, each storage node 150 has two ports, each port connected to one of the switches 620. This array of ports and switches 620 provides two routes for connecting each storage node 150 to another storage node 150. For example, the leftmost storage node 150 can be connected to the rightmost storage node 150 (or another storage node 150 in the storage cluster 160) by selecting either the first switch 620 or the second switch 620. it can. It should be understood that this architecture alleviates communication bottlenecks. Yet another embodiment with one switch 620, two switches 620 coupled to each other, three or more switches 620, and a number of other ports or networks is readily devised in light of the teachings described herein. Is done.

FIG. 8B shows a modification of the storage cluster 160 of FIG. 8A with a switch 620 that couples the storage unit 152. Similar to the embodiment of FIG. 8A, the switch 620 combines the storage nodes 150 and provides each storage node 150 with two routes for communicating with the other storage nodes 150. Further, the switch 620 couples the storage unit 152. Two of the storage units 152 at each storage node 150 are coupled to one of the switches 620, and one or more of the storage units 152 at each storage node 150 are coupled to another one of the switches 620. In this way, each storage unit 152 is connected to approximately half of the other storage units 152 in the storage cluster via one of the switches 620. In a variant, the switches 620 are coupled together (as shown by the dashed line in FIG. 8B), and each storage unit is connected to another storage unit 152 via the switch 620. Yet another embodiment with one switch 620, or an array of other numbers of switches 620 and connections, and the number of components connected is also readily devised in light of the teachings described herein.

FIG. 9A is a block diagram of compute nodes 626 coupled together with respect to the storage cluster 160. Switch 620 joins all compute nodes 626 together, allowing each compute node 626 to communicate with other compute nodes 626 via switch 620. In various embodiments, each compute node 626 is a compute-only storage node 150 or a special compute node 626. In the embodiment shown here, the compute node 626 has three processor complexes 628. Each processor complex 628 has port 630 and local memory and yet another support (eg, digital signal processing, direct memory access, various forms of I / O, graphic accelerometers, one or more processors, etc. Etc.) Each port 630 is coupled to a switch 620. Therefore, each processor complex 628 can communicate with each other processor complex 628 via the associated port 630 and switch 620 in this architecture. In certain embodiments, each processor complex 628 is a heartbeat, a regular communication that can be observed as an indicator of ongoing operations, indicating a possible defect or non-availability of a compute node or processor. Not). In certain embodiments, each compute node 626 generates a heartbeat. In yet another embodiment, the storage node 150 and / or the storage unit 152 also generate a heartbeat.

9B is a block diagram of yet another embodiment of the storage cluster 160 of FIGS. 1-5, with the compute node 626 of FIG. 9A. This embodiment is also shown with the storage node 150. Switch 620 joins all ports of all storage nodes 150, all ports of all compute nodes 626 (eg, all processor complexes 628 of all compute nodes 626), and all storage units 152. In a variant, fewer or more storage nodes 150, fewer or more compute nodes 626, fewer or more storage units 152, and fewer or more processor complexes 628 are installed in chassis 138. can do. Each storage node 150, storage unit 152 or compute node 626 occupies one or more slots 142 (see FIG. 1) of chassis 138. It should be understood that FIGS. 9A and 9B are merely examples and are not limited thereto. In one embodiment, a number of switches 620 are integrated into chassis 138, and compute node 626, similar to the embodiments of FIGS. 8A and 8B, achieves the communication flexibility provided by the embodiments described herein. Combined with a large number of switches.

FIG. 9C is a block diagram showing a variant of the storage cluster 160 with compute nodes 626 of FIG. 9B, showing storage nodes 150, storage units 152 and compute nodes 626 in a number of chassis 138, all of which are one or more. Combined together as a storage cluster 160 of. A number of chassis 138s are rack-mounted and can be coupled together as shown here for expansion of the storage cluster 160. In this embodiment, the switches 620 (s) of each chassis 138 combine the components of the chassis 138 as described above with reference to FIG. 9B, and the switches 620 (s) of all chassis 138. The plurality) are coupled together across all chassis 138. Various combinations of storage nodes 150 and / or compute nodes 626 allow storage capacity and / or compute capacity (eg, to run applications, operating systems, etc.) to be easily configured, expanded or contracted, or virtual. Virtualized in a computing environment. The use of switch 620 reduces or eliminates the usual patch wiring found in many other rack mount systems.

Some embodiments of this and other embodiments of the storage cluster 160 can support two or more independent storage clusters in one chassis 138, two chassis 138s or multiple chassis 138s. Each storage cluster 160 in a multi-storage cluster environment has storage nodes 150, storage units 152 and / or compute nodes 626 in various combinations in one, another, or more, or more chassis 138. Can be done. For example, the first storage cluster 160 may have a large number of storage nodes 150 in one chassis 138 and one or more storage nodes 150 in another chassis 138. The second storage cluster 160 can have one or more storage nodes 150 in the first chassis 138 and one or more storage nodes 150 in the second chassis 138. Any of these storage clusters 160 may have compute nodes 626 on either or both of chassis 138. Each storage cluster 160 has its own operating system and has its own application running independently of the other storage clusters 160.

Many features are apparent in some or all of the embodiments shown in FIGS. 6A-9C. Many embodiments provide a route through which each storage unit 152 can communicate directly with one or more other storage units 152 without assistance from the storage node 150. That is, the storage unit 152 can communicate with another storage unit 152 via the path, and the storage node 150 is not involved in such communication. There is no storage node 150 that intervenes or supports communication from one storage unit 152 to another storage unit 152 via this direct path. In one embodiment, such a direct route is provided for communicating from the storage unit 152 to another storage unit 152. In some embodiments, such a direct route from each storage unit 152 to one or more other storage units 152 is provided, but not necessarily all other storage units 152. In these cases, the storage unit 152 can communicate with another storage unit 152 via one or more storage nodes 150 and another path, i.e. with the support of the storage node 150.

In certain embodiments, the path for direct communication from one storage unit 152 to another storage unit 152 is included in the coupling of other components of the storage cluster 160. In certain embodiments, each storage node 150 can communicate directly with each storage unit 152 of all storage clusters 160. In one embodiment, each storage node 150 can communicate directly with some of the storage units 152 and with other storage units 152 via another storage node 150. In one embodiment, the routes for communication between the storage nodes 150 and between the storage units 152 are separated, and in other embodiments, the routes are combined. In some embodiments, the routes for communication between the storage node 150 and the storage unit 152, and for communication between the storage units 152, are separated, and in other embodiments, the routes are combined.

One aspect of the storage node 150 has two ports 608,610. Both ports 608,610 are used in certain embodiments to communicate to another storage node 150 via a selection of two different routes. In one embodiment, one port 610 is used to communicate with the other storage node 150 via one route, and another port 608 is used to communicate with the storage unit 152 via another route. Will be done. Both ports 608,610 are used in certain embodiments to communicate with the storage node 150 and the storage unit 152. By supporting direct communication between storage units 152, these various architectures can reduce communication bottlenecks. The storage node 150, as well as the processing and communication bandwidth, are not tied up to support communication between the storage units 152. As a result of this offloading, the storage node 150 can transfer those functions to the storage unit 152 for high-speed operation on user data.

Communication between storage units 152 may include data, metadata, messages, etc. to ensure that the storage unit 152 is alive, healthy, and / or state information, etc. With the storage unit 152 communicating directly with the other storage unit 152, the storage node 150 is free to manage other processes without the intervention of the storage node 150 (or the processor or controller of the storage node 150). Communication between the storage node 150 and the storage unit 152, or communication between the storage units 152 when taking over some functions of the storage node 150, is data, metadata (eg, data-related and data-related information). , As well as data fragments with metadata (eg, metadata about the metadata). Such communications can also include parity fragments, health, state and performance information. By making the storage unit 152 accessible by another storage unit 152 or a storage node 150 (for example, the processor of the storage node 150), the distinction of data ownership is shifted from the storage node 150 to the storage unit 152 in a variable degree. be able to. This includes shifting the authority 168 or word between the storage node 150 and the storage unit 152 in certain embodiments.

In the storage unit 152 on the network, the storage unit 152 can directly communicate with the compute node 626. Such communication embeds the compute node identifier in the request so that the storage unit 152 returns the data directly to the compute node 626 instead of returning the data to the storage node 150 and then to the compute node 626. Including letting you. Direct data connection and data caching are enabled for compute node 626, which has the intelligence to find data in storage unit 152. The compute node 626 can also be used for special processing in the data pipeline that performs filtering, conversion, etc. on the data that proceeds to or arrives from the storage unit 152. Therefore, the architecture shown in FIGS. 6A-9C demonstrates flexibility in the arrangement of components and communication between components in storage systems as well as storage and computing systems. Based on data throughput and communication throughput, as well as the needs and expected growth of absolute or relative quantities of data and computing capabilities, one architecture is more suitable than another. Storage capacity and computational capacity are adjustable, expandable, and adjustable in various embodiments. In addition, the embodiment provides greater flexibility for load balancing.

The storage cluster 160 is generally in contrast to the storage array in the various embodiments disclosed herein. The storage node 150 is a part of the set that creates the storage cluster 160. Each storage node 150 owns a slice of data and performs the calculations required to provide the data. A large number of storage nodes 150 are required to collaborate to store and retrieve data. Generally, the storage memory or storage device used for a storage array does not include much data processing and manipulation. A storage memory or storage device in a storage array receives commands to read, write, or erase data. Storage memory or storage devices in a storage array do not know what the large system or data in which they are embedded means. Storage memory or storage devices in a storage array include various types of storage memory, such as RAM, solid state drives, hard disk drives, and the like. The storage unit 152 described herein has a large number of interfaces that are active at the same time and serve many purposes. In one embodiment, some of the functions of the storage node 150 are shifted to the storage unit 152, converting the storage unit 152 into a combination of the storage unit 152 and the storage node 150. Putting the calculation (for storage data) into the storage unit 152 makes this calculation closer to the data itself. Various system embodiments have storage node layer hierarchy with different capabilities. In contrast, in a storage array, the controller owns and knows everything about all the data that the controller manages on the shelf or storage device. In the storage cluster 160 described herein, a large number of storage units 152 and / or a large number of controllers of storage nodes 150 are used in various ways (eg, erase coding, data fragmentation, metadata communication and redundancy, storage capacity expansion or contraction). Collaborate (for data recovery, etc.).

FIG. 10 is a flowchart of a storage cluster operation method that can be implemented in or according to an embodiment of a storage cluster, a storage node and / or a non-volatile solid-state storage or storage unit according to an embodiment. In action 1002, the first storage unit receives instructions regarding metadata or a portion of user data from the storage nodes of the storage cluster. For example, the instructions store a portion or data fragment of user data, read a portion or data fragment of user data, construct data from the data fragment, and read or write a parity fragment, health, state or performance. Includes instructions for responding to, etc.

In action 1004, the first storage unit communicates directly with the second storage unit via a route that does not require assistance from the storage node (s). This communication includes communication relating to a portion of user data or metadata. A good example of communication regarding metadata is heartbeat communication (which is related to instructions that respond to health, condition, or performance). An example of communication for a portion of user data is requesting a data fragment from another storage unit or sending a parity fragment to another storage unit for writing to the flash memory of that storage unit. Yet another example is easily devised in light of the teachings described herein. In action 1006, the second storage unit receives communication from the first storage unit via a route. More specifically, the second storage unit receives communication directly from the first storage unit rather than from the storage node.

In action 1008, the second storage unit determines the action based on the communication from the first storage unit. Based on the content of the communication, the second storage unit stores the data, stores the metadata, reads the data or metadata, and returns it to the first storage unit, from the first storage unit. Answer questions, and so on. Where appropriate, the response is sent from the second storage unit to the first storage unit, or to another storage unit via a route that does not require assistance from the storage node (s). Will be sent. Alternatively, for the second storage unit, an action is taken to communicate with one of the storage nodes or the compute node. Yet another example of an action is easily devised in light of the teachings presented here.

It is clear that the methods described here are carried out in digital processing systems such as conventional general purpose computer systems. A special purpose computer designed or programmed to perform only one function may be used as an alternative. FIG. 11 is a diagram showing a normative computing device that embodies the embodiments described herein. The computing device of FIG. 11 is used to carry out an embodiment of the function of a storage node or non-volatile solid state storage unit according to certain embodiments. The computing device includes a central processing unit (CPU) 1101, which is coupled to memory 1103 and mass storage device 1107 through bus 1105. Mass storage device 1107 represents a persistent data storage device, such as a disk drive that is local or remote in certain embodiments. Mass storage device 1107 can, in certain embodiments, implement backup storage. The memory 1103 includes a read-only memory, a random access memory, and the like. Applications resident in a computing device are stored or accessed in certain embodiments via a computer-readable medium such as memory 1103 or mass storage device 1107. The application may also be in the form of a modulated electronic signal accessed via a network modem or other network interface of a computing device. It is clear that the CPU 1101 may, in certain embodiments, be implemented in a general purpose processor, a special purpose processor, or a logic device of a special program.

The display 1111 communicates with the CPU 1101, the memory 1103, and the mass storage device 1107 through the bus 1105. Display 1111 is configured to display visualization tools or reports associated with the systems described herein. The input / output device 1109 is coupled to the bus 505 to communicate command selection information to the CPU 1101. Data to and from the external device is communicated through the input / output device 1109. The CPU 1101 is defined to perform the functions described herein in order to enable the functions described with reference to FIGS. 1-6. In certain embodiments, the code that performs this function is stored in memory 1103 or mass storage device 1107 for execution by a processor such as CPU 1101. The operating system of the computing device is MS-WINDOWS ^TM , UNIX ^TM , LINUX ^TM , iOS ^TM , CentOS ^TM , Android ^TM , Redhat Linux ^TM , z / OS ^TM , or other known operating system. It is clear that the embodiments described here can also be integrated with a virtual computing system.

Detailed normative embodiments are disclosed herein. However, the particular features disclosed herein are merely for the purpose of illustrating embodiments. However, those embodiments may be implemented in a number of other embodiments and are not limited to the embodiments described herein.

It should be understood that the terms first, second, etc. are used herein to describe the various steps or calculations, but those steps or calculations are not limited by those terms. Those terms are used only to distinguish one step or calculation from another. Without departing from the scope of the present disclosure, for example, the first calculation can be referred to as the second calculation, and similarly, the second step can be referred to as the first step. The word "and / or" and the "/" symbol used herein include any and all combinations of one or more related list items.

The singular forms "a", "an" and "the" as used herein shall also include the plural form unless the context explicitly indicates. “Comprises”, “comprising”, “includes” and / or “including”, when used herein, are the features, integers, steps, described above. Identify the existence of operations, elements, and / or components, but do not preclude the existence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Please understand further. Therefore, the terms used herein are merely to describe a particular embodiment and are not limited thereto.

It should also be noted that in another embodiment, the functions / actions described herein may be performed in a different order than shown. For example, the two figures shown in succession may be executed substantially simultaneously, or sometimes in reverse order, based on the functions / actions included.

With the above embodiments in mind, it should be understood that those embodiments can use various computer-implemented operations with data stored in the computer system. Those operations require physical manipulation of physical quantities. Usually, but not always, their quantities take the form of electronic or magnetic signals that can be stored, transferred, synthesized, compared, and otherwise manipulated. Moreover, the operations performed are often referred to by the terms generation, identification, or comparison. All of the operations described herein that form part of an embodiment are useful machine operations. The embodiments also relate to devices or devices for performing those operations. The device may be specially configured for the requested purpose, or the device may be a general purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general purpose machines may be used with computer programs written according to the teachings described herein, or it is more convenient to configure special equipment to perform the requested operation.

Modules, applications, layers, agents, or other methods-workable entities can be implemented as hardware, firmware, or processor execution software, or a combination thereof. Also, although software-based embodiments have been disclosed herein, it is clear that the software can be implemented on a physical machine such as a controller. For example, the controller includes a first module and a second module. The controller can be configured to perform various actions of, for example, a method, application, layer, or agent.

The embodiment can also be implemented as a computer-readable code on a non-temporary computer-readable medium. A computer-readable medium is a data storage device that stores data and then is read by a computer system. Computer-readable media include, for example, hard drives, network-attached storage (NAS), read-only memory, random access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical tapes. Includes optical data storage equipment. Computer-readable media can also be distributed across network-coupled computer systems so that computer-readable code is stored and executed in distributed form. The embodiments described herein can be implemented in various computer system configurations including handheld devices, tablets, microprocessor systems, microprocessor-based or programmable consumer electronic devices, minicomputers, mainframe computers, and the like. The embodiments can also be implemented in a decentralized computing environment in which tasks are performed by remote processing devices linked through wire-based or wireless networks.

The operations of the method have been described in a particular order, but other operations may be performed between the operations described here, such that the operations described here are performed at slightly different times. Alternatively, the operations described herein may be distributed in a system that allows processing operations to be performed at various processing-related intervals.

In various embodiments, one or more parts of the methods and mechanisms described herein may form part of a cloud computing environment. In such an embodiment, resources are provided via the Internet as a service according to one or more different models. Such a model includes the infrastructure as a service (IaaS), the platform as a service (PaaS), and the software as a service (SaaS). In Infrastructure as a Service, computer infrastructure is provided as a service. In such cases, the computing device is generally owned and operated by the service provider. In the PaaS model, software tools and basic equipment used by developers to develop software solutions are provided and hosted as services by service providers. SaaS typically includes a service provider that licenses the software as a service on demand. The service provider may host the software or deploy the software to the customer during a given period of time. Many combinations of the above models are conceivable and intended.

The various units, circuits or other components are described or claimed as "configured" to perform a task (s). In such a context, the phrase "configured" by indicating that a unit / circuit / component includes a structure (eg, a circuit) that performs a task (s) during operation. Used to imply a structure. Thus, a unit / circuit / component can be said to be configured to perform a task even if a particular unit / circuit / component is not currently operating (eg, not on). Units / circuits / components used with the term "configured" include hardware, eg, memory, circuits that store program instructions that can be executed to embody operations and the like. To state that a unit / circuit / component is "configured" to perform one or more tasks is 35 U.S. S. C. It is clearly intended not to cite 112, Section 6. In addition, "configured" is a general structure operated by software and / or firmware (eg, FPGA or general purpose processor execution software) to operate to perform a controversial task (s). Includes (eg, general circuits). "Constructing" also includes adapting a manufacturing process (eg, a semiconductor manufacturing facility) to manufacture a device (eg, an integrated circuit) for performing or performing one or more tasks.

The present invention has been described above with reference to the specific embodiments. However, the above description is not exhaustive and does not limit the present invention to the exact form disclosed herein. Many changes and modifications can be considered in view of the above teachings. The embodiments have been selected and described to best explain the principles of those embodiments and their practical applications, and will be adapted by those skilled in the art to suit their intended specific use. To make the best use of. Therefore, embodiments of the present invention are exemplary and not limited thereto, and the present invention is not limited to the details described herein and may be modified within the scope of the claims and their equivalents. ..

102: Enterprise computing system 104: Processing resource 106: Network resource 108: Storage resource 110: Processing controller 112: Network controller 114: Storage controller 116: Processor 118: RAM
120: Router 122: Switch 124: Server 126: Hard drive 128: Flash storage 130: Flash controller 132: Flash memory 138: Chassis 140: Network 142: Slot 144: Fan 146: Switch fabric 148: Flap 150: Storage node 152: Non-volatile solid state storage 154: Memory 156: CPU
158: Printed circuit board 160: Storage cluster 168: Authority 170: Communication interconnection 172: Distribution 174, 176: External port 178: External power supply 204: Non-volatile random access memory (NVRAM)
206: Flash memory 208: Programmable logic device (PLD)
210: Input / output (I / O) port 212: Controller 214: Direct memory access unit (DMA)
216: Dynamic Random Access Memory (DRAM)
218: Energy storage 222: Flash die 226: Registers 602, 604, 612, 614: Path 606: Power supply 616: First path 618: Second path 620: Network switch 622, 624: Path 626: Computation node 628: Processor Complex 630: Port 1101: CPU
1103: Memory 1105: Bus 1107: Mass storage device 1109: Input / output device 1111: Display

Claims (19)

In the storage system
It has multiple storage units, each of which has a storage memory for user data.
A plurality of storage nodes are further provided, and each of the plurality of storage nodes is configured to have ownership of a part of the user data, and includes a first path for connecting the plurality of storage units, and the plurality of storages. Each of the units is allowed to communicate with at least one other storage unit of the plurality of storage units via a first path without assistance from the plurality of storage nodes, the first path including a network switch.
Storage system. The first aspect of the present invention, wherein the second path for connecting the plurality of storage nodes is provided, and each of the plurality of storage nodes can communicate with each other storage unit of the plurality of storage nodes via the second path. Storage system. The first portion of the first path that joins the first subset of the plurality of storage units.
Further comprising, a storage system according to claim 1. A second path for connecting the plurality of storage nodes is provided.
The storage system according to claim 1, wherein the first path connects each of the plurality of storage units to each other storage unit of the plurality of storage units. The storage system according to claim 3 , further comprising a second portion of the first path that combines the second subsets of the plurality of storage units . The storage system according to claim 1, wherein the storage memory includes a solid state storage memory. Are coupled to each other, and, further Ru comprising a plurality of compute nodes for coupling said plurality of storage nodes through said first path,
The storage system according to claim 1. In a storage cluster
It comprises a plurality of slots, each slot being configured to accept a storage node or storage unit, each of which can occupy one or more of the plurality of slots.
Equipped with multiple storage units
Each of the plurality of storage units has a solid state storage memory for user data.
A first path is provided, and the first path intervenes in or supports communication from the first storage unit of the plurality of storage units to the second storage unit of the plurality of storage units via the first path. storage node to the plurality of storage units coupled so as not,
It comprises at least one network switch, the at least one network switch including said first path.
A plurality of storage nodes are provided, and the at least one network switch combines the plurality of storage nodes.
Storage cluster. A second path that connects to the plurality of storage nodes is further provided .
The storage cluster according to claim 8. Said first path couples each of the plurality of storage units to each other storage units of said plurality of storage units,
The storage cluster according to claim 8. With one or more compute nodes
Each of the one or more computing nodes, which have the one or more processor complex,
The storage cluster according to claim 8. With one or more compute nodes
The network switch, each of the one or more computing nodes, each of the storage nodes,
And combine each of the plurality of storage units together.
The storage cluster according to claim 8. The first path joins each of the one or more processor complexes of each of the one or more compute nodes together.
The storage cluster according to claim 12 . In the storage system
Equipped with multiple storage units
Each of the plurality of storage units has a solid state storage memory for user data.
A first path is further provided, and each of the plurality of storage units is coupled so as to communicate with at least one other storage unit of the plurality of storage units via the first path.
A plurality of storage nodes are further provided, and each of the plurality of storage nodes has a subset of a plurality of storage units in the plurality of storage nodes and has ownership of a part of user data, and the plurality of storage nodes are storage units. Not involved in direct communication between
Further provided with a second path for connecting the plurality of storage nodes ,
as well as
At least one network switch, the first path and the second path are included in the at least one network switch.
Storage system. The first path includes a first bus that connects the first and second storage units of the plurality of storage units to each other and to the first storage node of the plurality of storage nodes.
Said first path, said plurality of storage units of the third and fourth storage units to each other and second bus including binding to a second storage node of the plurality of storage nodes,
The storage system according to claim 14 . The first path includes a bus that connects the plurality of storage units, and this bus is not connected to the plurality of storage nodes except through the plurality of storage units.
The storage system according to claim 14 . Each of the plurality of storage units is coupled to communicate with each other storage unit of the plurality of storage units via the first path, and the storage node is such a direct communication between the storage units. 14. The storage system of claim 14 , which is not involved in. Further comprising at least one network switch, the first path and the second path is included in at least one network switch, storage system according to claim 14. Further comprising one or more computing nodes, each of which have a one or more processors complex storage system according to claim 14.

Images