DE112021000246T5 - MESSAGING FROM THE PEER STORAGE DEVICE VIA THE CONTROL BUS - Google Patents

Abstract

Systems and methods for messaging peer storage devices over a control bus to swap out and save internal operational data are disclosed. Storage devices may include a host interface configured for connection to a control system and a control bus interface for connection to a control bus. Peer storage devices can establish peer communication via the control bus interface, select internal operational data, and send the internal operational data to other peer storage devices via the control bus. In response to a recovery event, the peer storage devices may receive the paged out internal operational data from another peer storage device.

Description

TECHNICAL AREA

The present disclosure relates generally to computing system devices that are interconnected via a low-bandwidth bus, and more particularly to communication of peer devices via a low-bandwidth control bus without interference with the host system.

BACKGROUND

Some computing systems, such as storage arrays, may include multiple storage devices that support one or more host systems via a peripheral or storage interface bus, such as PCIe (Peripheral Component Interconnect Express), SATA (Serial Advanced Technology Attachment), or SAS (Serial Attached [Small Computer Serial Interface - SCSI]). These interfaces can be set up for high-bandwidth transfer rates, such as 3 Gigabit (Gb)/second (s), 6 Gb/s, 8 Gb/s, 12 Gb/s, 16 Gb/s, etc. to provide fast and to enable reliable data transfer to and from storage devices, with some versions even supporting 512 Gb/s.

Multi-device storage systems use multiple discrete storage devices, typically disk drives (solid state drives, hard drives, hybrid drives, tape drives, etc.) to store large amounts of data. These multi-device storage systems are generally arranged in an array of drives that are linked by a common communications fabric and in many cases controlled by a storage controller, redundant array of independent disks (RAID) controller, or general controller to provide storage and coordinate system activities about the placement of drives. The data stored in the array can be stored according to a defined RAID level, a combination of RAID schemes, or other configurations to achieve the desired data redundancy, performance, and capacity utilization. In general, these data storage configurations may involve a combination of redundant copies (mirroring), data striping and/or parity (computation and storage), as well as other data management, error correction, and data recovery processes that are sometimes specific to the type of drives used (eg. B. Solid State Drives versus Hard Disk Drives).

Each storage device in a multiple-device storage system can be connected to a host system via at least one high-bandwidth interface, such as PCle, using a storage protocol appropriate to the storage device, such as a non-volatile memory express protocol (NVMe ) to access solid-state drives (SSDs) or the storage blades of any flash array. Some multiple device storage systems utilize storage devices capable of communicating with each other and/or with host systems via the interconnect and/or network fabric and a high bandwidth interface. Such fabric-based distributed storage systems may include storage devices configured to allow more efficient data transfer to and from hosts and other systems via direct memory access.

In some configurations, each storage device may also include a different interface configured for power management and/or communication with low-bandwidth computing devices sharing the same power or control bus. For example, memory devices may include a control bus interface that conforms to I2C (inter-integrated circuit), I3C (improved inter-integrated circuit), SMBus (system management bus), or similar low-bandwidth control bus protocols. These command buses can interconnect the storage devices within a multi-device storage system and provide a synchronous, packet-switched, serial communication bus with multiple masters. In some computer systems, the control bus pins may be included as part of the physical connector of the peripheral interface. For example, a PCIe physical interface connector may include a set of I2C, I3C, or SMBus pins at one end of the connector to provide the low-bandwidth control interface. Compared to the high-bandwidth peripheral interfaces used for data transfer, these low-bandwidth control bus interfaces can support ~3 to 30 megabits (Mb)/s and operate at up to 100 kilobits (kb)/s.

In some systems, the low-bandwidth control bus can be initialized early in the boot cycle of the storage devices for communication, to facilitate power management, detection, and access to external resources such as temperature sensors, fan controls, light emitting diode (LED) displays, etc support. Storage devices may have access to the control bus prior to communication with the host or control over the high bandwidth peripheral bus.

Storage devices may encounter errors and/or failures that prevent host communication prior to establishing host communication over the high bandwidth peripheral bus. As a result, internal operational data such as reset events, debug data, device logs, and configuration information may be lost in individual storage devices, particularly in the event of an unrecoverable error that forces a system or storage device reset.

It may be advantageous to offload and/or back up internal operational data without waiting for or relying on the host system. A reliable method may be required to offload internal operational data during pre-host operation and/or without relying on communication with the host via peripheral or memory buses.

EXECUTIVE SUMMARY

Various aspects of peer storage device communication are described, particularly messaging between peer storage devices over a low-bandwidth control bus for offloading or backing up internal operational data.

A general aspect includes a system including a first device that may include: a processor; a memory; a host interface configured to connect to a host system; a control bus interface arranged to be connected to a control bus, wherein a plurality of peer devices arranged to be connected to the control bus; and a peer sharing service. The peer enabling service may be configured to establish peer communication with a target peer device of the plurality of peer devices via the control bus interface; select internal operating data for the first device; send the selected internal operational data to the target peer device; and receive the selected internal operational data from the target peer device in response to a recovery event.

Implementations may include one or more of the following features. The first device may further include: a storage medium configured to store host data received via the host interface; a memory interface protocol configured to communicate with the host system over the host interface; and a non-volatile storage device controller configured to manage input/output of data to the storage medium. The storage medium may include at least one non-volatile storage device. The memory interface protocol may be a non-volatile memory express protocol (NVMe). The plurality of peer devices may include a plurality of peer storage devices, and each peer storage device of the plurality of peer storage devices may include: a peer host interface configured to communicate using the NVMe -Protocol connects to the host system; a peer storage medium may include at least one non-volatile memory; and a peer control bus interface configured to connect to the control bus. The peer sharing service may further include a discovery service configured to: transmit a device identifier for the first device over the control bus interface; receive device identifiers from the plurality of peer devices via the control bus interface; and store the peer device identifiers in a peer device register. The peer sharing service may be further configured to: determine a peer device identifier for the target peer device; and format a peer message including the selected internal operational data as a message payload. Sending the selected internal operational data may include sending the peer message to the target peer device using the peer device identifier for addressing. The peer sharing service may further include an encryption service configured to encrypt the selected internal operational data prior to sending the peer message. The internal operational data can be selected from: reset event data; debug data; log data related to the operation of the device; or last good configuration data. The peer enabling service may be further configured to receive internal operational data from at least one of the plurality of peer devices via the control bus interface; store the internal peer operational data in a peer data partition in memory; and send the internal peer operational data to the at least one peer device in response to a peer recovery event. The system can continue to support a variety of peer Include devices, wherein the plurality of peer devices store the internal operational data received from the first device, and the peer sharing service may further include a recovery service configured to: determine the recovery event; in response to the recovery event and via the control bus interface, send a recovery request to at least one peer device of the plurality of peer devices; receive internal recovery mode data from the at least one peer device; and store the internal recovery operation data in an internal operation storage location. The recovery event may occur in a pre-host operational state. The internal recovery mode data may include a last good configuration data page, and the first device may be further configured to complete pre-host operations using the last good configuration data page received from the at least one peer device.

Another general aspect includes a computer-implemented method of: establishing a peer communication via a first device of a plurality of peer target devices of a plurality of peer devices, wherein the first device and each peer device of the plurality of peer devices includes: a host interface arranged to be connected to a host system and a control bus interface arranged to be connected to a control bus; selecting internal operating data for the first device; sending the selected internal operational data from the first device and over the control bus interface to the target peer device; and receiving the selected internal operational data from the target peer device to the first device and over the control bus interface, the receipt of the selected internal operational data being responsive to a recovery event.

Implementations may include one or more of the following features. The first device and each peer device of the plurality of peer devices may include: a storage medium configured to store host data received via the host interface; a memory interface protocol configured to communicate with the host system over the host interface; and a non-volatile storage device controller configured to manage input/output of data to the storage medium. The computer-implemented method may further include: sending a device identifier for the first device over the control bus interface; receiving peer device identifiers from a plurality of peer devices over the control bus interface; and storing the peer device identifiers in the first device in a peer device register. The computer-implemented method may further include: determining, by the first device, a peer device identifier for the target peer device; and formatting a peer message including the selected internal operational data as a message payload, wherein sending the selected internal operational data includes sending the peer message to the target peer device using the peer device identifier for addressing. The computer-implemented method may further include encrypting the selected internal operational data prior to sending the peer message. The internal operational data can be selected from: reset event data; debug data; log data related to the operation of the device; or last good configuration data. The computer-implemented method may include: receiving internal operational data from at least one peer device of the plurality of peer devices to the first device via the control bus interface; storing the peer's internal operational data in a memory of the first device in a peer data partition in the memory; and sending, by the first device and in response to a peer recovery event, the peer's internal operational data to the at least one peer device. The computer-implemented method may further include: determining, by the first device, the recovery event; sending a recovery request over the control bus interface to at least one peer device of the plurality of peer devices in response to the recovery event; receiving internal recovery mode data from at least one peer device and over the control bus interface; and storing the internal recovery operation data in an internal operation storage location. The computer-implemented method may further include the first device completing pre-host operations using a last good configuration data page received from the peer target device, the recovery event occurring in a pre-host operational state and the selected internal Operational data include the last good configuration data page.

Yet another general aspect includes a storage system having a control bus and a plurality of storage devices, wherein each storage device of the plurality of storage devices may include: a processor; a memory; a host interface configured to connect to a host system; a control bus interface configured to connect to the control bus; means for establishing peer communication between the plurality of storage devices via the control bus interface; means for selecting internal operational data from an internal operational storage location; means for sending the selected internal operational data to at least one other storage device of the plurality of storage devices; and means for receiving the selected internal operational data from at least one other peer storage device of the plurality of peer storage devices in response to a recovery event.

The various embodiments advantageously apply the principles of storage devices and/or multi-device storage systems to enhance the functionality of such computing systems. The various embodiments include operations that overcome or at least reduce the problems previously encountered with storage arrays and/or systems and are accordingly more reliable and/or efficient than other computing systems. That is, the various embodiments disclosed herein include hardware and/or software with functions to enhance communication and security of internal operational data, such as by using peer messages over a control bus between storage devices. Accordingly, the embodiments disclosed herein provide various improvements to storage area networks and/or storage systems.

It should be understood that the terminology used in the present disclosure is primarily selected for convenience and instructional purposes and is not intended to limit the scope of the subject matter disclosed herein.

character list

• 1 FIG. 12 schematically illustrates a multiple device memory system having a memory interface bus and a control bus.
• 2 FIG. 12 schematically illustrates a peer discovery architecture implemented by the multi-device storage system 1 can be used.
• 3 FIG. 12 schematically illustrates a peer messaging architecture emanating from the multi-device storage system 1 can be used.
• 4 FIG. 12 schematically illustrates a host node of the multi-device storage system 1 .
• 5 Figure 1 schematically illustrates some elements of the storage devices 1-5 more accurate.
• 6 Figure 12 is a flow diagram of an example method for peer messaging between storage devices using a control bus.
• 7 Figure 12 is a flow diagram of an example method for restoring internal operational data from a peer storage device.
• 8th Figure 12 is a flow diagram of an example method for supporting peer recovery of internal operational data.

DETAILED DESCRIPTION

1 1 shows one embodiment of an exemplary data storage system 100 having data storage devices 120 interconnected via both memory interface bus 108 and control bus 110. FIG. Although some exemplary features are illustrated, various other features have not been illustrated for the sake of brevity and so as not to unnecessarily distract from the pertinent aspects of the embodiments described herein. To this end, data storage system 100 includes, by way of non-limiting example, one or more data storage devices 120 (sometimes also called information storage devices, storage devices, disk drives, or disk drives). In some embodiments, the storage devices 120 may be implemented in a server or storage array blade or similar storage unit for use in data center storage racks or enclosures. Storage devices 120 may be coupled to one or more hosts 102 and provide data storage and retrieval functions to or through those host systems. In some embodiments, storage devices 120 may be arranged in a storage hierarchy that includes storage nodes, storage controllers, and/or other intermediate components between storage devices 120 and host 102 . For example, each controller may be responsible for a corresponding set of storage nodes and their respective storage devices literal, connected via a corresponding backplane network, although only storage devices 120 and host 102 are shown.

In the embodiment shown, multiple storage devices 120 are mounted on a common storage interface bus 108 for host communication. For example, storage devices 120 may include a number of drives arranged in a storage array, such as storage devices that share a common rack, unit, or blade in a data center, or the SSDs in an all-flash array. In some embodiments, storage devices 120 may share a backplane network, network switch(es), and/or other hardware and software components accessed via storage interface bus 108 and/or control bus 110 . For example, storage devices 120 may be coupled to storage interface bus 108 and/or control bus 110 via a variety of physical connectors that define physical, transport, and other logical channels for communication with the various components and subcomponents to provide a communication channel to the host 102 to provide. In some embodiments, memory interface bus 108 may provide the primary host interface for memory device management and host data transfer, and control interface bus 110 may include limited connectivity to the host for low-level control functions.

In some embodiments, storage devices 120 may be referred to as a peer group or peer storage devices because they are interconnected via memory interface bus 108 and/or control bus 110 . In some embodiments, storage devices 120 may be configured for peer communication between storage devices 120 over memory interface bus 108, with or without host 102 assistance. For example, storage devices 120 may be configured for direct memory access using one or more protocols , such as express non-volatile memory (NVMe), remote direct memory access (RDMA), NVMe over fabric (NVMeOF), etc., to provide command messages and data transfers between storage devices over the high-bandwidth memory interface and memory interface bus 108 .

In some embodiments, the storage devices 120 may be configured for peer communication using multi-master discovery and messages according to a low-bandwidth interface standard. For example, the storage devices 120 for packet-based messages may be implemented over the control bus 110 using a low-bandwidth bus protocol such as I2C (Inter-integrated Circuit), I3C (Improved Inter-integrated Circuit), SMBus (System Management Bus), etc Storage devices 120 may be interconnected via a common control bus to provide a low-bandwidth communication channel to host 102 and other system components to aid in power management, detection and access to external resources such as temperature sensors, fan controls, LED displays, etc. For example, the control bus 110 may connect the storage devices 120 to a baseboard management controller (BMC) to monitor the physical state of the storage devices 120 for the host 102 . The storage devices 120 can be defined as peer storage devices due to their connection to a common control bus 110 .

In some embodiments, data storage devices 120 are or include solid state drives (SSDs). Each data storage device 120.1 to 120.n may include a non-volatile memory device (NVM device) or device controller 130 based on computing resources (processor and memory) and a plurality of NVM or media devices 140 for data storage (e.g. one or more NVM devices (en), such as one or more flash memory devices). In some embodiments, a respective data storage device 120 of the one or more data storage devices includes one or more NVM controllers, such as flash controllers or channel controllers (e.g., for storage devices with NVM devices in multiple memory channels). In some embodiments, data storage devices 120 may each be housed in an enclosure, such as a multi-piece sealed enclosure having a defined form factor and ports and/or connectors for connection to memory interface bus 108 and/or control bus 110.

In some embodiments, a respective data storage device 120 may include a single medium, while in other embodiments, the respective data storage device 120 may include a plurality of media includes directions. In some embodiments, the media devices include NAND flash memory or NOR flash memory. In some embodiments, data storage device 120 may include one or more hard disk drives (HDDs). In some embodiments, data storage devices 120 may include a flash memory device, which in turn includes one or more flash memory chips, one or more flash memory packages, one or more flash memory channels, or the like. However, in some embodiments, one or more of the data storage devices 120 may hold other types of non-volatile data storage media (e.g., phase change random access memory (PCRAM), resistive random access memory (ReRAM), spin transfer torque random access memory (STT-RAM), magnetoresistive random access memory). (MRAM), etc.).

In some embodiments, each storage device 120 includes a device controller 130 that includes one or more processing units (sometime called CPUs or processors or microprocessors or microcontrollers) configured to execute instructions in one or more programs. In some embodiments, the one or more processors are shared by one or more components within and, in some cases, beyond the function of the device controllers. The media devices 140 are coupled to the device controllers 130 via connections that convey commands, typically in addition to the data, and optionally, metadata, error correction information, and/or other information in addition to the data values to be stored in the media devices and the data values read from the media devices 140 . Media devices 140 may include any number of (i.e., one or more) memory devices including, without limitation, non-volatile solid-state memory devices such as flash memory device(s).

In some embodiments, media devices 140 in storage devices 120 are divided into a number of addressable and individually selectable blocks, sometimes referred to as erase blocks. In some embodiments, the individually selectable blocks are the minimum sized erasable units in a flash memory device. In other words, each block contains the minimum number of memory cells that can be erased simultaneously (i.e., in a single erase operation). Each block is typically further divided into a plurality of pages and/or wordlines, with each page or wordline typically being an instance of the smallest individually accessible (readable) portion in a block. However, in some embodiments (e.g., using some types of flash memory), the smallest individually accessible unit of a data set is a sector or codeword, which is a subunit of a page. That is, a block includes a plurality of pages, each page containing a plurality of sectors or codewords, and each sector or codeword is the minimum data unit for reading data from the flash memory device.

A data unit can describe any size assignment of data, such as a host block, a data object, a sector, a page, a multilevel page, an erase/programming block, a media device/packet, etc. Storage spaces can be physical and /or include logical locations on the storage devices 120 and are described and/or assigned at different levels of granularity depending on the storage medium, configuration of the storage device/system, and/or context. For example, storage locations in a host's logical block addressing data unit (LBA data unit) may be allocated size and addressability for host read/write purposes, but managed as pages with storage device addressing managed in the media flash translation layer (FTL) in other contexts . Media segments may include physical storage locations on storage devices 120, which may also correspond to one or more logical storage locations. In some embodiments, media segments may include a continuous series of physical storage locations such as contiguous data units on a storage medium, and in the case of flash memory, may correspond to one or more media for erasing or programming the blocks. A logical data group may include a plurality of logical data entities that may be grouped on a logical basis regardless of storage location, such as data objects, files, or other logical data constructs composed of multiple host blocks.

In some embodiments, host or host system 102 may be connected to data storage system 100 via a network interface that is part of the host fabric network and includes storage interface bus 108 as a host fabric interface. In some embodiments, multiple host systems 102 (of which 1 only one is shown) over the structure network with the data storage system 100 connected. The network may include a storage network interface or other interface that supports communication with multiple host systems 102 . The fabric network may include a wired and/or wireless network (e.g., public and/or private computer networks of any number and/or configuration) that may be coupled in any suitable manner for data transmission. The fabric network may include, for example, any means of a conventional data communications network, such as a local area network (LAN), a wide area network (WAN), a telephone network such as the public switched telephone network (PSTN), an intranet, the Internet, or any other suitable communications network, or any combination of communication networks.

Host system 102, or a corresponding host in a multiple host system, may be any suitable computing device such as a computer, computer server, laptop, tablet, netbook, internet kiosk, personal digital assistant, a Mobile phone, smartphone, gaming device or other computing device. The host system 102 is also sometimes referred to as a host, client, or client system. In some embodiments, host system 102 is a server system, such as a server system in a data center, or a storage system, such as a storage array in a data center. In some embodiments, the one or more host systems 102 is one or more host devices other than a storage controller or storage node that houses the plurality of storage devices 120 . The one or more host systems 102 may be configured to store and access data on the plurality of storage devices 120 .

Host system 102 may include one or more central processing units (CPUs) 104 for performing computations or instructions to access storage devices 120 via memory interface bus 108 . In some embodiments, CPU 104 may include a processor and may be coupled to operational memory (not shown) to perform both memory operations and a memory interface protocol compatible with memory interface bus 108 and storage devices 120 . In some embodiments, a separate memory interface unit (not shown) may provide the memory interface protocol and associated processor and memory resources. From the storage devices 120 perspective, the storage interface bus 108 may be referred to as a host interface bus and provides a path for host data between the storage devices 120 and the host 102 .

Host system 102 may include a BMC 106 configured to monitor the physical condition of host 102, storage devices 120, and/or other components of data storage system 100. In some embodiments, BMC 106 may include processor, memory, sensor, and other resources that are integrated into BMC 106 and/or accessible via control bus 110 . BMC 106 may be configured to measure internal variables within a chassis, adjacent components, and/or the components themselves within host 102 or data storage system 100, such as temperature, humidity, supply voltage, fan speeds, communication parameters, and/or functions of the operating system (OS). BMC 106 may allow systems and components to be cycled on and off or restarted as needed by control signals over control bus 110 . In some embodiments, BMC 106 may be configured to receive status communications from storage devices 120 over control bus 110 during boot cycles before host communications over memory interface bus 108 are initiated.

In some embodiments, data storage system 100 includes one or more processors, one or more types of memory, a display, and/or other user interface components such as a keyboard, touch screen, mouse, trackpad, digital camera, and/or any number of additional ones devices to add functionality. In some embodiments, data storage system 100 has no display or other user interface components.

2 FIG. 12 shows a schematic representation of an exemplary storage system 200, such as the multiple device data storage system 100 of FIG 1 , which is implemented with a peer discovery architecture to support peer messaging over the control bus 110. Storage devices 120 may each be coupled to control bus 110 to communicate with a low-level physical and power management subsystem, such as BMC 106 in FIG 1 in In some embodiments, control bus 110 supports a multi-master packet-based messaging system to allow storage devices 120.1, in addition to their communi communication with BMC 106 or other components of the system via control bus 110 with peer storage devices. Peer messages over control bus 110 may be independent of communications over a primary host interface or a fabric interface such as memory interface bus 108 in 1 , be.

Each peer storage device 120.1-120.n connected to the control bus 110 can be arranged to initiate a discovery message 210. The detection message 210 may be sent to the control bus 110 during startup or initialization. For example, storage device 120 may use the control bus protocol to establish primary or temporary master control of control bus 110 and send a broadcast message to be received by all other devices connected to control bus 110. Storage device 120.1 may thereby send discovery message 210.1 the storage devices 120.2-120.n send, and each peer storage device may do the same during system initialization, power cycle, or reboot. Each peer storage device 120 connected to the control bus 110 may receive a discovery message from any other peer storage device 120 .

In some embodiments, the discovery message 210 may include a device identifier 212 and a sender identifier 214 . Device identifier 212 may include a unique identifier for that storage device relative to all other devices connected to control bus 110 . In some embodiments, device identifier 212 may include a product serial number and/or a unique address assigned to the respective storage device. For example, each device attached to control bus 110 may be assigned a 7-bit address that uniquely identifies that device on control bus 110. Messages destined for that storage device include the address in the message sequence and only the storage device with the matching address processes the content of the message. Messages can also be sent with a reserved broadcast or public address to allow all devices to receive and process the message.

Vendor identifier 214 may include a group identifier that identifies storage devices that meet certain criteria, such as storage devices that are manufactured or sold by a particular company or that are configured according to a particular peer messaging protocol or standard. Vendor identifier 214 may include, for example, a numeric identifier for storage devices from a particular vendor that are enabled for peer messaging over the control bus. Provider identifiers may be defined and assigned by certain manufacturers pursuant to working group or standard agreements or otherwise to ensure that the provider identifiers are unique and used to correctly identify storage devices that are compatible with specific peer messaging protocols and content .

In some embodiments, each storage device 120 may use discovery messages 210 received from peer storage devices to populate an internal data structure for storing the addresses of one or more peer storage devices. For example, each storage device 120 may maintain a peer device table 220 in an internal memory location to list the addresses, device identifiers 212 and/or provider identifiers 214 of every other peer storage device connected to the control bus 110 . In some embodiments, peer device table 220 may include a row, list, or array of peer device identifier entries 222 . For example, each time a discovery message is received with a different device identifier 212 (and includes a corresponding provider identifier 214 identifying the device as a peer storage device for message purposes), that device identifier 212 can be included as a new peer device identifier entry 222 in the peer Device table 220 are entered. After all discovery messages 210 have been received, each storage device 120 may include a complete peer device table 220 with the information required to send messages over the control bus 110 to every other peer storage device.

3 12 shows a schematic representation of an exemplary storage system 300, such as the data storage system with multiple devices 100 in FIG 1 , implemented with a peer message architecture to support various types of peer messages 310, 320, 330 over the control bus 110. The storage devices 120 can each be connected to the control bus 110 to communicate with a subsystem for physical management and low-level power management to communicate, for example the BMC 106 in 1 .

In some embodiments, control bus 110 supports a multi-master packet-based messaging system that may allow storage devices 120.1 to communicate with other storage devices via control bus 110 in addition to communicating with BMC 106 or other system components connected to control bus 110. For example, each device may be able to request or enforce master control or priority to send messages over control bus 110 to peer storage devices operating as slaves. Peer messages over control bus 110 may be independent of communications over a primary host interface or a fabric interface such as memory interface bus 108 in 1 , be.

Each peer storage device 120.1-120.n connected to the control bus 110 can be configured to initiate one or more messages 310, 320, 330 using the addresses and message syntax supported by the control bus 110. Although each example message originates from a particular storage device 120, each storage device 120 may be configured to use any or all types of messages. The peer message 310 may be used by the storage devices 120 to send direct messages to a destination storage device, such as a message containing selected internal operational data for swapping or backup and/or requesting the previously stored internal recovery operational data. Broadcast message 320 may be used by storage devices 120 to broadcast messages to all peer storage devices simultaneously, such as a message containing internal operational data to be redundantly stored or a request to restore previously stored internal recovery operational data , if more than one peer storage device contains the required data. Response message 330 may be used by storage devices 120 to send messages to another storage device that has requested data, such as responding to a restore request by sending the requested internal operational data. Note that the response message 330 includes data payload and, in some embodiments, assumes that the sending storage device has control or priority of the master, and is distinct from simple acknowledgment messages, which may be part of the slave's message handling protocol .

In some embodiments, message 310 may include a sender identifier (312), a destination identifier (314), a sender identifier (316), and payload (318). The sender identifier 312 may include, for example, the device identifier for the sending storage device and/or the control bus address of the sending storage device. The destination identifier 314 may include the device identifier for the storage device and/or the control bus address of the storage device. In some embodiments, destination identifier 314 may include a device identifier, separate from the control bus address, used by devices on control bus 110 to determine the destination. Sender identifier 316 may include a sender identifier as described above with respect to sender identifiers 214 in 2 was described. Payload 318 may include data created by the sending storage device, such as storage device 120.1, and stored by the destination storage device. For example, payload 318 may include internal operational data identified for offloading or backup to other storage devices. In some embodiments, payload 318 may include recovery parameters describing previously swapped or stored data required by the sending storage device to recover data, such as recovery of lost event logs, debug data, or last good page data.

In some embodiments, broadcast message 320 may include sender identifier 322 , broadcast address 324 , sender identifier 326 , and payload 328 . The sender identifier 322 may include, for example, the device identifier for the sending storage device and/or the control bus address of the sending storage device. Broadcast address 324 may include a reserved address, header tag, or similar identifier to identify a broadcast message to be sent to all devices on control bus 110 . Sender identifier 326 may include a sender identifier as described above with respect to sender identifiers 214 in 2 was described. Payload 328 may include data created by the sending storage device, such as storage device 120.1, for storage by one or more peer storage devices. For example, payload 328 may include internal operational data identified for offloading or backing up to other storage devices became. In some embodiments, payload 318 may include recovery parameters describing previously swapped or stored data required by the sending storage device to recover data, such as recovery of lost event logs, debug data, or last good page data.

In some embodiments, the response message 330 may include a sender identifier (332), a destination identifier (334), a sender identifier (336), and a data payload (338). The response message 330 may differ from the peer message 310 in that it is responsive to the peer message 310 being received. For example, peer message 310 may include a restore request containing parameters describing previously stored internal operational data, and response message 330 may respond to peer message 310 with the requested restore data. The sender identifier 332 may include the device identifier for the sending storage device and/or the control bus address of the sending storage device. The destination identifier 334 may include the device identifier for the storage device and/or the control bus address of the storage device, generally the storage device from which the request for restoration was received. Sender identifier 336 may include a sender identifier as described above with respect to sender identifiers 214 in 2 was described. Payload 338 may include data previously stored by the sending storage device from and about the requesting storage device, which may now be the target of the response message 330 storage device. For example, payload 318 may include internal operational data identified for recovery of the target storage device that requested the recovery data.

4 10 shows a schematic representation of an exemplary host system 102. The host system 102 may include a bus 410, a processor 420, local memory 430, one or more optional input devices 440, one or more optional output devices 450, and a communications interface 460. Bus 410 may include one or more conductors that enable communication between host 102 components. Processor 420 can include any type of conventional processor or microprocessor that interprets and executes instructions. The local memory 430 may be a random access memory (RAM) or other type of dynamic storage device that stores information and instructions for execution by the processor 420, and/or a read-only memory (ROM) or another type of static storage device that stores static information and instructions for use by processor 420, and/or any suitable storage element, such as a hard drive or solid-state storage element. An optional input device 440 may include one or more conventional mechanisms that allow an operator to input information into the host 102, such as a keyboard, mouse, pen, speech recognition and/or biometric mechanisms, etc. The optional output device 450 may include one or include several conventional mechanisms that output information to the operator, such as a display, printer, speaker, etc. Communications interface 460 may include any transceiver-like mechanism that allows host 102 to communicate with other devices and/or systems to communicate. In some embodiments, communication interface 460 may include one or more peripheral devices, such as a PCIe interface for connection to storage devices 120.

5 FIG. 12 schematically shows selected modules of a storage device 500 configured for peer messaging, such as storage devices 120. Storage device 500 may include elements and configurations that conform to those described in FIG 1-4 shown are similar. For example, storage device 500 may be implemented as one storage device 120 in a group of peer storage devices interconnected via a control bus and including a host or memory bus interface for data transfer to and from a host in a multiple device data storage system.

Storage device 500 may include a bus 510 interconnecting at least one processor 512, at least one memory 514, and at least one interface, such as memory bus interface 516 and control bus interface 518. Bus 510 may include one or more conductors that enable communication between the components of storage device 500. Processor 512 can include any type of processor or microprocessor that interprets and executes instructions or operations. Memory 514 may be random access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by processor 512, and/or a read-only memory (ROM) or other type of static storage device that stores static information and instructions for use by processor 512, and/or any suitable storage element such as a Hard drive or solid state storage device.

Storage bus interface 516 may include a physical interface for connecting to a host using an interface protocol that supports access to the storage device. For example, storage bus interface 516 may include a PCIe, SATA, SAS, or similar storage interface connector that supports NVMe access to solid-state media having non-volatile storage devices 520 . Control bus interface 518 may include a physical interface for attachment to a control bus that uses a low-bandwidth interface protocol for low-level messaging between computing components. Control bus interface 518 may include, for example, an I2C, I3C, SM Bus, or similar bus interface connector that supports component-to-component messaging, such as multi-master packet-based messaging over a two-wire bus.

Storage device 500 may include one or more non-volatile storage devices 520 configured to store host data. For example, non-volatile storage devices 520 may include a plurality of flash memory packages organized as an addressable memory array. In some embodiments, non-volatile memory devices 520 may include NAND or NOR flash memory having single-level cells (SLC), multi-level cells (MLC), or tri-level cells.

Storage device 500 may include a variety of modules or subsystems that are stored and/or instantiated in memory 514 for execution by processor 512 as instructions or operations. For example, storage 514 may include a host interface 530 configured to receive, process, and respond to data requests from client or host systems. Memory 514 may include a non-volatile memory controller (NVM) 534 configured to manage reads and writes to non-volatile memory devices 520 . Storage 514 may include pre-host operations arranged for low-level operating system and firmware functions within storage device 500 that execute independently of host commands, particularly during promotion, initialization, power transition, or unplanned shutdown (UGSD) the device. Memory 514 may include a control bus protocol configured to manage communications over control bus interface 518 according to a particular protocol or standard. Memory 514 may include internal operational data 550 configured to store low-level operational data for storage device 500 used for device management and debugging. Storage 514 may include a peer sharing service 560 configured for communication between storage devices attached to the same control bus as storage device 500 .

Host interface 530 may include an interface protocol and/or a set of functions and parameters to receive, analyze, respond to, and otherwise manage data requests from a host. For example, host interface 530 may include functions to receive and process host requests to read, write, modify, or otherwise manipulate data blocks and their respective client or host data and/or metadata according to host communication and storage protocols. In some embodiments, host interface 530 may enable direct memory access and/or access via NVMe protocols via memory bus interface 516 to host data units 520.3 stored in non-volatile storage devices 520. For example, host interface 530 may include host communication protocols compatible with PCIe, SATA, SAS, and/or any other bus interface that uses support for NVMe protocols to access host data 520.3.

In some embodiments, host interface 530 may include a variety of hardware and/or software modules configured to use processor 512 and memory 514 to process or manage defined host interface 530 operations. For example, host interface 530 may include a memory interface protocol 532 configured to conform to the physical, transport, and memory application protocols supported by the host for memory bus interface 516 communication. For example, storage interface protocol 532 may include both PCle and NVMe compliant communication, command, and syntax functions, procedures, and data structures. In some embodiments, host interface 530 may include additional modules (not shown) for command processing, buffer management, storage device management and reporting, and other host-side functions.

NVM controller 534 may include an interface protocol and/or a set of functions and parameters for reading, writing, and deleting data units in non-volatile memory devices 520. For example, host interface 530 may include functions for performing host data operations associated with host memory commands received via host interface 530. For example, PUT or write commands may be implemented to write host data units to non-volatile storage devices 520. GET or READ commands can be set up to read data from non-volatile memory devices 520 . DELETE commands can be configured to erase data from non-volatile storage devices 520 or to mark at least one memory location for deletion until a future garbage collection or similar operation actually deletes the data or reallocates the physical memory space for another purpose. In some embodiments, the NVM controller 534 may manage the flash translation layer (FTL), the data state machine, read/write buffer management, NVM device interface protocols, NVM device configuration/management/maintenance, and other device-side tasks include features. In some embodiments, NVM controller 534 may be configured to allocate a portion of the storage locations in non-volatile storage devices 520 for storage of data other than host data 520.3. For example, NVM controller 534 may allocate device data 520.1 as storage locations reserved for internal device data, including device configuration, parameter, and internal operational data. In some embodiments, NVM controller 534 may assign peer data 520.2 as storage locations reserved for internal device data received from and via peer storage devices, including internal operational data from those peer storage devices. In some embodiments, the storage space allocated to device data 520.1 and/or peer data 520.2 may include partitions of the device data and/or peer data that are excluded from the storage capacity provided for host data 520.3, such as hidden from the host over-provisioned storage spaces used to store internal operational data, FTL tables, replace bad blocks, etc.

Pre-host operations 540 may include functions and parameters for managing low-level storage device 500 operations. For example, pre-host operations 540 may include operations performed during startup or initialization of storage device 500 prior to establishing host communications over host interface 530, referred to as a pre-host operational state. In some embodiments, pre-host operations 540 may also include functions and parameters for managing low-level operations of the device during shutdown, power cycle, error, or failure when host communications are temporarily unavailable. Pre-host processes 540 may include, for example, read-only memory, bootloader, firmware, and/or operating system processes that govern the basic operation of storage device 500 as a set of interconnected components and interfaces, independent of the higher-level host data storage functions.

In some embodiments, front-end processes 540 may include a variety of hardware and/or software modules configured to use processor 512 and memory 514 to operate on or manage defined front-end processes 540 . For example, pre-host processes 540 may include read-only memory (ROM) 542 configured to provide the first set of instructions executed by processor 512 at startup to perform basic component recognition, configuration, and to initialize communication between components on the internal bus 510. Pre-host operations 540 may include a boot loader 544 configured to provide and initiate the startup or boot-time tasks and processes to load the operating system and/or firmware 546 for storage device 500 and its components in memory 514 to instantiate. Pre-host operations 540 may include firmware 546 configured to instantiate the data structures, functions, and interfaces for the storage device 500 components to enable them to interact and perform the various functions described herein. In some embodiments, firmware 546 may include host interface 530, NVM controller 534, control bus protocol 548, and peer sharing service 560 initialization. In some embodiments, execution of read-only memory 542, boot loader 544, and/or firmware 546 may create internal operational data 550, such as reset events 552, debug data 554, device logs 556, and last good configuration data 558.

The control bus protocol 548 may include interfaces, functions, and parameters for communication within the host or an associated baseboard management controller (BMC) that use multi-master discovery and messaging according to a low-bandwidth interface standard. For example, storage device 500 may be configured for a packet-based message via control bus interface 518 using a low-bandwidth bus protocol, such as 12C, 13C, SMBus, etc. Control bus protocol 548 may include a variety of hardware and/or software modules, arranged to use processor 512, memory 514, and control bus interface 518 to communicate with host components, including peer storage devices, on the same control bus using a control bus protocol supported by the attached control bus. In some embodiments, control bus protocol 548 can provide a low-bandwidth communication channel with the host and other system components to support power management, detection, and access to external resources such as temperature sensors, fan controls, LED displays, and so on. For example, control bus interface 518 may connect storage device 500 to a BMC for monitoring the physical state of storage device 500. In some embodiments, control bus protocol 548 may provide the detection and messaging functions used by peer sharing service 560.

Internal operational data 550 may include data structures and associated functions for instantiating data related to internal storage device operations in internal operational memory locations in memory 514 and/or device data 520.1 in non-volatile storage devices 520. Internal operational data 550 may include, for example, data created by internal management processes, such as pre-host operations 540, that is written to device data 520.1. In some embodiments, internal operational data 550 may include reset events 552, debug data 554, device logs 556, and/or last good configuration data 558.

For example, reset events 552 may include a reset event record of operating state parameters for storage device 500 at the time a firmware 546 reset and/or a hardware or software reset via control bus interface 518 was triggered. Reset events 552 may include reset operations during ROM 542, bootloader 544, and/or firmware 546 operation. The set of operating state parameters may be retrieved from registers or other data structures in memory 514 representing the current operating state of storage device 500 and stored in device data 520.1 during the reset operation. Debug data 554 may include a set of state parameters, trace data, or similar debug output created by a failed process or service within storage device 500 . For example, the debug data 554 may include process parameters that represent the state of the storage device 500 and/or a process resident therein before a fatal error or crash occurs. Device logs 556 may include one or more sets of operational parameters that relate to internal operations that are recorded in the operational log data over time (and generally ordered by timestamp) and may represent processes occurring in storage device 500 regardless of error , failure or power failure conditions. In some embodiments, device logs 556 may include a power mode log that includes changes in power mode, such as entering a low power mode and/or returning to normal power mode. Last good configuration data 558 may include a set of configuration parameters that were used when storage device 500 was last started and/or operated normally. For example, last good configuration data 558 may include parameters describing a last good page written by storage device 500 and/or the last firmware update processed into a last good configuration data page for a stable configuration. In some embodiments, the last well-written data page and/or a stable firmware version must be found and loaded during initialization or power down as part of a process to recover the device. For example, a storage device that is in recovery can be set up to query peer storage devices for firmware field updates, recover the most recent version (by comparing the version numbers associated with the current storage device firmware version/update and the firmware version/update of the peer devices) and uses them for the recovery process. In some embodiments, it is the most recent Firmware version may not be the firmware version previously used by the device, but may be a firmware field update that was released while the storage device was offline or otherwise unable to receive updates and/or successfully install such updates .

The peer sharing service 560 may include an interface protocol and a set of functions and parameters to find peer storage devices, send and receive messages with those peer storage devices, and/or restore data from those peer storage devices. For example, peer sharing service 560 may include functionality to leverage low bandwidth communication over control bus interface 518 using control bus protocol 548 . In some embodiments, the peer sharing service 560 may include maintaining peer data 520.2 in non-volatile storage devices 520 for storing internal operational data of the peers and/or support data encryption to encrypt internal operational data in transit and/or peer data 520.2 in to save in an encrypted state.

In some embodiments, peer-sharing service 560 may include a variety of hardware and/or software modules configured to use processor 512 and memory 514 to process or manage defined peer-sharing service 560 operations. The peer sharing service 560 may include a discovery service 562, an encryption service 564, a messaging service 566, and/or a recovery service 568, for example. For example, discovery service 562 can be set up to discover peer storage devices on a common control bus and store their addresses for use by messaging service 566 and/or recovery service 568. Encryption service 564 may be configured to encrypt and/or decrypt the payload for messaging service 566 for transmission, storage on peer data 520.2, and/or recovery and storage on device data 520.1. The messaging service 566 can be configured to send and receive one-to-one and/or broadcast messages to and from peer storage devices over the shared control bus. Recovery service 568 may be configured to request and receive data from peer storage devices to support storage device 500 recovery tasks.

Discovery service 562 may include data structures, functions, and interfaces to discover peer devices on a control bus and to determine peer device addresses used by peer enabling service 560 . For example, during initialization or startup of storage device 500, discovery service 562 may broadcast its device identifier or address and receive similar broadcasts from other devices to collect device information for other devices on the control bus. In some embodiments, discovery service 562 may include a peer device register 562.1 configured to store device identifiers, control bus addresses, and/or other information about peer storage devices to establish and support communication with peer storage devices. For example, peer device register 562.1 may include a data table or other data structure in memory 514 and/or device data 520.1 that contains a plurality of device discovery entries associated with corresponding control bus addresses. In some embodiments, discovery service 562 may include a device ID broadcaster 562.2 configured to determine the storage device and/or control bus address for storage device 500 and send a message on the control bus to allow other devices on the control bus to recognize storage device 500. For example, Device ID Broadcaster 562.2 may be triggered during ROM, bootloader, or firmware execution upon startup or initialization of storage device 500, in response to a detection message from another device and/or another device detection trigger according to control bus protocol 548 . In some embodiments, discovery service 562 may include a peer ID receiver 562.3 configured to monitor communications over control bus interface 518, identify device identifiers from other storage devices, and add a corresponding entry for the peer device identifier to peer device register 520.1 . For example, the control bus protocol 548 may define a syntax for broadcast messages, such as a broadcast address, tag, or similar parameter identifying a discovery broadcast and the payload including the storage device identifier, the control bus address, the provider identifier, etc., and the Peer ID receiver 562.3 can be configured to parse messages conforming to this syntax to extract peer device identifiers.

Encryption service 564 may include data structures, functions, and interfaces for encrypting and decrypting internal operational data 550 for exchange and/or storage. For example, encryption service 564 may involve exchanging encryption keys with peer storage devices to enable encrypted message exchange between peer storage devices and/or purely local encryption to allow encrypted internal operational data to be exchanged and/or stored using only the peer - The source storage device is able to decrypt the encrypted data. In some embodiments, when internal operations within storage device 500 create internal operational data 550, selected internal operational data may be encrypted by encryption service 564 and stored in device data 520.1 and/or sent to other peer storage devices via messaging service 566. The messaging service 566 may send the encryption key to the peer storage device to enable the peer storage device to decrypt encrypted payload data before storing it as peer data or upon subsequent access to peer data. In some embodiments, the encryption service 564 may implement a selected encryption algorithm and standard for the internal target operational data, such as Advanced Encryption Standard (AES)-128 or another encryption standard.

Message service 566 may include data structures, functions, and interfaces for sending and receiving messages to and from other peer devices over the control bus. For example, messaging service 566 may implement control bus protocol 548 to send and receive messages to other devices identified via discovery service 562 . In some embodiments, messaging service 566 may send packetized message payloads over the control bus using block write and block read commands between buffers in the peer storage devices.

In some embodiments, message service 566 may include peer enable selector 566.1 configured to select a peer storage device for destination or receipt of a message. For example, peer enable selector 566.1 may include a parameter setting or algorithm for selecting a peer storage device that receives a particular set of internal operational data, such as a selected peer storage device for swapping out or backing up storage device 500, or an algorithm (randomizer, round -Robin, etc.) to distribute internal operational data to peer storage devices. In some embodiments, messaging service 566 may include common trigger events 566.2 configured to determine when selected internal operational data is sent to one or more peer storage devices. For example, enabling trigger events 566.2 can include a variety of rule-based states to trigger the discharge of internal operational data, such as reset events, operational errors, operational mode changes, successful boot, saving a good page, improper shutdown, periodic backup plan, etc. In some According to embodiments, the messaging service 566 may include a common content selector 566.3 configured to select the amount of internal operational data to be offloaded via the messaging service 566. For example, the shared content selector 566.3 may include a set of parameters that determine the types and distance measurements of the internal operational data to be sent to a peer storage device. If multiple common triggering events 566.2 and/or internal operational data types are supported by the peer sharing service 560, a specific common triggering event may determine the type and amount of the externalized internal operational data, such as the debug data for a triggering operational error, a defined time window of the device log data for a periodic trigger, the contents of a last firmware update or the last good page for successful operation, etc. In some embodiments, the messaging service 566 may implement a shared message formatter 566.4 that formats a peer message according to the control bus protocol 548 such that that the control bus address of the target's peer storage device and the selected content are included in the payload. For example, the common message formatter 566.4 may create or designate a header containing a control bus address and a read/write identifier, followed by the message payload and/or other parameters such as command codes and/or error codes to specify the selected internal operational data to transmit.

In some embodiments, message service 566 may include a peer message monitor 566.5 configured to monitor control bus interface 518 for messages from peer storage devices, addressed to storage device 500 (directly or via broadcast). For example, peer message monitor 566.5 may receive peer messages initiated by peer storage devices and/or respond to messages sent by message service 566 over the control bus. Peer message monitor 566.5 may analyze received messages to determine whether further action is required, such as passing peer data to peer storage manager 566.6 or passing restore requests or restore data to restore service 568. In some embodiments, the Message service 566 may include a peer storage manager 566.6 arranged to store peer data received from peer storage devices for offloading or backup. For example, peer storage manager 566.6 may receive internal operational data from one or more peer storage devices and store the data in peer data 520.2. In some embodiments, peer storage manager 566.6 may maintain a lookup table or similar data structure for locating peer data by storage device identifier, internal operational data type, and other parameters to support selective retrieval of peer data on demand. In some embodiments, peer storage manager 566.6 may also maintain a reference table or similar data structure that indicates which peer storage devices received the offloaded internal operational data 550 from storage device 500.

Recovery service 568 may include data structures, functions, and interfaces for recovering internal recovery mode data from peer devices over the control bus for use in a recovery operation. For example, the recovery service 568 may be initiated during startup or power cycle after an improper shutdown to request lost, corrupted, or untrusted internal operational data from a peer storage device to which it was previously offloaded.

In some embodiments, recovery service 568 may include recovery events 568.1 configured to determine when previously offloaded data should be recovered and used by storage device 500. For example, the recovery events 568.1 may include a rule-based set of event conditions or parameters that trigger search and retrieval of paged internal operational data, such as starting after an improper shutdown, power failure, operational mode change, or selected fault condition. In some embodiments, storage device 500 may also initiate peer recovery functions upon installation without a reset event. For example, storage device 500 may be equipped with a plug and play feature that checks for firmware updates from peer storage devices and restores the last compatible firmware update. In some embodiments, the recovery service 568 may include a peer recovery selector 568.2 configured to determine the peer storage device containing the target recovery data (or previously swapped internal recovery operational data). For example, peer selector 568.2 may use peer storage manager 566.6 to identify the peer storage device or devices to which the desired recovery data has been offloaded, if known. In some embodiments, peer storage device 500 may not know which storage device contains the desired restore data, and peer selector 568.2 may be configured to use restore requester 568.3 to query the peer storage devices and determine whether one of contains the recovery data you want. For example, restore requester 568.3 may send restore requests to a plurality of peer storage devices, to which each peer storage device containing the restore data may respond with the restore data. In some embodiments, messaging service 566 (or restoration requestor 568.3) may send an initial query request. The messages that are then received may indicate whether and to what extent each peer storage device contains some or all of the data, and peer selector 568.2 may use the responses to determine which peer storage devices should receive restore requests. In some embodiments, the restore service 568 may include a restore requester 568.3 configured to send a restore request message over the control bus to one or more peer storage devices. For example, restore requester 568.3 may use message service 566 to send a restore message to a peer storage device containing some or all of the desired restore data. the Recovery request may identify the specific set of internal operational data desired for recovery operations in the data payload and/or include an opcode or process call configured to request the target recovery data. In some embodiments, the recovery service 568 may include a recovery manager 568.4 configured to receive recovery data from peer storage devices and store the received recovery data in storage 514 and/or device data 520.1 for use in completing the recovery operation . For example, the recovery data received from messaging service 566 may be forwarded to recovery manager 568.4, and recovery manager 568.4 may designate one or more storage locations in device data 520.1 for the recovered internal operational data. Once the recovered internal recovery operational data has been returned to device data 520.1, it may be used by storage device 500 to complete recovery operations, such as pre-host operations 540 or other recovery operations from an improper shutdown or operational error.

As in 6 1, storage device 500 may be operated in accordance with an exemplary method for sending messages between peer storage devices using a control bus, ie, method 600 defined by blocks 610-624 in 6 is illustrated.

At block 610, a device identifier may be sent to peer devices over a control bus. For example, a discovery service conforming to the control bus protocol may send a broadcast message identifying the storage device and its control bus address.

At block 612, device identifiers may be received from peer devices over the control bus. For example, the discovery service may also receive broadcast messages identifying other storage devices on the same control bus, along with their control bus addresses. These peer discovery messages may be received in response to the storage device broadcast in block 610, as part of the startup or initialization of each peer storage device, and/or based on a master discovery trigger or broadcast.

At block 614, device identifiers from the storage devices may be stored in a peer device register of the storage device. For example, the discovery service may store device identifiers, control bus addresses, and/or other information about each peer storage device in a peer storage device entry in the peer device register.

At block 616, peer communication may be established between the peer storage devices. For example, the peer device register can contain the necessary information to address messages to storage devices, and a peer messaging service can be set up to send broadcast and/or direct messages to the peer storage device via the control bus protocol.

At block 618, internal operational data may be selected for unloading by the storage device. For example, the messaging service may select a set of internal operational data based on one or more conditions or events that trigger sharing.

At block 620, a peer destination device for the offload message may be determined. For example, the messaging service may select one or more storage devices from the peer device register to receive the selected internal operational data.

At block 622, a peer message may be formatted to conform to the control bus protocol. For example, the messaging service may package the selected internal operational data as a data payload in one or more messages with appropriate address headers for the destination peer storage device.

At block 624, the peer message may be sent from the storage device that created the internal operational data to a peer storage device over the control bus. For example, the messaging service may send the peer message created in block 622 over a control bus interface.

As in 7 1, storage device 500 may be operated according to an example method for retrieving internal operational data from a peer storage device, ie, method 700, defined by blocks 710-720 in FIG 7 is illustrated.

At block 710, a recovery event may be determined. For example, a recovery service of a storage device that previously offloaded internal operational data to other storage devices may detect a recovery event, such as starting up after an unanticipated shutdown, power cycle, or operational mode change.

At block 712, a peer device may be determined. For example, the recovery service may determine peer storage devices that received the desired recovery data and select one or more corresponding peer storage devices to receive a recovery request.

At block 714, the restore request may be sent to one or more peer storage devices. For example, the recovery service may format and send a recovery request specifying the internal operational data needed for the recovery.

At block 716, the internal recovery mode data may be received. For example, the restore service may receive one or more response messages from peer storage devices that receive the restore requests. The response messages may include some or all of the internal recovery operation data requested in the recovery requests.

At block 718, the internal recovery mode data may be saved. For example, the recovery service may store the internal recovery operational data recovered from the peer storage devices in a device data store for the internal operational data of the storage device, such as at the location from which the internal operational data was missing, corrupted, or unreliable.

At block 720, an operation may be completed using the internal recovery operational data. For example, the storage device may perform one or more internal processes, such as ROM, bootloader, or firmware processes for startup, debugging, operating mode change, etc., using the internal operating data recovered from the peer storage devices.

As in 8th 11, storage device 500 may be operated in accordance with an example method for supporting internal recovery operational data, ie, method 800 defined by blocks 810-820 in FIG 8th is illustrated.

At block 810, the internal peer operational data may be received from a peer storage device. For example, a messaging service may receive a peer message that forwards internal operational data to the recipient storage device.

At block 812, the internal peer operational data may be stored by the receiver storage device. For example, the storage device may include a memory space dedicated to storage of internal peer operational data, and store the received internal peer operational data with metadata sufficient to identify, retrieve, and communicate with the stored internal peer operational data -Associate storage device from which it originated.

At block 814, a restore request may be received. For example, the messaging service may receive a peer message and identify it as a recovery request that includes parameters describing internal operational data needed for recovery.

At block 816, the internal recovery mode data may be determined. For example, a peer storage manager can use the restore request parameters to determine the amount of requested internal operational data in the peer data.

At block 818, the internal recovery operation data may be retrieved. For example, the peer storage manager can retrieve the portion of the requested internal recovery operation data that is stored in the storage devices in the peer data.

At block 820, the internal peer operational data may be sent. For example, the messaging service may send one or more response messages that include the internal recovery operation data retrieved in block 818 to the peer storage device that sent the recovery request.

While at least one exemplary embodiment has been presented in the foregoing detailed description of the technology is, it should be appreciated that a large number of variations may exist. It should also be noted that an exemplary embodiment or exemplary embodiments are examples, and are not intended in any way to limit the scope, applicability, or configuration of the technology. Rather, the foregoing detailed description will provide useful guidance for those skilled in the art to implement an example embodiment of the technology, it being understood that various modifications may be made to a function and/or arrangement of elements described in an example embodiment without to depart from the scope of the technology as set forth in the appended claims and their legal equivalents.

As known to one of ordinary skill in the art, various embodiments of the present invention may be embodied as a system, method, or computer program product. Accordingly, embodiments of the present disclosure may take the form of an entirely hardware implementation, an entirely software implementation (including firmware, resident software, microcode, or the like), or embodiments in the form of a combination of hardware and software, all generically herein referred to as circuitry , module, system and/or network. Furthermore, various embodiments of the present technology may take the form of a computer program product embodied in one or more computer-readable media, including computer-readable program code embodied thereon.

Any combination of one or more computer-readable storage media may be used. A computer-readable medium can be a computer-readable signal medium or a physical or computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, etc., or any suitable combination of the foregoing. Non-limiting examples of a physical computer-readable storage medium may include an electrical connection, including one or more wires, a portable computer disk, a hard drive, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory ( EEPROM), flash memory, optical fiber, compact disk read only memory (CD-ROM), optical processor, magnetic processor, etc., or any suitable combination of the above, but not limited thereto. In the context of this document, a computer readable and/or executable storage medium can be any tangible medium that can contain or store a program or data for use by or in connection with an instruction execution system, apparatus and/or apparatus .

Computer code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, radio frequency (RF), etc., or any suitable combination of the foregoing. Computer code for executing processes in embodiments of the present invention may be written in any static language, such as the C programming language or other similar programming languages. The computer code may execute entirely on a user's computing device, partially on a user's computing device, as a stand-alone software package, partially on a user's computing device and partially on a remote computing device, or entirely on the remote device or a server. In the latter mindset, a remote computing device can be connected to a user's computing device through any type of network or communication system, including but not limited to a local area network (LAN) or a wide area network (WAN), network convergence , or the connection can be made to an external computer (e.g. through the Internet using an Internet provider).

Various embodiments of the present invention may be described above with reference to flowchart diagrams and/or block diagrams of methods, apparatus, systems, and computer program products. It is understood that each block of the schematic flowchart and/or schematic block diagram, and combinations of blocks in the schematic flowchart and/or schematic block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit (processor) of a general purpose computer, special purpose computer, or other programmable data processing equipment to produce a machine, so that the instructions, which can be processed via the processing device or other programmable Data processing equipment can be executed, generate means for implementing the processes/acts in a flowchart and/or block(s) of a block diagram.

Some computer program instructions may also be stored on a computer-readable medium capable of directing a computer, other programmable computing device, or other device(s) to operate in a particular manner, so that the finished articles stored on a computer-readable medium will include instructions that implement the operation/process specified in a flowchart and/or block(s) of a block diagram. Some computer program instructions may also be loaded onto a computing device, other programmable computing device, or other device(s) to cause a series of operational steps to be performed on the computing device, other programmable device, or other device(s). such that the instructions, executed by the computer or other programmable device, provide one or more processes for implementing the operation/act specified in a flowchart and/or block(s) of a block diagram.

A flowchart and/or block diagram in the above figures may illustrate construction, functionality, and/or operation of possible implementations of apparatus, systems, methods, and/or computer program products according to various aspects of the present technology. In this regard, a block in a flowchart or block diagram may represent a module, segment, or portion of code, which may comprise one or more executable instructions for implementing one or more specified logical functions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown sequentially may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that a block of a block diagram and/or a flowchart representation, or a combination of blocks in a block diagram and/or a flowchart representation, may be implemented by dedicated hardware-based systems that perform one or more specified operations or actions, or by combinations of special purpose hardware and computer instructions.

While one or more aspects of the present technology have been illustrated and discussed in detail, one skilled in the art will recognize that modifications and/or adaptations can be made to the various aspects without departing from the scope of the present invention as set forth in the following claims.

Claims (20)

system comprising: a first device comprising: a processor; a memory; a host interface configured to connect to a host system; a control bus interface arranged to be connected to a control bus, wherein a plurality of peer devices arranged to be connected to the control bus; and a peer sharing service set up to: establish peer communication with a target peer device of the plurality of peer devices via the control bus interface; select internal operating data for the first device; send the selected internal operational data to the target peer device; and receive the selected internal operational data from the target peer device in response to a recovery event. system according to claim 1 wherein the first device further comprises: a storage medium configured to store host data received via the host interface; a memory interface protocol configured to communicate with the host system over the host interface; and a non-volatile memory device controller configured to manage input/output of data to the storage medium. system according to claim 2 wherein: the storage medium comprises at least one non-volatile storage device; the memory interface protocol is a non-volatile memory express protocol (NVMe); the plurality of peer devices includes a plurality of peer storage devices; and each peer storage device of the plurality of peer storage devices comprises: a peer host interface configured to connect to the host system using the NVMe protocol; a peer storage medium comprising at least one non-volatile memory; and a peer control bus interface configured to connect to the control bus. system according to claim 1 wherein the peer sharing service comprises a discovery service configured to: transmit over the control bus interface a device identifier for the first device; receive device identifiers from the plurality of peer devices via the control bus interface; and store the peer device identifier in a peer device register. system according to claim 1 wherein: the peer sharing service is further configured to: determine a peer device identifier for the target peer device; and format a peer message including the selected internal operational data as a message payload; and sending the selected internal operational data includes sending the peer message to the target peer device using the peer device identifier for addressing. system according to claim 5 wherein the peer sharing service comprises an encryption service arranged to encrypt the selected internal operational data prior to sending the peer message. system according to claim 1 , wherein the internal operational data is selected from: reset event data; debug data; log data related to the operation of the device; or last good configuration data. system according to claim 1 wherein the peer enabling service is further configured to: receive, via the control bus interface, internal operational data from at least one peer device of the plurality of peer devices; store the internal peer operational data in a peer data partition in memory; and send the internal peer operational data to the at least one peer device in response to a peer recovery event. system according to claim 1 further comprising a plurality of peer devices, wherein: the plurality of peer devices store the internal operational data received from the first device; and the peer sharing service further comprises a recovery service configured to: determine the recovery event; in response to the recovery event and via the control bus interface, send a recovery request to at least one peer device of the plurality of peer devices; receive internal recovery mode data from the at least one peer device; and save the internal recovery operation data to an internal operation storage location. system according to claim 9 , wherein: the recovery event occurs in a pre-host operational state; the internal recovery mode data includes a last good configuration data page; and the first device is further arranged to complete pre-host operations using the last good configuration data page received from the at least one peer device. Computer-implemented method, comprising: establishing a peer communication from a first device to a target peer device of a plurality of peer devices, wherein the first device and each peer device of the plurality of peer devices includes: a host interface configured to connect to a host system; and a control bus interface configured to connect to a control bus; selecting internal operating data for the first device; sending the selected internal operational data from the first device and over the control bus interface to the target peer device; and receiving the selected internal operational data from the target peer device at the first device and over the control bus interface, the receiving the selected internal operational data being in response to a recovery event. Computer-implemented method according to claim 11 wherein the first device and each peer device of the plurality of peer devices includes: a storage medium configured to store host data received via the host interface; a memory interface protocol configured to communicate with the host system over the host interface; and a non-volatile memory device controller configured to manage input/output of data to the storage medium. Computer-implemented method according to claim 11 and further comprising: sending a device identifier for the first device over the control bus interface; receiving peer device identifiers from a plurality of peer devices over the control bus interface; and storing the peer device identifiers in the first device in a peer device register. Computer-implemented method according to claim 11 , further comprising: determining, by the first device, a peer device identifier for the target peer device; and formatting a peer message including the selected internal operational data as a message payload, wherein sending the selected internal operational data includes sending the peer message to the target peer device using the peer device identifier for addressing. Computer-implemented method according to Claim 14 , further comprising: encrypting the selected internal operational data before sending the peer message. Computer-implemented method according to claim 11 , wherein the internal operational data is selected from: reset event data; debug data; log data related to the operation of the device; or last good configuration data. Computer-implemented method according to claim 11 , further comprising: receiving internal operational data from at least one peer device of the plurality of peer devices at the first device via the control bus interface; storing the peer's internal operational data in a memory of the first device in a peer data partition in the memory; and sending, by the first device and in response to a peer recovery event, the peer's internal operational data to the at least one peer device. Computer-implemented method according to claim 11 , further comprising: determining, by the first device, the recovery event; sending a recovery request over the control bus interface to at least one peer device of the plurality of peer devices in response to the recovery event; receiving internal recovery mode data from at least one peer device and over the control bus interface; and storing internal recovery operational data in an internal operational storage location. Computer-implemented method according to claim 11 , further comprising: the first device completing pre-host operations using a last good configuration data page received from the peer target device, wherein: the recovery event occurs in a pre-host operational state; and the selected internal operational data includes the last good configuration data page. Storage system comprising: a control bus; and a plurality of peer storage devices, each peer storage device of the plurality of peer storage devices comprising: a processor; a memory; a host interface configured to connect to a host system; a control bus interface configured to connect to the control bus; means for establishing peer communication between the plurality of storage devices via the control bus interface; means for selecting internal operational data from an internal operational storage location; means for sending the selected internal operational data to at least one other peer storage device of the plurality of peer storage devices; and means for receiving, in response to a recovery event, the selected internal operational data from at least one peer storage device of the plurality of peer storage devices.

Images