JP2022039501A - Storage control device, transmission state determination program and storage system - Google Patents

Abstract

To make it possible to determine a transmission state of data.SOLUTION: A CM#0 in charge is configured to designate a SQ-ID, and transmit an execution request of a read command requesting a NVMe SSD$0 to transfer read data to a memory #0 to a DCM#1. The DCM#1 is configured to, in response to the execution request, add a transfer request of prescribed data (data by one block) to a Doorbell register corresponding to a SQ-ID in an FPGA#0 to the read command, and transmit the transfer request thereof to a NVMe SSD$0. The FPGA#0 is configured to issue an interruption designating a MSI-X ID related to a Doorbell ID of the Doorbell register having writing completed to a CPU#0. The CPU#0 is configured to, when receiving a transfer completion reply designating the SQ-ID from the DCM#1, determine a transmission state of the read data with reference to an interruption flag corresponding to the designated SQ-ID in an ID corresponding table.SELECTED DRAWING: Figure 7

Description

The present invention relates to a storage control device, a delivery status determination program, and a storage system.

In recent years, a storage system that employs NVMe (Non-Voltile Memory Express) SSD (Solid State Drive) as a storage device has been developed. NVMe is a communication protocol for optimizing communication in flash storage using non-volatile memory.

Prior art, in a storage device, a controller creates a queue group containing multiple command queues with different priorities in its own controller or in a storage device, and among the commands to the storage device, it is faster. Some commands that need to be processed are posted to a higher priority command queue. Further, there is a technique in which a DMA (Direct Memory Access) controller reads the transfer status of data transferred to another CM (Controller Model) via a switch from the switch, and writes the read transfer status to a memory.

International Publication No. 2017/158799 Japanese Unexamined Patent Publication No. 2015-18314

However, in the prior art, when accessing a storage device such as an NVMe SSD via another storage control device, there is a problem that the delivery guarantee of the data transferred from the storage device cannot be ensured.

In one aspect, it is an object of the present invention to make it possible to determine the delivery status of data.

In one embodiment, an internal circuit can be reconfigured that includes a buffer area for data transfer and issues an interrupt with an identifier corresponding to the buffer area when a write is made to the buffer area. When accessing a storage device that has an integrated circuit or a dedicated integrated circuit and performs data transfer that does not require a response via another storage control device, the transfer of read data to its own memory is requested and the integration is performed. A read command execution request requesting the transfer of a predetermined data to the buffer area corresponding to the specified first identifier in the circuit is transmitted to the other storage control device, and the first identifier is transmitted from the integrated circuit. When the interrupt for which is specified is received, the interrupt flag corresponding to the first identifier is enabled, and the execution of the read command issued from the other storage control device to the storage device in response to the execution request is completed. When the completion response with the specified first identifier is received from the other storage control device and the completion response is received, the read data is referred to with reference to the interrupt flag corresponding to the first identifier. A storage control device having a control unit for determining the delivery status of the data is provided.

According to one aspect of the present invention, there is an effect that the delivery status of data can be discriminated.

FIG. 1 is an explanatory diagram showing an example of a system configuration of a storage system according to an embodiment. FIG. 2 is an explanatory diagram showing an embodiment of the storage system 100. FIG. 3 is an explanatory diagram showing an example of a data structure of an NVMe command. FIG. 4 is an explanatory diagram showing an example of the stored contents of the ID correspondence table. FIG. 5 is a block diagram showing a functional configuration example of the CM # i in charge. FIG. 6 is a block diagram showing a functional configuration example of DCM # i. FIG. 7 is an explanatory diagram showing an operation example of the storage system 100. FIG. 8 is a sequence diagram showing an example of a normal processing procedure of the storage system 100. FIG. 9 is an explanatory diagram (No. 1) showing an example of updating the ID correspondence table 400. FIG. 10 is a sequence diagram showing an example of a processing procedure when an abnormality occurs in the storage system 100. FIG. 11 is an explanatory diagram (No. 2) showing an example of updating the ID correspondence table 400. FIG. 12 is an explanatory diagram (No. 1) showing another embodiment of the storage system 100. FIG. 13 is an explanatory diagram (No. 2) showing another embodiment of the storage system 100.

Hereinafter, embodiments of the storage control device, the delivery status determination program, and the storage system according to the present invention will be described in detail with reference to the drawings.

(Embodiment)
FIG. 1 is an explanatory diagram showing an example of a system configuration of a storage system according to an embodiment. In FIG. 1, the storage system 100 includes CM # 0, CM # 1, and a storage device $ 0. CM # 0, # 1 and the storage device $ 0 are provided in CE (Control Rule Enclosure) $ 0. The CE is a case (box) for accommodating a commercial or the like.

CM # 0 and # 1 are examples of storage control devices that control access to the storage device $ 0 under their own control. CMs # 0 and # 1 in the same CE $ 0 are directly connected to each other so as to be able to communicate with each other. CE $ 0 is realized as, for example, one storage device.

The storage device $ 0 is a storage device that transfers data without requesting a response. For example, the storage device $ 0 is an NVMe SSD. In NVMe, PCIe (Peripheral Component Interconnect-Express) is used as the SSD connection standard. In PCIe, since it is a posted write, write transfer that does not require a response is performed.

In the following description, "NVMe SSD $ 0" will be described as an example of the storage device $ 0. NVMe SSD $ 0 is an SSD that uses the NVMe communication protocol. However, as the storage device $ 0, another storage device that transfers data without requesting a response may be used.

The host device 110 is connected to CM # 0 and # 1. The host device 110 is, for example, a server that executes a business application or the like. CM # 0, # 1 and the host device 110 are connected via a SAN (Storage Area Network) using, for example, FC (Fibre Channel) or iSCSI (Internet Small Computer System Interface).

In the following description, any CM among CM # 0 and # 1 in the storage system 100 may be referred to as “CM # i” (i = 0, 1). Further, the CM # i that has received the request (I / O request) from the host device 110 may be referred to as "the CM in charge".

The CM # i includes, for example, a CPU (Central Processing Unit) # i, a memory # i, a CA (Channel Adapter) # i, an FPGA (Field Programmable Gate Array) # i, and a PCIe SW (switch) # i. ,including. Each component is connected by a bus.

CPU # i controls the entire CM # i. CPU #i may have a plurality of cores. Memory #i stores data and programs. The memory #i includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash ROM, and the like. Specifically, for example, a flash ROM or ROM stores various programs, and RAM is used as a work area of CPU # i. The program stored in the memory # i is loaded into the CPU # i to cause the CPU # i to execute the coded process. Memory #i may also be used as a cache memory for temporarily storing data and programs.

CA # i is an interface for communicating with the host device 110. CA # i has, for example, an adapter compliant with LAN, SAN, FC and the like. FPGA # i is a programmable logic device, which is an integrated circuit in which an internal circuit can be reconfigured. The PCIe SW # i is an interface for communicating with other devices (CM, storage device). CM # i may have, for example, an IOC (Input / Output Controller) that controls access to NVMe SSD $ 0.

The number of CEs included in the storage system 100 is not limited to one, and the number of CMs included in the CE is not limited to two. For example, the storage system 100 may include a plurality of CEs (see, for example, FIG. 12 described later).

Further, in FIG. 1, the case where one host device 110 is connected to CM # 0 and # 1 has been described as an example, but the present invention is not limited to this. For example, each of the plurality of host devices may be connected to one or more of CMs # 0 and # 1.

Further, in the example of FIG. 1, CM # i is determined to have FPGA # i, but the present invention is not limited to this. For example, the processing performed by FPGA # i in CM # i may be realized by a dedicated integrated circuit (for example, LSI: Large Scale Integration). That is, the CM # i may have a dedicated integrated circuit having the same function as the FPGA # i instead of the FPGA # i.

Here, in the case of a SAS (Serial Attached SCSI) SSD, the CM in charge can access the SSD in both the own EXP and the other EXP because there is a path from the own IOC to the other EXP. The EXP (expander) is a SAS EXP, which is a relay device for connecting a controller and a plurality of SAS disk drives.

On the other hand, when using NVMe SSD as a storage device instead of SAS SSD, if SAS EXP is replaced with PCIe SW, there will be no path from the own IOC to another EXP, so the CM in charge is only one SSD. I can't access it. Therefore, for example, in order to exchange data with an NVMe SSD under another CM, it is accessed via another CM. At this time, for example, if the access is made via the buffer of another CM, the response is slow, so it is desirable to directly exchange data (read data) between the CM in charge and the SSD.

However, in a conventional storage system using an NVMe SSD as a storage device, there is a problem that data delivery cannot be guaranteed when accessing an NVMe SSD in a different PCIe domain. For example, assume a case where a failure occurs between the PCIe SW of the CM in charge and the NVMe SSD.

In this case, the CM in charge requests another CM to access the NVMe SSD. The other CM is a CM that can access the SSD in which the data that the responsible CM wants to read is stored. In the following description, a CM that can access the SSD in which the data to be read by the CM in charge is stored may be referred to as "DCM (Drive CM)".

DCM issues a read command to NVMe SSD in response to a request from the CM in charge. The NVMe SSD transfers the read data to the memory of the CM in charge by DMA in response to the read command from the DCM. At this time, due to the posted write of PCIe, the delivery of the read data transferred to the memory of the CM in charge cannot be guaranteed.

That is, even if a failure such as a temporary disconnection of the link between the PCIe SWs occurs, it is not possible to determine whether or not the read data has been normally transferred from the NVMe SSD to the memory of the CM in charge. If data delivery guarantee cannot be ensured, data omission cannot be detected and data may be garbled in the host device.

Specifically, for example, the NVMe SSD issues an MSI (Message Signaled Interrupts) -X interrupt to the DCM when the transfer processing of the read data to the memory of the CM in charge is completed. When the DCM determines that the read data transfer process is completed by the MSI-X interrupt from the NVMe SSD, the DCM sends a command completion response to the CM in charge.

However, even if the completion response is transmitted from the DCM to the CM in charge, the CM in charge cannot determine whether or not the data transfer has been performed normally. That is, since there is no response from the memory of the CM in charge due to the data transfer (posted write), the CM in charge cannot recognize when the data is not transferred normally. Even in this case, the completion response is transmitted from the DCM to the CM in charge, and data garbled occurs in the host device.

Therefore, in the present embodiment, even when CM # i (CM in charge) accesses NVMe SSD $ k via another CM # j (DCM), it is possible to secure the guarantee of data delivery. The storage system 100 that can be described will be described. Specifically, for example, CM # i (CM in charge) includes a buffer area for data transfer, and when writing to the buffer area is performed, an interrupt is issued by designating an identifier corresponding to the buffer area. FPGA # i is provided. Then, CM # i (CM in charge) changes from the write status to the buffer area in FPGA # i, which is performed using the same route as the data transfer from NVMe SSD $ k to memory # i, to memory # i. Determine the delivery status of the read data of.

FIG. 2 is an explanatory diagram showing an embodiment of the storage system 100. Here, CM # 0 is the responsible CM, and CM # 1 is the DCM. Further, it is assumed that a failure occurs between PCIe SW # 0 of the CM # 0 in charge and NVMe SSD $ 0. In this case, CM # 0 in charge accesses NVMe SSD $ 0 via DCM # 1. Note that FIG. 2 shows an excerpt of a part of the hardware configuration of each CM # 0 and # 1.

The CM # 0 in charge provides the FPGA # 0 with a Doorbell register (DBL0, 1, ..., N) corresponding to the SQ ID. The Doorbell register (DBR) is an example of a buffer area for data transfer. The SQ ID is an identifier that uniquely identifies the SQ (Submission Queue) described later. The responsible CM # 0 maps the Doorbell register area of FPGA # 0 to the memory # 0. For example, Doorbell0 in memory # 0 corresponds to DBL0 in FPGA # 0.

Further, the CM # 0 in charge provides MSI-X0, 1, ..., N for setting an interrupt from the FPGA # 0 in the memory # 0. When the Doorbell register is written, FPGA # 0 issues an MSI-X interrupt corresponding to the ID of the Doorbell register to CPU # 0 of CM # 0.

DCM # 1 creates N pairs of SQ and CQ (Completion Queue) in memory # 1. N can be arbitrarily set, and for example, a value corresponding to the number of cores of CPU # 1 is set. SQ and CQ are used to access NVMe SSD $ 0 from CPU # 1 of DCM # 1 for each NVMe command.

SQ is a queue for storing commands. The CQ is a queue that stores information indicating the completion status of the command. Specifically, for example, DCM # 1 creates a pair of SQ and CQ in memory # 1 for each SQ ID (0, 1, ..., N). For example, "SQ 0" and "CQ 0" in the memory # 1 indicate a pair of SQ and CQ corresponding to the SQ ID "0".

Here, an example of the data structure of the NVMe command will be described.

FIG. 3 is an explanatory diagram showing an example of a data structure of an NVMe command. In FIG. 3, the NVMe command cmd includes Opcode, data address 1, data address 2, Starting LBA (Logical Block Addressing) and Number of Logical Blocks.

Here, Opcode indicates the type of command. For example, in the case of a read command, Opcode is Read. The data address 1 indicates a transfer destination address of a data area in the memory of the CM in charge. The data address 2 indicates the address of the Doorbell register in the FPGA of the CM in charge.

The Starting LBA indicates the first address of the block to read from the NVMe SSD. Number of Logical Blocks indicates the number of blocks to be read from the NVMe SSD. One block is, for example, 512 [byte]. For example, the number of blocks of data to be transferred to the data area in the memory of the CM in charge is L block.

In this case, the Number of Logical Blocks is, for example, a (L + α) block in order to add data transfer to the Doorbell register. The α block is allocated to the transfer to the Doorbell register. α can be arbitrarily set, and is set to 1, for example.

In the storage system 100, the responsible CM # 0 associates the SQ and CQ of DCM # 1, the Doorbell register of FPGA # 0, and the ID of MSI-X by using the ID correspondence table 400 as shown in FIG. The ID correspondence table 400 is stored in, for example, the memory # 0 of the CM # 0 in charge.

FIG. 4 is an explanatory diagram showing an example of the stored contents of the ID correspondence table. In FIG. 4, the ID correspondence table 400 has fields for command ID, SQ ID, CQ ID, Doorbell ID, MSI-X ID, and interrupt flag. By setting information in each field, ID correspondence information (for example, ID correspondence information 400-1) is stored as a record.

The command ID is an identifier that uniquely identifies the command. The SQ ID is an identifier that uniquely identifies the SQ in the memory # 1 of the DCM # 1. The CQ ID is an identifier that uniquely identifies the CQ in the memory # 1 of the DCM # 1. The Doorbell ID is an identifier that uniquely identifies the Doorbell register in FPGA # 0 of the CM # 0 in charge.

The MSI-X ID is an identifier that uniquely identifies the MSI-X interrupt from FPGA # 0. Each ID is a value from 0 to N. N corresponds to the number of pairs of SQ and CQ created in the memory of the DCM. The interrupt flag indicates the MSI-X interrupt state from FPGA # 0. The interrupt flag "0" indicates a state (invalid) in which an interrupt has not occurred. The interrupt flag "1" indicates a state (valid) in which an interrupt has occurred. In the initial state, the interrupt flag is "0".

In the storage system 100, the same ID is used for the SQ, CQ, Doorbell register, and MSI-X interrupt in a series of flows related to each read command. As a result, CPU # 0 of the CM # 0 in charge can determine which SQ ID command is completed (delivery status) by confirming the ID of the MSI-X interrupt received from FPGA # 0.

(Example of functional configuration of CM # i)
Next, an example of a functional configuration of CM # i will be described. First, with reference to FIG. 5, a functional configuration example in which CM # i operates as the CM in charge will be described. Here, the DCM when CM # i is the responsible CM may be expressed as "DCM # j" (j ≠ i). For example, when CM # 0 is the responsible CM (i = 0), CM # 1 is DCM (j = 1). Further, the NVMe SSD of the access destination having the read data may be expressed as "NVMe SSD $ k".

FIG. 5 is a block diagram showing a functional configuration example of the CM # i in charge. In FIG. 5, the CM # i in charge includes a first reception unit 501, a first request processing unit 502, a determination unit 503, and a restoration unit 504. The first reception unit 501 to the recovery unit 504 are functions that serve as control units. Specifically, for example, by causing the CPU #i to execute a program stored in the memory #i, or by using CA # i, The function is realized by PCIe SW # i. The processing result of each functional unit is stored in, for example, the memory #i.

The first reception unit 501 receives a request from the host device 110. Here, the request is, for example, a read request (read request) for NVMe SSD $ k. The request includes, for example, access destination information and data size of read data. Specifically, for example, the first reception unit 501 receives a read request from the host device 110 via CA # i.

When the first request processing unit 502 accesses the NVMe SSD $ k via DCM # j (another storage control device), the first request processing unit 502 transfers the read data to the own memory #i to the NVMe SSD $ k. Is sent to DCM # j by designating the identifier (first identifier) corresponding to the read command.

Here, when accessing NVMe SSD $ k via DCM # j, for example, there is a case where a failure occurs between PCIe SW # i and NVMe SSD $ k of the CM # i in charge. Further, when accessing NVMe SSD $ k via DCM # j, it is a case where NVMe SSD $ k is located in a CE different from the CM # i in charge.

The read command is, for example, the NVMe command cmd shown in FIG. The read data is data for L blocks read from the Starting LBA of NVMe SSD $ k, and is transferred to the data area in the memory #i of the CM # i in charge.

The first identifier corresponding to the read command is, for example, the SQ ID assigned to the read command. For example, among the SQ IDs from 0 to N, an empty ID with a smaller value is assigned to each read command. The empty ID is an ID that does not correspond to another read command, for example, an ID that is not registered in the ID correspondence table 400 shown in FIG.

When a read command execution request with the first identifier specified is sent to DCM # j, DCM # j sends NVMe SSD $ k to the buffer area corresponding to the first identifier in FPGA # i. A read command is issued with a request to transfer the specified data added. The buffer area corresponding to the first identifier is, for example, the Doorbell register corresponding to the SQ ID.

That is, the execution request of the read command in which the first identifier is specified requests the transfer of the read data to the own memory #i, and is predetermined to the buffer area corresponding to the first identifier in the own FPGA # i. Corresponds to the execution request of the read command that requests the transfer of data.

The predetermined data can be arbitrarily set, and is, for example, data having a smaller amount of data than the read data requested from the host device 110. The predetermined data may be one block of data following the read data. Further, the predetermined data is requested to be transferred to the buffer area corresponding to the first identifier in the FPGA # i, for example, after the transfer of the read data from the NVMe SSD $ k to the memory #i is completed. To. The specific processing content on the DCM side will be described later with reference to FIG.

Specifically, for example, when the first request processing unit 502 accesses NVMe SSD $ k via DCM # j in response to a read request from the host device 110, refer to the ID correspondence table 400. Then, specify the SQ ID to be assigned to the read command. For example, when the ID correspondence information is not registered in the ID correspondence table 400, SQ0 (ID: 0) is specified.

The first request processing unit 502 creates ID correspondence information (for example, ID correspondence information 400-1 shown in FIG. 4) corresponding to the specified SQ0 and registers it in the ID correspondence table 400. Then, the first request processing unit 502 specifies the specified SQ0 (ID: 0) and transmits a read command execution request to the NVMe SSD $ k to DCM # j. The access destination information and the data size of the read data are specified from the read request.

Further, when the first request processing unit 502 receives an interrupt with the first identifier specified, which is issued from FPGA # i, the first request processing unit 502 enables the interrupt flag corresponding to the first identifier. The interrupt specifying the first identifier is issued from FPGA # i when the buffer area corresponding to the first identifier provided in FPGA # i is written.

Specifically, for example, FPGA # i issues an MSI-X interrupt specifying Doorbell 0 (ID: 0) to CPU # 0 when the Doorbell 0 (ID: 0) is written to the Doorbell register. do. The MSI-X interrupt with Doorbell0 (ID: 0) is the MSI-X interrupt with MSI-X0 (ID: 0) associated with Doorbell0 (ID: 0). In this case, the first request processing unit 502 sets the interrupt flag of the ID correspondence information 400-1 corresponding to MSI-X0 (ID: 0) to "1" with reference to the ID correspondence table 400.

When the execution of the read command issued from DCM # j to NVMe SSD $ k in response to the execution request transmitted by the first request processing unit 502 is completed, the determination unit 503 specifies the first identifier. Accept the response from DCM # j.

The read command issued is a read command that requests transfer of read data to memory # i and also requests transfer of predetermined data to the buffer area corresponding to the first identifier in FPGA # i. A request for transferring predetermined data to the buffer area corresponding to the first identifier in FPGA # i is added, for example, by operating a read command in DCM # j requesting transfer of read data to memory # i. Will be done.

Further, when the determination unit 503 receives the completion response in which the first identifier is specified, the determination unit 503 determines the delivery status of the read data with reference to the interrupt flag corresponding to the first identifier. Specifically, for example, when the determination unit 503 receives the completion response with the SQ ID specified from DCM # j, the determination unit 503 refers to the ID correspondence table 400 and the value of the interrupt flag corresponding to the specified SQ ID. To confirm.

Here, when the interrupt flag is "1 (valid)", the determination unit 503 determines that the read data has been delivered. That is, the determination unit 503 determines that the read data has been normally transferred to the memory #i. The read data is, for example, data for L blocks requested by the host device 110.

On the other hand, when the interrupt flag is "0 (invalid)", the determination unit 503 determines that the read data has not been delivered. That is, the determination unit 503 determines that the read data has not been normally transferred to the memory #i. The determination unit 503 determines that the read data has not been delivered even when the error response with the SQ ID specified is received from DCM # j, for example.

When it is determined that the read data has not been delivered, the recovery unit 504 executes a predetermined recovery process. The predetermined recovery process is, for example, a process of reconstructing a route (link) between the memory #i and the NVMe SSD $ k. That is, a temporary hardware failure may occur and the link may be unstable, or the communication speed may not be linked at a specified speed. Therefore, the recovery process is performed.

Further, when the predetermined recovery process is completed, the first request processing unit 502 requests the NVMe SSD $ k to execute a read command requesting the transfer of the read data to the own memory #i. You may specify the first identifier corresponding to and send it to DCM # j again.

If the read data is not delivered even if the read command execution request is re-executed after the recovery process is executed, the recovery unit 504 may notify the host device 110 to that effect. Further, the read data normally transferred to the memory #i is transferred to the host device 110.

Next, with reference to FIG. 6, a functional configuration example in which CM # i operates as a DCM will be described. Here, the CM in charge when CM # i is DCM may be expressed as "CM # j in charge" (j ≠ i). In addition, NVMe SSD under DCM # i may be referred to as "NVMe SSD $ k".

FIG. 6 is a block diagram showing a functional configuration example of DCM # i. In FIG. 6, DCM # i includes a second reception unit 601 and a second request processing unit 602. The second reception unit 601 and the second request processing unit 602 are functions that serve as control units, and specifically, for example, by causing the CPU #i to execute a program stored in the memory #i, or. The function is realized by CA # i and PCIe SW # i. The processing result of each functional unit is stored in, for example, the memory #i.

The second reception unit 601 corresponds to a read command requesting NVMe SSD $ k to transfer read data from the responsible CM # j (another storage control device) to the memory # j of the responsible CM # j. Accepts the execution request of the read command for which the identifier (second identifier) is specified. The second identifier specified is, for example, the SQ ID assigned to the read command.

When the second request processing unit 602 receives the execution request of the read command in which the second identifier is specified, the second request processing unit 602 requests the transfer of the read data to the other memory # j and is designated in the other FPGA # j. Issue a read command to NVMe SSD $ k requesting the transfer of predetermined data to the buffer area corresponding to the second identifier.

That is, the execution request of the read command in which the second identifier is specified requests the transfer of the read data to the other memory # j, and at the same time, the predetermined read command to the buffer area corresponding to the second identifier in the other FPGA # j is specified. Corresponds to the execution request of the read command that requests the transfer of data. The address of the buffer area corresponding to the second identifier in the other FPGA # j may be included in the execution request from the responsible CM # j, or may be inquired to the responsible CM # j. ..

When the NVMe SSD $ k notifies the transfer completion of the data requested by the read command, the second request processing unit 602 sends a completion response specifying the second identifier to the CM # j in charge. The completion response indicates that the read command has completed. Further, when the NVMe SSD $ k notifies the transfer error of the data requested by the read command, the second request processing unit 602 transmits an error response specifying the second identifier to the CM # j in charge. The error response indicates that the read command did not complete successfully.

Specifically, for example, the second request processing unit 602 sends a read command requesting the transfer of read data to the other memory # j to the buffer area corresponding to the second identifier in the other FPGA # j. Add a transfer request for the specified data. Then, the second request processing unit 602 stores the read command to which the transfer request is added in the first queue corresponding to the second identifier in the own memory #i. The first queue is a queue for storing commands, for example, SQ.

Next, when the information indicating the completion of the read command is written by the NVMe SSD $ k to the second queue corresponding to the second identifier in the own memory #i, the second request processing unit 602 is second. The transfer completion response specifying the identifier of is sent to the responsible CM # j. The second queue is a queue that stores information indicating the completion status of the command, and is, for example, a CQ.

A more detailed operation example when accessing NVMe SSD $ k will be described later with reference to, for example, FIGS. 8 and 10.

(Operation example of storage system 100)
Next, an operation example of the storage system 100 will be described with reference to FIG. 7. However, CM # 0 is the responsible CM, and CM # 1 is the DCM. Further, it is assumed that a failure occurs between PCIe SW # 0 of the responsible CM # 0 and NVMe SSD $ 0, and the responsible CM # 0 cannot directly access the NVMe SSD $ 0.

FIG. 7 is an explanatory diagram showing an operation example of the storage system 100. In FIG. 7, (1) each CM # 0 and # 1 initializes the constituent device and performs memory mapping of each memory # 0 and # 1. (2) The CM # 0 in charge specifies the SQ-ID and sends a read command execution request to the DCM # 1. The read command is a read command that requests NVMe SSD $ 0 to transfer read data to memory # 0.

(3) DCM # 1 has a command format as shown in FIG. 3, and is a predetermined data (data for one block) in the Doorbell register corresponding to the SQ-ID in FPGA # 0 of the CM # 0 in charge. Add the transfer request to the read command. Then, DCM # 1 transmits the read command to NVMe SSD $ 0.

Specifically, for example, DCM # 1 sets the transfer destination address of the data area in the memory # 0 of the CM # 0 in charge to the data address 1 of the read command. Further, DCM # 1 sets the address of the Doorbell register corresponding to the SQ-ID in FPGA # 0 of the CM # 0 in charge to the data address 2. Further, DCM # 1 sets the Number of Logical Blocks as "L + 1".

(4) NVMe SSD $ 0 reads L blocks from Starting LBA to the data area of memory # 0 of CM # 0 in charge (the address stored in the data address 1 of the read command) in response to the read command. Write data.

(5) NVMe SSD $ 0 writes data for one block following the read data to the Doorbell register area (address stored in the data address 2) in FPGA # 0 of the responsible CM # 0.

(6) FPGA # 0 issues an interrupt to CPU # 0 of the responsible CM # 0 by specifying the MSI-X ID associated with the Doorbell ID of the Doorbell register for which writing has been completed. For example, when the writing to the Doorbell register of Doorbell ID "0" is completed, an interrupt specifying MSI-X ID "0" is issued to CPU # 0. In this case, CPU # 0 sets the interrupt flag corresponding to the MSI-X ID "0" in the ID correspondence table 400 to "1".

(7) When CPU # 0 receives a transfer completion response with SQ-ID specified from DCM # 1, it refers to the interrupt flag corresponding to the specified SQ-ID in the ID correspondence table 400 and reads data. Determining the delivery status of.

Thereby, the CM # 0 in charge can determine whether or not the transfer of the read data is normally performed when the NVMe SSD $ 0 is accessed via the DCM # 1.

(Processing procedure of storage system 100)
Next, the processing procedure of the storage system 100 will be described. First, the processing procedure in the normal state of the storage system 100 will be described with reference to FIG. However, CM # 0 is the responsible CM, and CM # 1 is the DCM. Further, it is assumed that a failure occurs between PCIe SW # 0 of the responsible CM # 0 and NVMe SSD $ 0, and the responsible CM # 0 cannot directly access the NVMe SSD $ 0.

FIG. 8 is a sequence diagram showing an example of a normal processing procedure of the storage system 100. FIG. 9 is an explanatory diagram (No. 1) showing an example of updating the ID correspondence table 400. In the sequence diagram of FIG. 8, first, CPU # 0 of the CM # 0 in charge creates ID correspondence information in the ID correspondence table 400 (step S801).

Here, the SQ ID assigned to the newly issued command is set to "1". In this case, as shown in FIG. 9A, the ID correspondence information 400-2 corresponding to the SQ ID “1” is created and newly registered in the ID correspondence table 400. In the ID correspondence table 400, the ID correspondence information 400-1 corresponding to the SQ ID "0" assigned to the read command being executed is registered.

Next, CPU # 0 of CM # 0 in charge specifies the specified SQ ID "1" and requests NVMe SSD $ k to execute a read command requesting transfer of read data to memory # i. It is transmitted to DCM # j (step S802).

When the CPU # 1 of the DCM # 1 receives the execution request of the read command with the SQ ID "1" specified, the CPU # 1 of the DCM # 1 receives a predetermined command to the Doorbell register corresponding to the SQ ID "1" in the FPGA # 0 of the CM # 0 in charge. The read command to which the data transfer request is added is written to the SQ corresponding to the SQ ID “1” on the memory # 1 (step S803).

For example, a read command in the format shown in FIG. 3 is written to the SQ. Information for specifying the read command Opcode, data address 1, data address 2, Starting LBA, and Number of Logical Blocks is included in, for example, the execution request from the CM # 0 in charge. More specifically, for example, CPU # 1 of DCM # 1 sets the transfer destination address to the Doorbell register corresponding to SQ ID "1" to data address 2, and sets Number of Logical Blocks as "L + 1". do.

Next, CPU # 1 of DCM # 1 writes the tail value of SQ to the Submission Queue Doorbell register of NVMe SSD $ 0 (step S804). The Tail value is to indicate to NVMe SSD $ 0 that a new command has been sent for processing.

Next, the NVMe SSD $ 0 acquires a read command from the SQ on the memory # 1 of the DCM # 1 (step S805). Then, the NVMe SSD $ 0 transfers the read data for the L block read from the Starting LBA to the memory # 0 of the responsible CM # 0 based on the data address 1 of the read command (step S806).

Next, NVMe SSD $ 0 reads the data for one block following the read data for the L block based on the data address 2 of the read command, and the read data for one block is used by the CM # 0 in charge. Transfer to the Doorbell register of FPGA # 0 (step S807).

FPGA # 0 of the responsible CM # 0 issues an interrupt specifying the MSI-X ID associated with the Doorbell ID of the Doorbell register for which writing has been completed (step S808). CPU # 0 of the CM # 0 in charge sets the interrupt flag corresponding to the MSI-X ID in the ID correspondence table 400 to “1” (step S809).

Here, it is assumed that the writing to the Doorbell register of Doorbell ID "1" is completed and the interrupt specifying MSI-X ID "1" is issued. In this case, as shown in FIG. 9B, the interrupt flag of the ID correspondence information 400-2 corresponding to the MSI-X ID "1" is set to "1". Since the read command with the SQ ID "0" completed normally, the ID correspondence information 400-1 has been deleted. Further, the ID correspondence information 400-3 corresponding to the SQ ID "2" assigned to the newly issued read command is registered.

When the transfer of the data requested by the read command is completed, the NVMe SSD $ 0 writes a transfer completion notification in the CQ corresponding to the CQ ID "1" on the memory # 1 of the DCM # 1 (step S810). Then, NVMe SSD $ 0 issues an MSI-X interrupt to CPU # 1 of DCM # 1 (step S811).

Upon receiving the MSI-X interrupt, the CPU # 1 of the DCM # 1 confirms the CQ corresponding to the CQ ID "1" and completes the process (step S812). Next, CPU # 1 of DCM # 1 writes the Head value of CQ to the Complex Queue Doorbell register of NVMe SSD $ 0 (step S813). The Head value is for indicating to NVMe SSD $ 0 that the processing is completed.

Then, CPU # 1 of DCM # 1 transmits a completion response (Good) for which SQ-ID "1" is specified to CPU # 0 of CM # 0 in charge (step S814). When CPU # 0 of the CM # 0 in charge receives the completion response with SQ-ID "1" specified, it confirms the interrupt flag corresponding to SQ-ID "1" in the ID correspondence table 400 and reads data. (Step S815).

Here, since the interrupt flag corresponding to the SQ-ID "1" is "1", the read data is delivered, and it is determined that the read command with the specified SQ ID "1" completed normally. Then, CPU # 0 of the CM # 0 in charge deletes the entry corresponding to the completed read command from the ID correspondence table 400 (step S816).

Here, as shown in (C) of FIG. 9, the ID correspondence information 400-2 (entry) corresponding to the SQ-ID "1" is deleted from the ID correspondence table 400.

As a result, even when the NVMe SSD $ 0 is accessed from the responsible CM # 0 via the DCM # 1, it is possible to secure the delivery guarantee of the read data.

Next, a processing procedure when an abnormality occurs in the storage system 100 will be described with reference to FIG.

FIG. 10 is a sequence diagram showing an example of a processing procedure when an abnormality occurs in the storage system 100. FIG. 11 is an explanatory diagram (No. 2) showing an example of updating the ID correspondence table 400. In the sequence diagram of FIG. 10, first, CPU # 0 of the CM # 0 in charge creates ID correspondence information in the ID correspondence table 400 (step S1001).

Here, the SQ ID assigned to the newly issued command is set to "1". In this case, as shown in FIG. 11D, the ID correspondence information 400-2 corresponding to the SQ ID "1" is created and newly registered in the ID correspondence table 400. In the ID correspondence table 400, the ID correspondence information 400-1 corresponding to the SQ ID "0" assigned to the read command being executed is registered.

Next, CPU # 0 of CM # 0 in charge specifies the specified SQ ID "1" and requests NVMe SSD $ k to execute a read command requesting transfer of read data to memory # i. It is transmitted to DCM # j (step S1002).

When the CPU # 1 of the DCM # 1 receives the execution request of the read command specifying the SQ ID "1", the CPU # 1 of the DCM # 1 receives a predetermined command to the Doorbell register corresponding to the SQ ID "1" in the FPGA # 0 of the CM # 0 in charge. The read command to which the data transfer request is added is written to the SQ corresponding to the SQ ID “1” on the memory # 1 (step S1003).

Next, CPU # 1 of DCM # 1 writes the tail value of SQ to the Submission Queue Doorbell register of NVMe SSD $ 0 (step S1004). Next, the NVMe SSD $ 0 acquires a read command from the SQ on the memory # 1 of the DCM # 1 (step S1005).

Then, the NVMe SSD $ 0 transfers the read data for the L block read from the Starting LBA to the memory # 0 of the responsible CM # 0 based on the data address 1 of the read command (step S1006). Next, NVMe SSD $ 0 reads the data for one block following the read data for the L block based on the data address 2 of the read command, and the read data for one block is used by the CM # 0 in charge. Transfer to the Doorbell register of FPGA # 0 (step S1007).

Here, due to a temporary hardware failure or the like, the link between PCIe SW # 0 of CM # 0 in charge and PCIe SW # 1 of DCM # 1 is broken, and data transfer to memory # 0 and to the Doorbell register are performed. Suppose the data transfer fails. In this case, since the writing to the Doorbell register of Doorbell ID "1" is not executed, the MSI-X interrupt is not issued. Therefore, as shown in FIG. 11 (E), the interrupt flag of the ID correspondence information 400-2 corresponding to the SQ ID “1” remains “0”.

When the transfer of the data requested by the read command is completed, the NVMe SSD $ 0 writes the transfer completion notification to the CQ corresponding to the CQ ID "1" on the memory # 1 of the DCM # 1 (step S1008). Then, NVMe SSD $ 0 issues an MSI-X interrupt to CPU # 1 of DCM # 1 (step S1009).

Upon receiving the MSI-X interrupt, the CPU # 1 of the DCM # 1 confirms the CQ corresponding to the CQ ID "1" and completes the process (step S1010). Next, CPU # 1 of DCM # 1 writes the Head value of CQ to the Complex Queue Doorbell register of NVMe SSD $ 0 (step S1011).

Then, CPU # 1 of DCM # 1 transmits a completion response (Good) for which SQ-ID "1" is specified to CPU # 0 of CM # 0 in charge (step S1012). When CPU # 0 of the CM # 0 in charge receives the completion response with SQ-ID "1" specified, it confirms the interrupt flag corresponding to SQ-ID "1" in the ID correspondence table 400 and reads data. (Step S1013).

Here, as shown in FIG. 11 (F), since the interrupt flag corresponding to the SQ-ID "1" is "0", the read data is not delivered and the read with the SQ ID "1" specified. It is determined that the command has not completed normally. Then, the CPU # 0 of the CM # 0 in charge retransmits the execution request of the read command specifying the SQ ID “1” to the DCM # j after performing the predetermined recovery process (step S1014).

As a result, it is possible to detect that the data transfer has failed due to a temporary failure of the hardware or the like. In addition, the read command can be re-requested from DCM # j by performing recovery processing such as reconstructing the route (link).

(Other Examples of Storage System 100)
Next, another embodiment of the storage system 100 will be described with reference to FIGS. 12 and 13. Here, a case where a plurality of CEs (12CM configurations) are included in the storage system 100 will be described.

FIG. 12 is an explanatory diagram (No. 1) showing another embodiment of the storage system 100. In FIG. 12, the storage system 100 includes CE $ 0 to $ 5. Each CE $ 0- $ 5 has two CMs and an NVMe SSD, respectively. In FIG. 12, a part of the hardware configuration of each CM # i is excerpted and displayed.

In the storage system 100, the CMs are communicably connected to each other via an FRT (Front End Router) 1200. The FRT 1200 has a PCIe SW and connects CMs to each other so as to be communicable. Although only the FRT1200 is described as the FRT connecting the CMs, it may be made redundant by two or more FRTs.

Here, CM # 0 is the responsible CM, and CM # 1 is the DCM. Further, it is assumed that a failure occurs between PCIe SW # 0 of the responsible CM # 0 and NVMe SSD $ 0, and the responsible CM # 0 cannot directly access the NVMe SSD $ 0. In this case, the execution request of the read command from the responsible CM # 0 to the DCM # 1 is made via the FRT1200.

In addition, data transfer from NVMe SSD $ 0 to CM # 0 in charge is also performed via FRT1200. In FIG. 12, the thick arrow indicates the route from the read data requested from the host device 110 to the NVMe SSD # 0 to reach the host device 110.

Next, a case where the CM in charge and the NVMe SSD having the read data are arranged in different CEs will be described with reference to FIG. However, the system configuration of the storage system 100 is the same as the system configuration shown in FIG. Further, in FIG. 13, a part of the hardware configuration of each CM # i is excerpted and displayed.

FIG. 13 is an explanatory diagram (No. 2) showing another embodiment of the storage system 100. Here, CM # 11 is the responsible CM, and CM # 1 (or CM # 0) is the DCM. That is, the NVMe SSD having the read data is set to NVMe SSD $ 0 in CE $ 0.

In this case, the execution request of the read command from the responsible CM # 11 to the DCM # 1 is made via the FRT1200. Data transfer from NVMe SSD $ 0 to CM # 11 in charge is also performed via FRT1200. In FIG. 13, the thick arrow indicates the route from the read data requested from the host device 110 to the NVMe SSD # 0 to reach the host device 110.

As described above, according to the CM # i (CM in charge) according to the embodiment, when the NVMe SSD $ k is accessed via the DCM # j, the own memory #i is used for the NVMe SSD $ k. Requests the transfer of read data to DCM # j and the execution request of the read command to request the transfer of the predetermined data to the Doorbell register corresponding to the specified ID (SQ ID) in the own FPGA # i. Can be sent.

As a result, when the NVMe SSD $ k cannot be directly accessed, the transfer of the read data to the own memory #i is requested, and the predetermined data is transferred to the Doorbell register corresponding to the specified SQ ID in the own FPGA # i. You can request the execution of the requested read command.

Further, according to CM # i (CM in charge), when an interrupt for which an ID (MSI-X ID related to SQ ID) is specified is received from FPGA # i, the interrupt flag corresponding to the ID is enabled. Can be done. FPGA # i is an integrated circuit included in CM # i, includes a Doorbell register for data transfer, and specifies an ID (MSI-X ID) corresponding to the Doorbell register when writing to the Doorbell register is performed. Issue the interrupt to CPU # i.

This makes it possible to determine the writing status to the Doorbell register in FPGA # i.

Further, according to CM # i (CM in charge), the transfer of the read data to the own memory #i issued from DCM # j to NVMe SSD $ k in response to the execution request is requested, and the specified ID ( When the execution of the read command requesting the transfer of the predetermined data to the Doorbell register corresponding to the SQ ID) is completed, the completion response specifying the ID (SQ ID) can be received from DCM # j.

This makes it possible to know that the execution of the read command submitted to DCM # j has been completed. However, because of the PCIe posted write, it is not possible to know whether or not the read data has been normally transferred to the memory #i only by the completion response from the DCM # j.

Further, according to CM # i (CM in charge), when a completion response in which an ID (SQ ID) is specified is received from DCM # j, the interrupt flag corresponding to the ID is referred to and the memory # i is sent to the own memory # i. The delivery status of read data can be determined.

As a result, the read status to the memory # i from the write status (data delivery status) to the Doorbell register in the FPGA # i, which is performed using the same route as the data transfer from the NVMe SSD $ k to the memory # i. The delivery status of the data can be determined.

Further, according to CM # i (CM in charge), a Doorbell register corresponding to each ID (Doorbell ID) of a plurality of IDs (0 to N) is included, and when writing to the Doorbell register is performed, the Doorbell It is equipped with FPGA # i that issues an interrupt that specifies the ID corresponding to the register, and specifies the ID (SQ ID) that does not correspond to the other read commands among multiple IDs for the execution request of the read command, and DCM. It can be sent to # j.

This allows the SQ, CQ, Doorbell register, MSI to use the same ID that does not support other read commands for the SQ, CQ, Doorbell register, MSI-X interrupt in a series of flows for each read command. -Association by X ID can be performed. Therefore, even when a plurality of read commands are executed, by checking the MSI-X ID received from the FPGA # i by the CPU # i, it is possible to determine which SQ ID read command was completed normally. It becomes possible to distinguish.

Further, according to CM # i (CM in charge), when a completion response with an ID (SQ ID) is received from DCM # j and the interrupt flag corresponding to the ID is valid, the own memory # i It can be determined that the interrupt data has been delivered. Further, according to CM # i (CM in charge), if the interrupt flag corresponding to the ID is invalid, it can be determined that the read data has not been delivered to the own memory #i.

As a result, if a temporary hardware failure occurs and the writing to the Doorbell register in FPGA # i is not performed, it is determined that the data transfer to the own memory # i is not performed normally. be able to. On the other hand, when the writing is performed to the Doorbell register in FPGA # i, it can be determined that the data transfer to the own memory #i is also performed normally.

Further, according to CM # i (DCM), when another CM # j (in charge CM) accesses NVMe SSD $ k via the own CM, the other CM # j to NVMe SSD $ k. A read command that requests the transfer of read data to the memory # j of another CM # j and also requests the transfer of predetermined data to the Doorbell register corresponding to the specified ID in the FPGA # j of the other CM # j. Can accept execution requests. Then, according to CM # i (DCM), a read command can be issued to NVMe SSD $ k in response to an execution request from another CM # j. Further, according to CM # i (DCM), when NVMe SSD $ k notifies the transfer completion of the data requested by the read command, the completion response specifying the ID can be transmitted to another CM # j. can.

As a result, in response to the execution request from the CM # j in charge, the read command requesting the transfer of read data to another memory # j is sent to the Doorbell register of the Doorbell ID associated with the SQ ID corresponding to the read command. A transfer request for predetermined data can be added and issued to NVMe SSD $ k. Further, when the transfer of the data requested by the read command by NVMe SSD $ k is completed, the completion response indicating that the execution of the read command is completed can be notified to other CM # j.

Specifically, for example, DCM # i sets the transfer destination address of the data area in the memory # j of the responsible CM # j to the data address 1 of the read command. Further, DCM # i sets the address of the Doorbell register corresponding to the specified SQ-ID in FPGA # j of the CM # j in charge to the data address 2. Further, DCM # i sets the Number of Logical Blocks as "L + 1". As a result, after the transfer of the read data to the memory # j is completed, it is possible to request that the data for one block be transferred to the Doorbell register, and the read data delivery status is effectively suppressed while suppressing the increase in the communication volume. It is possible to make a distinction.

Further, according to CM # i (DCM), when another CM # j (in charge CM) accesses NVMe SSD $ k via the own CM, each ID of a plurality of IDs (0 to N) is used. In the own memory # i, an SQ in which the command corresponding to the ID is stored and a CQ in which the information indicating the completion status of the command is stored can be created. Then, according to CM # i (DCM), when an execution request is received from another CM # j, the read command can be stored in the SQ corresponding to the specified ID (SQ ID) in the own memory #i. can. Further, according to CM # i (DCM), when the information indicating the completion of the read command is written in the CQ corresponding to the ID in the own memory #i by NVMe SSD $ k, the completion specifying the ID is completed. The response can be sent to another CM # j.

This makes it possible to distinguish the completion status of each read command even when the execution of a plurality of read commands is requested.

From these facts, according to the storage system 100 and CM # i according to the embodiment, even when the responsible CM # i accesses NVMe SSD $ k via another CM # j (DCM). It is possible to secure the guarantee of data delivery, and it is possible to prevent data garbled in the host device 110 due to data omission (non-delivery).

The delivery status determination method described in the present embodiment can be realized by executing a program prepared in advance on a computer such as a storage control device. This delivery status determination program is recorded on a computer-readable recording medium such as a hard disk, a flexible disk, a CD (Compact Disc) -ROM, a DVD (Digital Versaille Disk), or a USB (Universal Serial Bus) memory, and is recorded by the computer. It is executed by reading from the medium. Further, the delivery status determination program may be distributed via a network such as the Internet.

The following additional notes are further disclosed with respect to the above-described embodiment.

(Appendix 1) An integrated circuit that includes a buffer area for data transfer and can reconfigure an internal circuit that issues an interrupt that specifies an identifier corresponding to the buffer area when writing to the buffer area is performed. Or with a dedicated integrated circuit
When accessing a storage device that transfers data without requesting a response via another storage control device, the transfer of read data to its own memory is requested and the specified first identifier in the integrated circuit is used. A read command execution request requesting the transfer of predetermined data to the corresponding buffer area is transmitted to the other storage control device.
When an interrupt with the specified first identifier is received from the integrated circuit, the interrupt flag corresponding to the first identifier is enabled and the interrupt flag is enabled.
When the execution of the read command issued from the other storage control device to the storage device is completed in response to the execution request, a completion response specifying the first identifier is received from the other storage control device. ,
When the completion response is received, the delivery status of the read data is determined with reference to the interrupt flag corresponding to the first identifier.
A storage control device characterized by having a control unit.

(Appendix 2) The integrated circuit is
A buffer area corresponding to each of the identifiers of the plurality of identifiers is included, and when writing to the buffer area is performed, an interrupt specifying the identifier corresponding to the buffer area is issued.
The control unit
The execution request of the read command is transmitted to the other storage control device by designating the first identifier corresponding to the other read command among the plurality of identifiers.
The storage control device according to Appendix 1, wherein the storage control device is characterized by the above.

(Appendix 3) The control unit is
When the completion response specifying the first identifier is received from the other storage control device, if the interrupt flag corresponding to the first identifier is valid, it is determined that the read data has been delivered, and the read data is determined. If the interrupt flag corresponding to the first identifier is invalid, it is determined that the read data has not been delivered.
The storage control device according to Appendix 1 or 2, wherein the storage control device is described.

(Appendix 4) The control unit is
When another storage control device accesses the storage device via its own device, the other storage control device requests transfer of read data from the other storage control device to the memory of the other storage control device, and the other Accepts a read command execution request requesting the transfer of predetermined data to the buffer area corresponding to the specified second identifier in the integrated circuit of the storage controller of
When the execution request is received, the read command is issued to the storage device.
When the storage device notifies the transfer completion of the data requested by the read command, a completion response specifying the second identifier is transmitted to the other storage control device.
The storage control device according to any one of Supplementary note 1 to 3, wherein the storage control device is characterized by the above.

(Appendix 5) The control unit is
When another storage control device accesses the storage device via its own device, the first queue in which the command corresponding to the identifier is stored for each identifier of the plurality of identifiers and the completion of the command are completed. Create a second queue in the own memory that stores the information indicating the status,
When the execution request is received, the read command is stored in the first queue corresponding to the second identifier in the own memory.
When the storage device writes the information indicating the completion of the read command to the second queue corresponding to the second identifier in its own memory, the completion response specifying the second identifier is sent to the other storage. Send to the controller,
The storage control device according to Appendix 4, wherein the storage control device is characterized by the above.

(Supplementary Note 6) The storage control device according to any one of Supplementary note 1 to 5, wherein the integrated circuit is an FPGA (Field Programmable Gate Array).

(Supplementary Note 7) The storage control device according to any one of Supplementary note 1 to 6, wherein the storage device is an NVMe (Non-Voltile Memory Express) SSD (Solid State Drive).

(Supplementary Note 8) The predetermined data is transferred to the buffer area corresponding to the first identifier after the transfer of the read data from the storage device to the own memory is completed. The storage control device described in.

(Appendix 9) An integrated circuit that includes a buffer area for data transfer and is capable of reconfiguring an internal circuit that issues an interrupt that specifies an identifier corresponding to the buffer area when writing to the buffer area is performed. Or for a storage control device with a dedicated integrated circuit,
When accessing a storage device that transfers data without requesting a response via another storage control device, the transfer of read data to its own memory is requested and the specified first identifier in the integrated circuit is used. A read command execution request requesting the transfer of predetermined data to the corresponding buffer area is transmitted to the other storage control device.
When an interrupt with the specified first identifier is received from the integrated circuit, the interrupt flag corresponding to the first identifier is enabled and the interrupt flag is enabled.
When the execution of the read command issued from the other storage control device to the storage device is completed in response to the execution request, a completion response specifying the first identifier is received from the other storage control device. ,
When the completion response is received, the delivery status of the read data is determined with reference to the interrupt flag corresponding to the first identifier.
A delivery status determination program characterized by executing a process.

(Appendix 10) A storage system including a plurality of storage control devices and a storage device for data transfer that does not require a response.
The storage control device of any of the plurality of storage control devices is
An integrated circuit that can reconfigure an internal circuit or a dedicated integrated circuit that includes a buffer area for data transfer and issues an interrupt that specifies an identifier corresponding to the buffer area when writing to the buffer area is performed. Equipped with a circuit
When accessing a storage device that transfers data without requesting a response via another storage control device, the transfer of read data to its own memory is requested and the specified first identifier in the integrated circuit is used. A read command execution request requesting the transfer of predetermined data to the corresponding buffer area is transmitted to the other storage control device.
The other storage control device is
When the execution request is received, the read command is issued to the storage device.
The storage control device is
When an interrupt with the specified first identifier is received from the integrated circuit, the interrupt flag corresponding to the first identifier is enabled and the interrupt flag is enabled.
When the execution of the read command issued from the other storage control device to the storage device is completed in response to the execution request, a completion response specifying the first identifier is received from the other storage control device. ,
When the completion response is received, the delivery status of the read data is determined with reference to the interrupt flag corresponding to the first identifier.
A storage system that features that.

(Appendix 11) The other storage control device is
For each of the plurality of identifiers, a first queue in which the command corresponding to the identifier is stored and a second queue in which the information indicating the completion status of the command is stored are created in the own memory.
When the execution request is received, the read command is stored in the first queue corresponding to the first identifier in the own memory.
When the storage device writes information indicating the completion of the read command to the second queue corresponding to the first identifier in its own memory, the storage control device sends a completion response specifying the first identifier. Send to,
The storage system according to Appendix 10, wherein the storage system is characterized in that.

100 Storage system 110 Host device 400 ID correspondence table 501 First reception unit 502 First request processing unit 503 Judgment unit 504 Recovery unit 601 Second reception unit 602 Second request processing unit 1200 FRT
cmd NVMe command # 0, # 1, # i, # j CM, CPU, memory, CA, FPGA, PCIe SW
$ 0- $ 5 CE, NVMe SSD

Claims (9)

An integrated circuit that can reconfigure an internal circuit or a dedicated integrated circuit that includes a buffer area for data transfer and issues an interrupt that specifies an identifier corresponding to the buffer area when writing to the buffer area is performed. Equipped with a circuit
When accessing a storage device that transfers data without requesting a response via another storage control device, the transfer of read data to its own memory is requested and the specified first identifier in the integrated circuit is used. A read command execution request requesting the transfer of predetermined data to the corresponding buffer area is transmitted to the other storage control device.
When an interrupt with the specified first identifier is received from the integrated circuit, the interrupt flag corresponding to the first identifier is enabled and the interrupt flag is enabled.
When the execution of the read command issued from the other storage control device to the storage device is completed in response to the execution request, a completion response specifying the first identifier is received from the other storage control device. ,
When the completion response is received, the delivery status of the read data is determined with reference to the interrupt flag corresponding to the first identifier.
A storage control device characterized by having a control unit. The integrated circuit is
A buffer area corresponding to each of the identifiers of the plurality of identifiers is included, and when writing to the buffer area is performed, an interrupt specifying the identifier corresponding to the buffer area is issued.
The control unit
The execution request of the read command is transmitted to the other storage control device by designating the first identifier corresponding to the other read command among the plurality of identifiers.
The storage control device according to claim 1. The control unit
When the completion response specifying the first identifier is received from the other storage control device, if the interrupt flag corresponding to the first identifier is valid, it is determined that the read data has been delivered, and the read data is determined. If the interrupt flag corresponding to the first identifier is invalid, it is determined that the read data has not been delivered.
The storage control device according to claim 1 or 2. The control unit
When another storage control device accesses the storage device via its own device, the other storage control device requests transfer of read data from the other storage control device to the memory of the other storage control device, and the other Accepts a read command execution request requesting the transfer of predetermined data to the buffer area corresponding to the specified second identifier in the integrated circuit of the storage controller of
When the execution request is received, the read command is issued to the storage device.
When the storage device notifies the transfer completion of the data requested by the read command, a completion response specifying the second identifier is transmitted to the other storage control device.
The storage control device according to any one of claims 1 to 3. The control unit
When another storage control device accesses the storage device via its own device, the first queue in which the command corresponding to the identifier is stored for each identifier of the plurality of identifiers and the completion of the command are completed. Create a second queue in the own memory that stores the information indicating the status,
When the execution request is received, the read command is stored in the first queue corresponding to the second identifier in the own memory.
When the storage device writes the information indicating the completion of the read command to the second queue corresponding to the second identifier in its own memory, the completion response specifying the second identifier is sent to the other storage. Send to the controller,
The storage control device according to claim 4. The storage control device according to any one of claims 1 to 5, wherein the integrated circuit is an FPGA (Field Programmable Gate Array). The storage control device according to any one of claims 1 to 6, wherein the storage device is an NVMe (Non-Voltile Memory Express) SSD (Solid State Drive). An integrated circuit that can reconfigure an internal circuit or a dedicated integrated circuit that includes a buffer area for data transfer and issues an interrupt with an identifier corresponding to the buffer area when writing to the buffer area is performed. For storage control devices equipped with circuits,
When accessing a storage device that transfers data without requesting a response via another storage control device, the transfer of read data to its own memory is requested and the specified first identifier in the integrated circuit is used. A read command execution request requesting the transfer of predetermined data to the corresponding buffer area is transmitted to the other storage control device.
When an interrupt with the specified first identifier is received from the integrated circuit, the interrupt flag corresponding to the first identifier is enabled and the interrupt flag is enabled.
When the execution of the read command issued from the other storage control device to the storage device is completed in response to the execution request, a completion response specifying the first identifier is received from the other storage control device. ,
When the completion response is received, the delivery status of the read data is determined with reference to the interrupt flag corresponding to the first identifier.
A delivery status determination program characterized by executing a process. A storage system that includes multiple storage control devices and storage devices that perform data transfers that do not require a response.
The storage control device of any of the plurality of storage control devices is
An integrated circuit that can reconfigure an internal circuit or a dedicated integrated circuit that includes a buffer area for data transfer and issues an interrupt that specifies an identifier corresponding to the buffer area when writing to the buffer area is performed. Equipped with a circuit
When accessing a storage device that transfers data without requesting a response via another storage control device, the transfer of read data to its own memory is requested and the specified first identifier in the integrated circuit is used. A read command execution request requesting the transfer of predetermined data to the corresponding buffer area is transmitted to the other storage control device.
The other storage control device is
When the execution request is received, the read command is issued to the storage device.
The storage control device is
When an interrupt with the specified first identifier is received from the integrated circuit, the interrupt flag corresponding to the first identifier is enabled and the interrupt flag is enabled.
When the execution of the read command issued from the other storage control device to the storage device is completed in response to the execution request, a completion response specifying the first identifier is received from the other storage control device. ,
When the completion response is received, the delivery status of the read data is determined with reference to the interrupt flag corresponding to the first identifier.
A storage system that features that.

Images