CN110109626B

CN110109626B - NVMe SSD command processing method based on FPGA

Info

Publication number: CN110109626B
Application number: CN201910420004.8A
Authority: CN
Inventors: 张京超; 刘旺; 孟凡廓; 朱凯晖; 乔立岩; 彭喜元
Original assignee: Harbin Institute of Technology
Current assignee: Harbin Institute of Technology
Priority date: 2019-05-20
Filing date: 2019-05-20
Publication date: 2022-01-25
Anticipated expiration: 2039-05-20
Also published as: CN110109626A

Abstract

An NVMe SSD command processing method based on an FPGA belongs to the technical field of data storage. The method solves the problem that the complexity of the process control state machine is increased along with the increase of commands or the increase of command types of NVMe SSD read-write operation. The invention designs the NVMe SSD command processing flow control module, combines various command execution flows, and can realize all command execution flows by using a simple flow control state machine, thereby being convenient for development and maintenance; and the method optimizes the state quantity and the state transfer condition of the flow control state machine while ensuring the function integrity, reduces the time overhead of the judgment step in the flow control state machine, and improves the development efficiency. The invention can be applied to the technical field of data storage.

Description

NVMe SSD command processing method based on FPGA

Technical Field

The invention belongs to the technical field of data storage, and particularly relates to an NVMe SSD command processing method.

Background

The high-speed data storage equipment is widely applied in the fields of high-speed data acquisition, cloud storage and the like. In order to improve the read-write speed of the storage device, reduce the volume and power consumption of the device and adapt to the development trend of miniaturization and compactness, the portable high-speed storage device can be constructed by using the NVMe SSD emerging in the storage field. The device uses FPGA as a control chip, and can realize continuous writing speed of more than 1 GByte/s.

The key for realizing the function of controlling the NVMe SSD by the FPGA is the design of control software FPGA, and a process control state machine can be written by using Verilog language to realize the NVMe command processing process. According to the content of the NVMe protocol, the NVMe SSD executes the read-write command in eight steps, each step including the transmission of a plurality of PCIe data packets. When the data volume required to be transmitted by the read-write command is large, the more PCIe data packets are transmitted in a corresponding command flow; when NVMe commands to be realized increase or command types increase, the states of the flow control state machine also increase, thereby causing the problems of complexity increase, consumption of FPGA logic resources increase and the like of the flow control state machine.

Disclosure of Invention

The invention aims to solve the problem that the complexity of a process control state machine is increased along with the increase of commands or the increase of command types of NVMe SSD read-write operation.

The technical scheme adopted by the invention for solving the technical problems is as follows: an NVMe SSD command processing method based on an FPGA comprises the following steps:

step A1, after receiving a user command, the FPGA updates a command Submission Queue, sends a register writing request to an NVMe (Non-Volatile Memory host controller interface specification) SSD (solid state disk), updates a command Submission Queue Tail Doorbell register (SQ Tail DB register, sub Queue Tail Doorbell) in the NVMe SSD, and jumps to step A2;

step A2, the FPGA enters a waiting state and judges whether a data packet is received, and when the FPGA receives the data packet, the FPGA jumps to step A3;

step A3, judging whether the received data packet is a memory writing request, if yes, jumping to step A4; otherwise, jumping to step A8;

step A4, judging whether the received data packet is a data writing request; if the data is requested to be written, the FPGA receives the data read from the NVMe SSD, and the step A2 is skipped; otherwise, jumping to step A5;

step A5, judging whether the received data packet is a write command completion queue request, if yes, jumping to step A6; otherwise, jumping to step A12;

step A6, judging whether the command is executed successfully, if so, updating the command completion queue, and jumping to step A7; otherwise, jumping to step A12;

step A7, the FPGA sends a memory writing request to the NVMe SSD, updates a command Completion Queue Head Doorbell register (CQ Head DB register) in the NVMe SSD, and jumps to step A13;

step A8, judging whether the received data packet is a memory reading request, if yes, jumping to step A9; otherwise, jumping to step A12;

step A9, judging whether the received data packet is a read command submission queue request, if so, sending a completion message by the FPGA, returning NVMe command information, and jumping to step A2; otherwise, jumping to step A10;

step A10, judging whether the received data packet is an address list reading request, if so, sending a completion message by the FPGA, returning address list information, and jumping to the step A2; otherwise, jumping to step A11;

step A11, judging whether the received data packet is a data reading request, if so, sending a completion message by the FPGA, returning data to be stored, and jumping to the step A2; otherwise, jumping to step A12;

step A12, error reporting, and jumping to step A13;

and step A13, ending.

The invention has the beneficial effects that: the invention relates to an NVMe SSD command processing method based on FPGA, which designs an NVMe SSD command processing flow control module, combines multiple command execution flows, and can realize all command execution flows by using a simple flow control state machine, thereby being convenient for development and maintenance; and the method optimizes the state quantity and the state transfer condition of the flow control state machine while ensuring the function integrity, reduces the time overhead of the judgment step in the flow control state machine, and improves the development efficiency.

Drawings

FIG. 1 is a flow chart of an NVMe SSD command processing method based on FPGA of the present invention;

FIG. 2 is a command control flow diagram of a prior NVMe SSD;

FIG. 3 is a process flow diagram of a prior NVMe SSD read-write command;

FIG. 4 is a process flow diagram of a prior NVMe SSD acquire State information command;

FIG. 5 is a process flow diagram of a prior NVMe SSD queue management, clear command;

FIG. 6 is a flowchart illustrating the step A4 of determining the corresponding function of the data packet according to the present invention;

FIG. 7 is a flow chart of the present invention updating the command submission queue doorbell register in the NVMe SSD;

FIG. 8 is a flow chart of the present invention updating the command completion queue doorbell register in NVMe SSD;

fig. 9 is a flowchart of the FPGA returning the completion message of the present invention.

Detailed Description

The first embodiment is as follows: as shown in fig. 1, the NVMe SSD command processing method based on FPGA of this embodiment includes the following steps:

step A1, after receiving a user command, the FPGA updates the command submission queue, sends a register writing request to the NVMe SSD, updates a doorbell register at the tail of the command submission queue in the NVMe SSD, and jumps to step A2;

step A7, the FPGA sends a memory writing request to the NVMe SSD, updates a command completion queue head doorbell register in the NVMe SSD, and jumps to step A13;

step A12, error reporting, and jumping to step A13;

and step A13, ending.

The NVMe SSD command processing flow is jointly participated by an NVMe host (FPGA in the invention) and an NVMe device (NVMe SSD). The FPGA host sends a read-write instruction to the NVMe SSD, sends data to be stored and receives data read from the NVMe SSD. Registers related to NVMe SSD command control include a command queue register in the FPGA and a doorbell register in the NVMe SSD. The command control flow of the NVMe SSD in the existing method is shown in fig. 2.

Step 1, writing a new command into a command Submission Queue (Submission Queue) by a host;

step 2, the host updates a command submission queue Tail Doorbell register (SQ Tail Doorbell) of the equipment and informs the equipment of a new command;

step 3, the device sends a memory reading request to the host, and the host returns a command of a corresponding address in the queue to the device;

step 4, the equipment executes the command;

step 5, the device sends a memory writing request to the host computer, and updates a command Completion Queue (Completion Queue) in the host computer;

step 6, the equipment sends MSI-X interruption to the host computer to remind the host computer to check the command to complete the queue;

step 7, the host judges whether the command is finished according to the status indicator in the command finishing queue;

step 8, the host updates a command completion queue Head Doorbell register (CQ Head Doorbell) of the equipment to inform the equipment of command completion;

step 4 includes processing data transmission processes such as a read address list request, a read data request, and a write data request, and specifically includes the following steps:

(1) the device sends a memory reading request to the host according to the address of the address list defined in the command;

(2) the host returns a completion message containing the address list to the equipment; if the device executes the write command, jumping to the step (3); if the device executes the read command, jumping to the step (5);

(3) the device sends a memory reading request to the host according to the address information in the address list;

(4) the host returns a completion message containing data to be written to the device;

(5) the device sends a memory writing request to the host according to the address information in the address list and sends the read data to the host;

the commands for controlling the NVMe SSD to perform read-write operation by using the FPGA comprise the following steps: queue management command, read-write command, clear command, and command to obtain status information.

The processing flow of the NVMe SSD read-write command is shown in fig. 3, the processing flow of the NVMe SSD obtain status information command is shown in fig. 4, and the processing flow of the NVMe SSD queue management and clear command is shown in fig. 5.

By comprehensively considering the execution flows of the above commands, the following rules can be found in conjunction with the command execution flow framework shown in fig. 2:

1. in the NVMe SSD command processing flow, except the step 4, other command execution flows are the same;

2. in the NVMe SSD command processing flow, the steps 1 and 7 are completed inside the FPGA, and data interaction with the NVMe SSD is not generated;

3. when the steps 2 and 8 are executed, the FPGA serves as the main equipment of the PCIe bus to initiate data transmission, and a doorbell register in the NVMe SSD is configured;

4. when the step 3-5 is executed, the NVMe SSD is used as a main device of a PCIe bus to initiate data transmission, and command queues, memory spaces and the like in the FPGA are accessed;

5. the MSI-X interrupt of step 6 is used to alert the host to check the command completion queue. Since the parallel nature of the FPGA allows it to monitor the command completion queue in real time without affecting the execution of other control flows, the MSI-X interrupt function of the NVMe SSD can be shut down, omitting this step.

In short, the NVMe command control flow can be divided into three parts, namely, FPGA local processing command, FPGA configuration NVMe SSD register, and NVMe SSD initiation of data transmission. According to the above rule, the present invention provides the control flow shown in fig. 1.

In the determining step of the flow shown in fig. 1, the FPGA parses a field used for indicating a "Type" (Format + Type) of a data packet in a received data packet header, and determines whether an operation of the data packet on the memory space is a read or a write; the FPGA analyzes the Address information in the received data packet header to obtain the Address of the memory space to be accessed, and judges the specific function of the data packet according to the function corresponding to the memory space. According to the invention, addresses of memory spaces corresponding to different functions are written into the control register of the NVMe SSD, when the NVMe SSD wants to acquire certain information or realize a certain function, only a data packet containing a corresponding type and an address needs to be sent, and after the FPGA receives a request from the NVMe SSD, the corresponding data packet is returned, so that data interaction between the NVMe SSD and the FPGA can be realized.

The memory space addresses, corresponding functions and addresses of NVMe SSD registers used to store these addresses in the FPGA defined by the present invention are shown in table 1 below:

TABLE 1 addresses and corresponding roles of memory spaces in FPGA

Addresses of memory spaces in FPGA	Memory space mapping function
		0x08005000～0x08005100	Storage address list
0x00000000～0x00001000	Storage command submission queue
		0x00001000～0x00002000	Storage command completion queue
0x01000000～0x07000000	Storing data to be read and written

The second embodiment is as follows: as shown in fig. 6, the first embodiment differs from the first embodiment in that: in step a4, it is determined whether the received data packet is a data writing request, and the specific process is as follows:

b1, the FPGA analyzes the address information from the received data packet header and jumps to the step B2;

b2, judging whether the address information analyzed in the step B1 is in the range of 0x 01000000-0 x07000000, if so, jumping to the step B4, otherwise, jumping to the step B3;

b3, jumping to A5 and ending;

step B4, judging that the function of the data packet received by the FPGA is to write the data read from the NVMe SSD into the FPGA, and jumping to step B5;

and B5, jumping to the step A2 and ending.

The steps A5, A9, A10 and A11 of the present invention are similar to the above judgment process of step A4.

One NVMe SSD corresponds to multiple command queues and doorbell registers. Specifically, one NVMe SSD corresponds to one management command submission queue, one management command completion queue, a plurality of I/O command submission queues, and a plurality of I/O command completion queues. And doorbell registers corresponding to the command queues are arranged in the NVMe SSD and are used for storing head and tail information of the corresponding queues. The register addresses of these doorbell registers in NVMe SSD are shown in table 2 below:

TABLE 2 doorbell register Address in NVMe SSD

Doorbell register variety	Address in NVMe SSD
		Management command submission queue tail doorbell register	0x28～0x2f
Management command completion queue head doorbell register	0x30～0x37
		I/O Command submit queue Tail doorbell register (nth one)	0x1000+4(2n-2)～0x1003+4(2n-2)
I/O Command completion queue head doorbell register (nth)	0x1000+4(2n-1)～0x1003+4(2n-1)

The third concrete implementation mode: as shown in fig. 7, the first embodiment differs from the first embodiment in that: the specific process of updating the command submission queue tail doorbell register in the NVMe SSD in the step a1 is as follows:

step C1, judging the number of a command Submission Queue (Submission Queue) updated inside the FPGA;

step C2, adding 1 to the value of the tail doorbell register to be updated in the command submission queue updated in the FPGA to obtain the value of the tail doorbell register to be updated;

step C3, determining the address of the tail doorbell register to be updated according to the number of the step C1;

step C4, packing the address obtained in step C3 and the value obtained in step C2, and supplementing the command type (format + type) information, length (length) information, number (Tag) information, and requesting device identification (request ID) information of the data packet header;

step C5, sending PCIe (peripheral component interconnect express) data packet;

and C6, ending.

The fourth concrete implementation mode: as shown in fig. 8, the first embodiment differs from the first embodiment in that: the updating of the command completion queue head doorbell register in the NVMe SSD in step a7 specifically comprises the following steps:

d1, the FPGA analyzes the completion queue identification in the received command completion entry;

d2, determining the address of the doorbell register to be updated according to the resolved completion queue identifier;

d3, the FPGA processes the command to complete the entry, and determines the value of the doorbell register to be updated;

step D4, packing the address and value of the header doorbell register to be updated, and supplementing the command type (format + type) information, length (length) information, number (Tag) information and requesting device identification (request ID) information of the data packet header;

step D5, sending PCIe data packet;

and D6, ending.

The fifth concrete implementation mode: as shown in fig. 9, the first embodiment differs from the first embodiment in that: the FPGA in steps a9, a10 and a11 sends a completion message, and the specific process is as follows:

e1, the FPGA analyzes the address of the received data packet header of the read memory request and records the number field in the current data packet header;

e2, reading data from the memory of the address analyzed in the step E1 by the FPGA;

a step E3 of packetizing the number field of the step E1 and the data read in the step E2 and supplementing command type (format + type) information, length (length) information, number (Tag) information, and requesting device identification (request ID) information of the data packet header;

step E4, sending PCIe data packet;

and E5, ending.

The sixth specific implementation mode: the first difference between the present embodiment and the specific embodiment is: when the FPGA receives a data packet, the FPGA analyzes a field used for expressing the type of the data packet in the received data packet header, and judges whether the operation of the data packet on the memory space is reading or writing; the FPGA analyzes the address information in the received data packet header to obtain a memory space address to be accessed, and the specific function of the data packet is judged according to the function corresponding to the memory space address.

The invention sets a wait state in which the FIFO buffer is scanned. And when a new data packet is read from the FIFO buffer, the FPGA judges the type of the data packet and executes corresponding operation. According to the design, a plurality of command execution flows are combined into a total command execution flow, and the command execution flows can be controlled by using one flow control state machine, so that the development and maintenance are facilitated, and the logic resources in the FPGA are saved. Meanwhile, aiming at the continuous judging steps in the dashed line frame of the figure 1, the if-else if-else statement is used, and the judging processes can be executed within 1 clock cycle (8ns) by means of the parallel characteristic of the FPGA, so that the time overhead generated by the FPGA executing the judging steps is very small, and the speed of the FPGA controlling the NVMe SSD to execute the read-write command is indirectly accelerated. In addition, the MSI-X interruption function is closed by configuring the control register of the NVMe SSD, so that the NVMe SSD cannot send the MSI-X interruption to the FPGA, the design work of related software is further simplified, and the state of waiting and judging the MSI-X interruption in a flow control state machine is omitted.

In order to embody the advantages of the design method proposed by the present invention, the control flows in fig. 3, 4, and 5 and the control flow in fig. 1 are implemented respectively, and other functional modules are omitted in the implementation process to highlight the logic resource consumption situation of the flow control module. The result shows that the traditional flow control scheme occupies 14 triggers and 46 lookup tables, and the scheme provided by the invention only occupies 8 triggers and 16 lookup tables, namely, the scheme provided by the invention saves the resources of the triggers and the lookup tables by 43% and 65% respectively compared with the traditional scheme.

The above-described calculation examples of the present invention are merely to explain the calculation model and the calculation flow of the present invention in detail, and are not intended to limit the embodiments of the present invention. It will be apparent to those skilled in the art that other variations and modifications of the present invention can be made based on the above description, and it is not intended to be exhaustive or to limit the invention to the precise form disclosed, and all such modifications and variations are possible and contemplated as falling within the scope of the invention.

Claims

1. An NVMe SSD command processing method based on an FPGA is characterized by comprising the following steps:

step A12, error reporting, and jumping to step A13;

and step A13, ending.

2. The method for processing the NVMe SSD command based on the FPGA of claim 1, wherein the step a4 judges whether the received packet is a request for writing data, and the specific process is as follows:

b3, jumping to A5 and ending;

and B5, jumping to the step A2 and ending.

3. The method for processing the NVMe SSD command based on the FPGA of claim 1, wherein the step A1 comprises the specific steps of updating a command submission queue tail doorbell register in the NVMe SSD:

step C1, judging the number of the command submission queue updated in the FPGA;

step C4, packaging the address obtained in step C3 and the value obtained in step C2, and supplementing the command type information, length information, number information and requesting device identification information of the data packet header;

step C5, PCIe data packet is sent;

and C6, ending.

4. The method for processing the NVMe SSD command based on the FPGA of claim 1, wherein the step A7 comprises the following specific steps of updating a command completion queue head doorbell register in the NVMe SSD:

step D4, packing the address and value of the head doorbell register to be updated, and supplementing the command type information, length information, number information and request equipment identification information of the data packet head;

step D5, sending PCIe data packet;

and D6, ending.

5. The NVMe SSD command processing method based on FPGA according to claim 1, wherein the FPGA in the steps A9, A10 and A11 sends a completion message, and the specific process is as follows:

step E3, packaging the number field of step E1 and the data read in step E2, and supplementing the command type information, length information, number information and requesting device identification information of the data packet header;

step E4, sending PCIe data packet;

and E5, ending.

6. The NVMe SSD command processing method based on the FPGA according to claim 1, wherein when the FPGA receives a data packet, the FPGA analyzes a field used for indicating the type of the data packet in the received data packet header, and determines whether the operation of the data packet on the memory space is read or written; the FPGA analyzes the address information in the received data packet header to obtain a memory space address to be accessed, and the specific function of the data packet is judged according to the function corresponding to the memory space address.