CN115586867A - NVMe controller - Google Patents
NVMe controller Download PDFInfo
- Publication number
- CN115586867A CN115586867A CN202211182563.8A CN202211182563A CN115586867A CN 115586867 A CN115586867 A CN 115586867A CN 202211182563 A CN202211182563 A CN 202211182563A CN 115586867 A CN115586867 A CN 115586867A
- Authority
- CN
- China
- Prior art keywords
- nvme
- module
- command
- queue
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0658—Controller construction arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/14—Handling requests for interconnection or transfer
- G06F13/16—Handling requests for interconnection or transfer for access to memory bus
- G06F13/1668—Details of memory controller
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/14—Handling requests for interconnection or transfer
- G06F13/16—Handling requests for interconnection or transfer for access to memory bus
- G06F13/18—Handling requests for interconnection or transfer for access to memory bus based on priority control
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
- G06F3/0679—Non-volatile semiconductor memory device, e.g. flash memory, one time programmable memory [OTP]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Advance Control (AREA)
Abstract
The application provides an NVMe controller, includes: one NVMe command subsystem and a plurality of NVM subsystems connected to the control register, respectively; the NVMe command subsystem is used for realizing an admin queue based on hardware and is used for executing an NVMe admin command and configuring a plurality of NVME IO queues; the NVM subsystems are arranged in parallel, the NVM subsystems are multiplexing systems, each NVM subsystem realizes an NVMe IO queue based on hardware, and each NVM subsystem is used for executing an NVMe IO command in parallel. According to the application, when the NVMe controller executes the command, interaction with the ARM core is not needed any more, communication overhead and resource consumption are effectively reduced, parallelism can be improved, delay is stable, the influence of an operating system and an algorithm cannot be caused, and the application reliability and the service life of the NVMe controller can be improved.
Description
Technical Field
The application relates to the technical field of computers, in particular to an NVMe controller.
Background
The NVMe protocol has been widely applied to consumer-level Solid State Disk (SSD) and enterprise-level SSD, and includes an all flash memory array (AFA) formed by the NVMe SSD, which is used in the data center, wherein the NVMe controller is used as a management interface for communicating the host with the SSD back end, and has an important influence on the performance of the NVMe SSD. The NVMe standard command flow is shown in FIG. 1, and comprises the following specific steps:
I. the host creates a command to be executed in a particular command submission queue, SQ.
And II, the host updates a queue tail doorbell register of the command submission queue SQ, and a new pointer pointing to a queue tail entry is stored in the register. By which the NVMe controller is instructed that a new command to be executed is submitted.
The nvme controller fetches the command from the command submission queue SQ in the host for subsequent execution.
And carrying out command arbitration by the NVMe controller, and selecting a command to be executed next time according to an arbitration mechanism from the plurality of commands which are acquired.
V. when the command execution is completed, the NVMe controller writes a command completion entry into the associated command completion queue. The completion entry includes the relevant command submission queue and identification information of the command.
And the NVMe controller sends an interrupt request to the host, and indicates that the host has a command to complete the entry to be processed.
And VII, processing a command completion entry in the completion queue command. The processing includes error processing required according to error presentation.
And VIII, updating a queue head doorbell register of the completion queue command by the host to indicate that the entry of the command completion entry is processed.
At present, an existing NVMe controller is composed of one or more ARM cores, the performance of an NVMe SSD has a great relationship with the number of ARM cores and the performance of the ARM cores themselves, the parallelism of a queue is increased, the overhead for the ARM cores is gradually increased, and especially for NVM commands, i.e., IO data commands, the execution mode is simple, so that it is urgently needed to design an NVMe controller capable of reducing resource consumption and improving the parallelism of queue execution.
Disclosure of Invention
In view of this, embodiments of the present application provide an NVMe controller to obviate or mitigate one or more of the disadvantages in the prior art.
The application provides an NVMe controller, includes: one NVMe command subsystem and a plurality of NVM subsystems connected to the control register, respectively;
the NVMe command subsystem is used for realizing an admin queue based on hardware and is used for executing an NVMe admin command and configuring a plurality of NVME IO queues;
the NVM subsystems are arranged in parallel, the NVM subsystems are multiplexing systems, each NVM subsystem realizes NVMe IO queues based on hardware, and each NVM subsystem is used for executing NVMe IO commands in parallel.
In some embodiments of the present application, the NVMe command subsystem comprises: the first handling analysis module group and the first processing interruption module group are connected with each other;
the first transport analysis module group is in communication connection with the control register and is used for transporting and analyzing an NVMe admin command issued by the host computer to the interior of the NVMe controller through the control register;
and the first processing interrupt module group is in communication connection with a host memory and is used for processing the NVMe admin command, sending a corresponding processing result to the host memory and then performing queue interrupt processing.
In some embodiments of the present application, the first transport resolution module group includes: the first doorbell arbitration module, the first command fetching module and the first decoding module are connected in sequence;
the first doorbell arbitration module is in communication connection with the control register and is used for checking whether a current queue contains an NVMe admin command issued by the host computer or not through the control register;
the first command fetching module is used for sending a corresponding DMA request to obtain an NVMe admin command when the first doorbell arbitration module detects that the current queue contains the NVMe admin command, and sending the NVMe admin command to the first decoding module;
the first decoding module is configured to analyze the NVMe admin command, determine an operation type of the NVMe admin command, and send the NVMe admin command to a corresponding processing module in the first processing interrupt module group according to the operation type.
In some embodiments of the present application, the first set of processing interruption modules comprises: the system comprises a first processing module group, a first sending CQ and interrupt module, a first sending module and a second sending module, wherein the first processing module group and the first sending CQ and interrupt module are connected with each other;
the first processing module group is connected with the first decoding module and used for processing NVMe admin commands of different operation types;
and the first issuing CQ and interrupt module is in communication connection with a host memory and is used for assembling a corresponding NVMe CQ according to the completion state of the NVMe admin command after the NVMe admin command is executed, sending the NVMe CQ to the host memory and then issuing an interrupt instruction of a corresponding queue.
In some embodiments of the present application, the first processing module group comprises: the system comprises an information processing module, a feature processing module and a queue processing module which are connected in sequence;
the information processing module, the feature processing module and the queue processing module are arranged in parallel, and the input ends of the information processing module, the feature processing module and the queue processing module are all connected to the first decoding module; the output ends of the information processing module, the characteristic processing module and the queue processing module are all connected to the first sending CQ and interruption module;
the information processing module is used for executing an NVMe admin command with an operation type of information processing;
the characteristic processing module is used for executing an NVMe admin command with the operation type of characteristic processing;
the queue processing module is used for executing the NVMe admin command with the operation type of queue processing to create a corresponding NVME IO queue request and dynamically configuring the NVM subsystem to which the NVME IO queue request belongs.
In some embodiments of the present application, each of the NVM subsystems shares a second doorbell arbitration module;
the second doorbell arbitration module is respectively connected with the control register and the NVMe command subsystem, and is used for checking whether a host issues an NVME IO queue request or not through the control register and/or receiving the NVME IO queue request sent by the NVMe command subsystem;
the second doorbell arbitration module is further configured to arbitrate the NVM subsystem to which the NVME IO queue request belongs after receiving or detecting the NVME IO queue request.
In some embodiments of the present application, each of the NVM subsystems further comprises: the second transport analysis module group and the second processing interruption module group are connected with each other;
the second transport analysis module group is connected with the second doorbell arbitration module and is used for transporting and analyzing the IO command corresponding to the NVME IO queue request into the NVME controller;
and the second processing interrupt module group is in communication connection with the host memory and is used for processing the IO command, sending a corresponding processing result to the host memory and then performing queue interrupt processing.
In some embodiments of the present application, the second transport resolution module group includes: the second command fetching module and the second decoding module are connected with each other;
the second command fetching module is used for sending a corresponding DMA request to obtain the IO command and sending the IO command to the second decoding module;
the second decoding module is used for analyzing the IO command and sending the IO command to the second processing interrupt module group.
In some embodiments of the present application, the second set of process interrupt modules comprises: the second processing module and the second CQ and interrupt sending module are connected with each other;
the second processing module is connected with the second decoding module and used for processing the IO command;
and the second sending CQ and interrupt module is in communication connection with the host memory and is used for assembling a corresponding NVMe CQ according to the completion state of the IO command after the IO command is executed, sending the NVMe CQ to the host memory and sending the interrupt instruction of the corresponding queue.
In some embodiments of the present application, the sending NVMe CQ to the host memory includes:
and generating and outputting a DMA request corresponding to the NVMe CQ so as to send the NVMe CQ to the host memory.
The NVMe controller provided by the application is provided with an NVMe command subsystem and a plurality of NVM subsystems which are respectively connected to a control register; the NVMe command subsystem is used for realizing an admin queue based on hardware and is used for executing an NVMe admin command and configuring a plurality of NVME IO queues; the NVM subsystems are arranged in parallel, the NVM subsystems are multiplexing systems, each NVM subsystem realizes an NVMe IO queue based on hardware, and each NVM subsystem is used for executing an NVMe IO command in parallel; compared with the NVMe controller adopting the ARM core, the NVMe controller provided by the application is firstly used for realizing a basic admin command through hardware, so that a plurality of NVME IO queues can be configured through the hardware, interaction with the ARM core is not needed, and the communication overhead is reduced; secondly, the NVME IO queue is realized through hardware, NVME IO commands can be executed in parallel, and parallelism is improved; each IO subsystem can be multiplexed, can be dynamically expanded according to performance requirements, and meets corresponding bandwidth; finally, hardware is adopted to realize a data IO path, compared with an ARM core, the delay is stable, the delay is not influenced by an operating system and an algorithm, and the delay has higher stability.
Additional advantages, objects, and features of the application will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and drawings.
It will be appreciated by those skilled in the art that the objects and advantages that can be achieved with the present application are not limited to the specific details set forth above, and that these and other objects that can be achieved with the present application will be more clearly understood from the detailed description that follows.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application, are incorporated in and constitute a part of this application, and are not intended to limit the application. The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the application. For purposes of illustrating and describing certain portions of the present application, the drawings may have been enlarged, i.e., may be larger, relative to other features of the exemplary devices actually made in accordance with the present application. In the drawings:
fig. 1 is a schematic diagram of a command standard flow of NVMe in the prior art.
Fig. 2 is a schematic structural diagram of an NVMe controller in an embodiment of the present application.
Fig. 3 is a schematic structural diagram of an NVMe command subsystem 1 in an NVMe controller in an embodiment of the present application.
Fig. 4 is a schematic structural diagram of the NVM subsystem 2 in the NVMe controller in an embodiment of the present application.
Fig. 5 is a schematic structural diagram of an NVMe controller for hardware auto-configuration multi-channel according to an application example of the present application.
Reference numerals:
1. an NVMe command subsystem;
2. an NVM subsystem;
3. a control register;
4. a first transport analysis module group;
41. a first doorbell arbitration module;
42. a first command fetching module;
43. a first decoding module;
5. a first set of processing interrupt modules;
51. a first group of processing modules;
511. an information processing module;
512. a feature processing module;
513. a queue processing module;
52. a first issue CQ and interrupt module;
6. a second doorbell arbitration module;
7. a second transport analysis module group;
71. a second command fetching module;
72. a second decoding module;
8. a second processing interrupt module group;
81. a second processing module;
82. the second issues the CQ and interrupt module.
Detailed Description
To make the objects, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below with reference to the following embodiments and the accompanying drawings. The exemplary embodiments and descriptions of the present application are provided to explain the present application and not to limit the present application.
Here, it should be further noted that, in order to avoid obscuring the present application with unnecessary details, only the structures and/or processing steps closely related to the scheme according to the present application are shown in the drawings, and other details not so relevant to the present application are omitted.
It should be emphasized that the term "comprises/comprising" when used herein, is taken to specify the presence of stated features, elements, steps or components, but does not preclude the presence or addition of one or more other features, elements, steps or components.
It is also noted that, unless otherwise specified, the term "coupled" is used herein to refer not only to a direct connection, but also to an indirect connection with an intermediate.
Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals denote the same or similar parts, or the same or similar steps.
Considering that the existing NVMe controller is composed of one or more ARM cores, the performance of the NVMe SSD has a great relationship with the number of ARM cores and the performance of the ARM cores themselves, increasing the parallelism of the queue will gradually increase the overhead for the ARM cores, but according to the execution process of the NVMe command, especially the NVM command, i.e. the IO data command, the execution mode is simple, the advantage of hardware parallelism (hardware multiple channels) can be utilized, and the resource consumption is very low (compared with the ARM cores).
When carrying out design of the NVMe controller, researchers of the application find that although there are many NVMe admin commands, by implementing the commands that must be implemented (the command identified as mandory in the NVMe1.3 protocol), hardware can dynamically open multiple queue channels, which not only improves the parallelism of queue execution, but also reduces the interaction overhead of admin commands and ARM cores, improves the performance of the NVMe controller, and has lower delay and higher throughput.
Based on this, the purpose of this application is to solve the situation that the parallelism of the NVMe controller found by the ARM core is too high to be improved, and to realize an NVMe controller design in which all the basic command sets are realized by hardware.
In one or more embodiments of the present application, NVM represents an acronym for non-volatile memory (non-volatile memory), which is a common form of flash memory for Solid State Drives (SSDs). The specification mainly provides a low-delay and internal concurrency native interface specification for the storage device based on the flash memory, and also provides native storage concurrency support for modern CPU, computer platforms and related applications, so that host hardware and software can fully utilize the parallelization storage capacity of the solid-state storage device. Compared with the AHCI of the previous era of mechanical Hard Disk Drive (HDD), the NVMe/NVMHCI reduces the waiting time of I/O operation, promotes the operation number in the same time, has larger capacity of operation queue and the like. NVM Express (NVMe), or called Non-Volatile Memory Host Controller Interface Specification (English: non-Volatile Memory Host Controller Interface Specification, abbreviated as NVMHCIS), is a logical device Interface Specification. It is a bus transfer protocol specification (equivalent to the application layer in the communication protocol) similar to AHCI based on a logical interface of a device for accessing a non-volatile memory medium (e.g., a solid state drive employing flash memory) attached through a PCI Express (PCIe) bus.
An embodiment of the NVMe controller is described in detail below with reference to fig. 2 to 4.
The embodiment of the application provides an NVMe controller, which specifically comprises the following contents:
one NVMe command subsystem 1 and a plurality of NVM subsystems 2 connected to the control register 3, respectively;
the NVMe command subsystem 1 is used for realizing an admin queue based on hardware and executing an NVMe admin command and configuring a plurality of NVME IO queues;
the NVM subsystems 2 are arranged in parallel, the NVM subsystems 2 are multiplexing systems, each NVM subsystem 2 realizes an NVMe IO queue based on hardware, and the NVM subsystems 2 are used for executing NVMe IO commands in parallel.
The admin queue refers to a management queue, the NVMe admin command refers to a basic instruction, and the NVMe IO queue refers to an input/output queue; NVMe IO command refers to input/output command.
As can be seen from the above description, in the NVMe controller provided in the embodiment of the present application, first, a basic admin command is implemented by hardware, multiple NVMe IO queues can be configured by hardware, interaction with an ARM core is not required, and communication overhead is reduced; secondly, the NVME IO queue is realized through hardware, NVME IO commands can be executed in parallel, and parallelism is improved; each IO subsystem can be multiplexed, can be dynamically expanded according to performance requirements, and meets corresponding bandwidth; finally, hardware is adopted to realize a data IO path, compared with an ARM core, the delay is stable, the delay is not influenced by an operating system and an algorithm, and the delay has higher stability.
In order to further improve the application reliability of the NVMe command subsystem 1, in the NVMe controller provided in the embodiment of the present application, the NVMe command subsystem 1 specifically includes the following contents:
a first transport analysis module group 4 and a first processing interruption module group 5 which are connected with each other;
the first carrying and analyzing module group 4 is in communication connection with the control register 3 and is used for carrying and analyzing an NVMe admin command issued by the host computer to the NVMe controller through the control register 3;
the first processing interrupt module group 5 is in communication connection with a host memory, and is configured to process the NVMe admin command, send a corresponding processing result to the host memory, and perform queue interrupt processing.
In order to further improve the application reliability of the first transport analysis module group 4, in the NVMe controller provided in the embodiment of the present application, the first transport analysis module group 4 specifically includes the following contents:
the first doorbell arbitration module 41, the first command fetching module 42 and the first decoding module 43 are connected in sequence;
the first doorbell arbitration module 41 is in communication connection with the control register 3, and is configured to check, via the control register 3, whether a current queue includes an NVMe admin command issued by a host;
the first command fetching module 42 is configured to, when the first doorbell arbitration module 41 detects that the current queue includes an NVMe admin command, send a corresponding DMA request to obtain the NVMe admin command, and send the NVMe admin command to the first decoding module 43;
the first decoding module 43 is configured to analyze the NVMe admin command, determine an operation type of the NVMe admin command, and send the NVMe admin command to a corresponding processing module in the first processing interrupt module group 5 according to the operation type.
In order to further improve the application reliability of the first processing interrupt module group 5, in the NVMe controller provided in the embodiment of the present application, the first processing interrupt module group 5 specifically includes the following contents:
a first processing module group 51 and a first CQ-issuing and interrupt module 52 connected to each other;
the first processing module group 51 is connected to the first decoding module 43, and is configured to process NVMe admin commands of different operation types;
the first sending CQ and interrupt module 52 is in communication connection with the host memory, and is configured to, after the NVMe admin command is executed, assemble a corresponding NVMe CQ according to a completion status of the NVMe admin command, send the NVMe CQ to the host memory, and send out an interrupt instruction of a corresponding queue.
In order to further improve the application reliability of the first processing module group 51, in the NVMe controller provided in the embodiment of the present application, the first processing module group 51 specifically includes the following contents: an information processing module 511, a feature processing module 512 and a queue processing module 513 connected in sequence;
the information processing module 511, the feature processing module 512 and the queue processing module 513 are arranged in parallel, and the input ends of the information processing module 511, the feature processing module 512 and the queue processing module 513 are all connected to the first decoding module 43; the output ends of the information processing module 511, the feature processing module 512 and the queue processing module 513 are all connected to the first outgoing CQ and interrupt module 52;
the information processing module 511 is configured to execute an NVMe admin command whose operation type is information processing;
the feature processing module 512 is configured to execute an NVMe admin command whose operation type is feature processing;
the queue processing module 513 is configured to execute an NVMe admin command whose operation type is queue processing, so as to create a corresponding NVMe IO queue request, and dynamically configure the NVM subsystem 2 to which the NVMe IO queue request belongs.
In order to further improve the application reliability of the NVM subsystems 2, in the NVMe controller provided in the embodiment of the present application, each NVM subsystem 2 shares a second doorbell arbitration module 6;
the second doorbell arbitration module 6 is respectively connected to the control register 3 and the queue processing module 513, and is configured to check whether a host issues an NVME IO queue request through the control register 3, and/or receive the NVME IO queue request sent by the NVME command subsystem 1;
the second doorbell arbitration module 6 is further configured to, after receiving or detecting the NVME IO queue request, obtain the NVM subsystem 2 to which the NVME IO queue request belongs, and send the NVME IO queue request to the corresponding NVM subsystem 2.
In order to further improve the application reliability of the NVM subsystem 2, in the NVMe controller provided in the embodiment of the present application, the NVM subsystem 2 further includes the following contents:
a second transport analysis module group 7 and a second processing interruption module group 8 which are connected with each other;
the second transport analysis module group 7 is connected with the second doorbell arbitration module 6, and is configured to transport and analyze the IO command corresponding to the NVME IO queue request into the NVME controller;
and the second processing interrupt module group 8 is in communication connection with the host memory and is used for processing the IO command, sending a corresponding processing result to the host memory and then performing queue interrupt processing.
In order to further improve the application reliability of the second transportation analysis module group 7, in the NVMe controller provided in the embodiment of the present application, the second transportation analysis module group 7 further includes the following contents:
a second fetch command module 71 and a second decode module 72 connected to each other;
the second command fetching module 71 is configured to issue a corresponding DMA request to obtain the IO command, and send the IO command to the second decoding module 72;
the second decoding module 72 is configured to parse the IO command and send the IO command to the second processing interrupt module group 8.
In order to further improve the application reliability of the second processing interrupt module group 8, in the NVMe controller provided in the embodiment of the present application, the second processing interrupt module group 8 further includes the following contents:
a second processing module 81 and a second CQ and interrupt issuing module 82 connected to each other;
the second processing module 81 is connected to the second decoding module 72, and is configured to process the IO command;
the second CQ sending and interruption module 82 is communicatively connected to the host memory, and configured to assemble, according to the completion status of the IO command, a corresponding NVMe CQ after the IO command is executed, send the NVMe CQ to the host memory, and send out an interruption instruction of the corresponding queue. Wherein NVMe CQ refers to completion queues.
In addition, in one or more embodiments described above, the specific implementation manner of sending the NVMe CQ to the host memory may be:
and generating and outputting a DMA request corresponding to the NVMe CQ so as to send the NVMe CQ to the host memory.
In order to further explain the scheme, the application example of the NVMe controller is further provided, specifically, the NVMe controller with the hardware auto-configuration multi-channel is provided, and the whole NVMe controller is composed of two parts, an admin subsystem and a plurality of NVM subsystems (IO subsystem and NVMe IO queue). The Admin subsystem and the NVM subsystem have the same integral structure, and are composed of doorbell arbitration modules (Admin _ db, NVM _ db), command fetching modules (Admin _ sq _ fetch, NVM _ sq _ fetch), decoding modules (Admin _ decode, NVM _ decode), processing modules (Admin _ info, admin _ feature, admin _ queue, NVM _ feature), and a command issuing cq and interrupt module (Admin _ cq, NVM _ cq). When a host end issues a queue request, a doorbell arbitration module can receive a response, arbitrate a queue with the current request, send the request to a command taking module, the module sends a DMA request, carries an NVMe admin command or an IO command into an NVMe controller, after the command is carried, a decoding module is responsible for analyzing the NVMe command, a processing module executes different operations according to different operation types, configures the NVME IO queue or reads and writes data, when the NVMe command is completed, sends a CQ and an interruption module completes the state according to the command of the NVMe module, assembles an NVMe CQ, sends the CQ to a host memory through DMA, and sends an interruption corresponding to the queue after the sending is completed. The mapping relation between the software queues and the hardware queues adopts a modular Hash method, so that each software queue can be mapped to the corresponding hardware queue, and the command execution parallelism is improved.
Specifically, for the NVMe admin subsystem, only one group of queues is provided, a command is acquired to the parsing module when the command is detected to exist in the queues, the NVMe IO queue request is parsed and created in the admin _ queue module, the IO hardware queues are dynamically configured, and after the execution is finished, the corresponding CQ and interrupt are sent out.
The NVM subsystem, each individual queue functions similarly, accepts the command distribution of NVM _ db, starts the execution of the command, decodes, executes, finally assembles CQ of the command and issues the interrupt.
Referring to fig. 5, the NVMe controller with hardware auto-configuration and multiple channels provided in the application example of the present application specifically includes the following contents:
( One) NVMe command subsystem 1 (which may also be referred to as: non-volatile memory host controller interface command Subsystem (Admin Subsystem) )
For the NVMe command subsystem 1, there is only one group of queues, which includes a first doorbell arbitration module 41 (admin _ db), a first command fetching module 42 (admin _ sq _ fetch), a first decoding module 43 (admin _ decode), a first processing module group 51, and a first CQ issuing and interrupting module 52, which are connected in sequence.
Wherein, the first processing module group 51 includes: an information processing module 511 (admin _ info), a feature processing module 512 (admin _ feature), and a queue processing module 513 (admin _ queue).
Through the construction of the NVMe command subsystem 1, basic admin commands are realized through hardware, NVMe admin command type pipelines are realized, multiple NVME IO queues can be configured through hardware, interaction with ARM cores is not needed, interaction process overhead between basic NVMe commands and ARM cores can be effectively reduced, and communication overhead is reduced.
The concrete description is as follows:
(1) And a first doorbell arbitration module 41, configured to detect whether an NVMe admin command exists in the current queue through the control register 3 (NVMe _ reg).
(2) The first fetching module 42 is configured to issue a DMA request, and transport the NVMe admin command to the NVMe controller, that is, fetch the NVMe admin command to the first decoding module 43.
(3) The first decoding module 43 is configured to analyze the NVMe admin command and determine an operation type of the NVMe admin command, so as to select one of the information processing module 511, the first feature processing module 512, and the queue processing module 513 in the first processing module group 51 as a target processing module according to the operation type, and send the NVMe admin command to the target processing module for processing.
(4) And the information processing module 511 is configured to perform corresponding information processing after receiving the NVMe admin command with the operation type of information processing.
(5) And the feature processing module 512 is configured to perform corresponding feature processing after receiving the NVMe admin command with the operation type being feature processing.
(6) And a queue processing module 513, configured to, after receiving the NVMe admin command whose operation type is queue processing, parse and create an NVMe IO queue request, and dynamically configure an IO hardware queue to which the NVMe IO queue belongs.
(7) The first sending CQ and interrupt module 52 is configured to assemble an NVMe CQ according to an NVMe command completion state after the execution of the NVMe admin command is completed, send a completion queue CQ (completion request) to a host memory through DMA, and send an interrupt corresponding to the queue after the sending is completed. The mapping relation between the software queues and the hardware queues adopts a modulo Hash method, so that each software queue can be mapped to the corresponding hardware queue, and the command execution parallelism is improved.
( Two) NVM subsystem 2 (which may also be referred to as: nonvolatile memory Subsystem (NVM Subsystem) )
For the NVM subsystems 2, NVME IO queues are realized through hardware, NVME IO commands can be executed in parallel, each NVM subsystem 2 shares the same second doorbell arbitration module 6 (NVM _ db), and each NVM subsystem 2 further includes: a second fetch command module 71 (nvm _ sq _ fetch), a second decode module 72 (nvm _ decode), a second processing module 81 (nvm _ feature), a second issue CQ and interrupt module 82 (nvm _ CQ).
By constructing the NVM subsystem 2, the NVME IO queue can be realized through hardware, NVMe IO commands can be executed in parallel, and the parallelism is improved; each IO subsystem can be multiplexed, can be dynamically expanded according to performance requirements, and meets corresponding bandwidth; and hardware can be adopted to realize a data IO path, compared with an ARM core, the delay is stable, the delay is not influenced by an operating system and an algorithm, and the delay has higher stability.
The concrete description is as follows:
(1) The second doorbell arbitration module 6 is configured to receive a response when it is detected that the host issues an NVMe IO queue request through the control register 3 (NVMe _ reg), arbitrate a queue currently having the NVMe IO queue request in each NVM subsystem 2, and send the request to the second command fetching module 71 in the queue.
The second doorbell arbitration module 6 may be further connected to the queue processing module 513, and is configured to receive an NVMe IO queue request (which may also be referred to as an IO command) sent by the queue processing module 513 after the queue processing module 513 executes the NVMe admin command whose operation type is queue processing, arbitrate a queue currently having the NVMe IO queue request in each NVM subsystem 2, and send the request to the second command fetching module 71 in the queue.
(2) The second fetching module 71 is configured to issue a DMA request, and carry the IO command into the NVMe controller, that is, fetch the IO command into the second decoding module 72.
(3) The second decoding module 72 is configured to parse the IO command and send the IO command to the second information processing module 511 for processing.
(4) The second information processing module 511 is configured to perform corresponding processing after receiving the IO command.
(5) And the second CQ and interruption sending module 82 is used for assembling the NVMe CQ according to the IO command completion state after the execution of the IO command is finished, sending the CQ to the host memory through DMA, and sending interruption of the corresponding queue after the sending is finished. The mapping relation between the software queues and the hardware queues adopts a modulo Hash method, so that each software queue can be mapped to the corresponding hardware queue, and the command execution parallelism is improved.
Therefore, the NVMe controller provided by the application at least comprises the following improvements:
(1) NVMe admin command class pipeline implementation; the method has the technical effects that the interaction process overhead between the basic NVMe command and the ARM core is reduced;
(2) The hardware realizes an NVMe IO queue and an admin queue; the technical effect is that the overhead of an ARM core control data path is removed, and the time delay is reduced;
(3) Parallel processing is carried out on multiple NVME IO queues; the technical effect is that the IO command processing capacity is improved, and the parallelism is improved.
In summary, compared with the NVMe controller design adopting the ARM core, the multi-channel NVMe controller with hardware auto-configuration provided by the application example of the present application first implements the basic admin command through hardware, can configure the multi-NVMe IO queue through hardware, does not need to interact with the ARM core, and reduces communication overhead; secondly, the NVME IO queue is realized through hardware, NVME IO commands can be executed in parallel, and parallelism is improved; each IO subsystem can be multiplexed, can be dynamically expanded according to performance requirements, and meets corresponding bandwidth; finally, hardware is adopted to realize a data IO path, compared with an ARM core, the delay is stable, the delay is not influenced by an operating system and an algorithm, and the delay has higher stability.
It is to be understood that the present application is not limited to the particular arrangements and instrumentality described above and shown in the attached drawings. A detailed description of known structures is omitted herein for the sake of brevity. In the above embodiments, several specific structures are described and shown as examples. However, the device structure of the present application is not limited to the specific structure described and illustrated, and various changes, modifications, and additions may be made by those skilled in the art after comprehending the spirit of the present application.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made to the embodiment of the present application by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. An NVMe controller, comprising: one NVMe command subsystem and a plurality of NVM subsystems connected to the control register, respectively;
the NVMe command subsystem is used for realizing an admin queue based on hardware and is used for executing an NVMe admin command and configuring a plurality of NVME IO queues;
the NVM subsystems are arranged in parallel, the NVM subsystems are multiplexing systems, each NVM subsystem realizes NVMe IO queues based on hardware, and each NVM subsystem is used for executing NVMe IO commands in parallel.
2. The NVMe controller of claim 1, wherein the NVMe command subsystem comprises: the first handling analysis module group and the first processing interruption module group are connected with each other;
the first transport analysis module group is in communication connection with the control register and is used for transporting and analyzing an NVMe admin command issued by the host computer to the interior of the NVMe controller through the control register;
and the first processing interrupt module group is in communication connection with a host memory and is used for processing the NVMe admin command, sending a corresponding processing result to the host memory and then performing queue interrupt processing.
3. The NVMe controller of claim 2, wherein the first set of transport resolution modules comprises: the first doorbell arbitration module, the first command fetching module and the first decoding module are sequentially connected;
the first doorbell arbitration module is in communication connection with the control register and is used for checking whether a current queue contains an NVMe admin command issued by the host computer or not through the control register;
the first command fetching module is used for sending a corresponding DMA request to obtain an NVMe admin command when the first doorbell arbitration module detects that the current queue contains the NVMe admin command, and sending the NVMe admin command to the first decoding module;
the first decoding module is used for analyzing the NVMe admin command and determining the operation type of the NVMe admin command so as to send the NVMe admin command to a corresponding processing module in the first processing interrupt module group according to the operation type.
4. The NVMe controller of claim 3, wherein the first set of process interrupt modules comprises: the system comprises a first processing module group, a first sending CQ and interrupt module, a first sending module and a second sending module, wherein the first processing module group and the first sending CQ and interrupt module are connected with each other;
the first processing module group is connected with the first decoding module and used for processing NVMe admin commands of different operation types;
and the first sending CQ and interrupt module is in communication connection with a host memory and is used for assembling a corresponding NVMe CQ according to the completion state of the NVMe admin command after the NVMe admin command is executed, sending the NVMe CQ to the host memory and then sending an interrupt instruction of a corresponding queue.
5. The NVMe controller of claim 4, wherein the first set of processing modules comprises: the system comprises an information processing module, a feature processing module and a queue processing module which are connected in sequence;
the information processing module, the characteristic processing module and the queue processing module are arranged in parallel, and the input ends of the information processing module, the characteristic processing module and the queue processing module are all connected to the first decoding module; the output ends of the information processing module, the characteristic processing module and the queue processing module are all connected to the first sending CQ and interruption module;
the information processing module is used for executing an NVMe admin command with an operation type of information processing;
the characteristic processing module is used for executing an NVMe admin command with the operation type of characteristic processing;
the queue processing module is used for executing an NVMe admin command with the operation type of queue processing to create a corresponding NVME IO queue request and dynamically configuring an NVM subsystem to which the NVME IO queue request belongs.
6. The NVMe controller of claim 1, wherein each of the NVM subsystems shares a second doorbell arbitration module;
the second doorbell arbitration module is respectively connected with the control register and the NVMe command subsystem, and is used for checking whether a host issues an NVME IO queue request or not through the control register and/or receiving the NVME IO queue request sent by the NVMe command subsystem;
the second doorbell arbitration module is further configured to arbitrate the NVM subsystem to which the NVME IO queue request belongs after receiving or detecting the NVME IO queue request.
7. The NVMe controller of claim 6, wherein each of the NVM subsystems further comprises: the second transport analysis module group and the second processing interruption module group are connected with each other;
the second transport analysis module group is connected with the second doorbell arbitration module and is used for transporting and analyzing the IO command corresponding to the NVME IO queue request into the NVME controller;
and the second processing interrupt module group is in communication connection with the host memory and is used for processing the IO command, sending a corresponding processing result to the host memory and then performing queue interrupt processing.
8. The NVMe controller of claim 7, wherein the second set of transport resolution modules comprises: the second command fetching module and the second decoding module are connected with each other;
the second command fetching module is used for sending a corresponding DMA request to obtain the IO command and sending the IO command to the second decoding module;
the second decoding module is used for analyzing the IO command and sending the IO command to the second processing interrupt module group.
9. The NVMe controller of claim 8, wherein the second set of process interrupt modules comprises: the second processing module and the second CQ and interrupt sending module are connected with each other;
the second processing module is connected with the second decoding module and used for processing the IO command;
and the second CQ sending module is in communication connection with the host memory and is used for assembling a corresponding NVMe CQ according to the completion state of the IO command after the IO command is executed, sending the NVMe CQ to the host memory and sending an interrupt instruction of a corresponding queue.
10. The NVMe controller of claim 4 or 9, wherein the sending the NVMe CQ to the host memory comprises:
and generating and outputting a DMA request corresponding to the NVMe CQ so as to send the NVMe CQ to the host memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211182563.8A CN115586867B (en) | 2022-09-27 | 2022-09-27 | NVMe controller |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211182563.8A CN115586867B (en) | 2022-09-27 | 2022-09-27 | NVMe controller |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115586867A true CN115586867A (en) | 2023-01-10 |
| CN115586867B CN115586867B (en) | 2023-08-22 |
Family
ID=84778515
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211182563.8A Active CN115586867B (en) | 2022-09-27 | 2022-09-27 | NVMe controller |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115586867B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117149278A (en) * | 2023-10-30 | 2023-12-01 | 苏州元脑智能科技有限公司 | Command processing system, control method and host equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017131724A1 (en) * | 2016-01-29 | 2017-08-03 | Hewlett Packard Enterprise Development Lp | Host devices and non-volatile memory subsystem controllers |
| CN109147862A (en) * | 2018-08-30 | 2019-01-04 | 北京智芯微电子科技有限公司 | NVM tests accelerated method and system |
| US20190146684A1 (en) * | 2017-11-13 | 2019-05-16 | Western Digital Technologies, Inc. | System and method for qos over nvme virtualization platform using adaptive command fetching |
| CN110109626A (en) * | 2019-05-20 | 2019-08-09 | 哈尔滨工业大学 | A kind of NVMe SSD command handling method based on FPGA |
| CN110362274A (en) * | 2019-06-28 | 2019-10-22 | 苏州浪潮智能科技有限公司 | A kind of NVMe controller, networked storage devices and system |
-
2022
- 2022-09-27 CN CN202211182563.8A patent/CN115586867B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2017131724A1 (en) * | 2016-01-29 | 2017-08-03 | Hewlett Packard Enterprise Development Lp | Host devices and non-volatile memory subsystem controllers |
| US20190146684A1 (en) * | 2017-11-13 | 2019-05-16 | Western Digital Technologies, Inc. | System and method for qos over nvme virtualization platform using adaptive command fetching |
| CN109147862A (en) * | 2018-08-30 | 2019-01-04 | 北京智芯微电子科技有限公司 | NVM tests accelerated method and system |
| CN110109626A (en) * | 2019-05-20 | 2019-08-09 | 哈尔滨工业大学 | A kind of NVMe SSD command handling method based on FPGA |
| CN110362274A (en) * | 2019-06-28 | 2019-10-22 | 苏州浪潮智能科技有限公司 | A kind of NVMe controller, networked storage devices and system |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117149278A (en) * | 2023-10-30 | 2023-12-01 | 苏州元脑智能科技有限公司 | Command processing system, control method and host equipment |
| CN117149278B (en) * | 2023-10-30 | 2024-02-06 | 苏州元脑智能科技有限公司 | Command processing system, control method and host equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115586867B (en) | 2023-08-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10019181B2 (en) | Method of managing input/output(I/O) queues by non-volatile memory express(NVME) controller | |
| US9043494B2 (en) | Providing indirect data addressing in an input/output processing system where the indirect data address list is non-contiguous | |
| US9021215B2 (en) | Storage system exporting internal storage rules | |
| US9927984B2 (en) | Electronic system with interface control mechanism and method of operation thereof | |
| EP2958027A1 (en) | Memory controller for processing a number of commands | |
| US9400614B2 (en) | Method and system for programmable sequencer for processing I/O for various PCIe disk drives | |
| PT2245545E (en) | Providing indirect data addressing for a control block at a channel subsystem of an i/o processing system | |
| CN103282887A (en) | Controller and method for performing background operations | |
| KR20120087980A (en) | Multi-interface solid state disk(ssd), processing method and system thereof | |
| US8473699B2 (en) | Facilitating data compression during replication using a compressible configuration bit | |
| CN105190539A (en) | Method and apparatus to avoid deadlock during instruction scheduling using dynamic port remapping | |
| US20210019077A1 (en) | Automatic host buffer pointer pattern detection | |
| US8359425B2 (en) | Memory control device, memory device, and shutdown control method | |
| CN115586867B (en) | NVMe controller | |
| US10133492B2 (en) | Electronic system with interface control mechanism and method of operation thereof | |
| KR102645983B1 (en) | Open channel vector command execution | |
| US8230134B2 (en) | Fast path SCSI IO | |
| WO2017072868A1 (en) | Storage apparatus | |
| EP2194458A2 (en) | Request processing device, request processing system, and access testing method | |
| US20200242040A1 (en) | Apparatus and Method of Optimizing Memory Transactions to Persistent Memory Using an Architectural Data Mover | |
| US20140136740A1 (en) | Input-output control unit and frame processing method for the input-output control unit | |
| JP2014048810A (en) | Host system, storage device and communication method | |
| US10747615B2 (en) | Method and apparatus for non-volatile memory array improvement using a command aggregation circuit | |
| US10223013B2 (en) | Processing input/output operations in a channel using a control block | |
| CN112732176B (en) | SSD (solid State disk) access method and device based on FPGA (field programmable Gate array), storage system and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |