CN115543474B - NVMe-oF semi-unloading processing method and device and hardware acceleration equipment - Google Patents
NVMe-oF semi-unloading processing method and device and hardware acceleration equipment Download PDFInfo
- Publication number
- CN115543474B CN115543474B CN202211049777.8A CN202211049777A CN115543474B CN 115543474 B CN115543474 B CN 115543474B CN 202211049777 A CN202211049777 A CN 202211049777A CN 115543474 B CN115543474 B CN 115543474B
- Authority
- CN
- China
- Prior art keywords
- nvme
- host
- storage system
- data
- data storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 41
- 230000001133 acceleration Effects 0.000 title claims abstract description 34
- 238000012545 processing Methods 0.000 claims abstract description 104
- 238000013500 data storage Methods 0.000 claims abstract description 76
- 238000004088 simulation Methods 0.000 claims abstract description 41
- 230000005540 biological transmission Effects 0.000 claims abstract description 34
- 238000000034 method Methods 0.000 claims abstract description 30
- 230000003993 interaction Effects 0.000 claims abstract description 20
- 238000004590 computer program Methods 0.000 claims description 8
- 238000007726 management method Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 abstract description 12
- 239000010410 layer Substances 0.000 description 36
- 230000006870 function Effects 0.000 description 13
- 238000004891 communication Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000007792 addition Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 239000003999 initiator Substances 0.000 description 2
- 239000002346 layers by function Substances 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/445—Program loading or initiating
- G06F9/44594—Unloading
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/14—Handling requests for interconnection or transfer
- G06F13/20—Handling requests for interconnection or transfer for access to input/output bus
- G06F13/24—Handling requests for interconnection or transfer for access to input/output bus using interrupt
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
- H04L69/08—Protocols for interworking; Protocol conversion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2213/00—Indexing scheme relating to interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F2213/0026—PCI express
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application provides an NVMe-oF semi-unloading processing method and device and hardware acceleration equipment, wherein the method comprises the following steps: based on a local preset NVMe equipment simulation unit, acquiring an NVMe request command corresponding to a data processing request oF a data storage system from a host, converting the NVMe request command into an NVMe-oF request command, and initializing the NVMe equipment simulation unit by the host in advance based on a preset NVMe-oF semi-unloading rule; transmitting the NVMe-oF request command from the corresponding network transmission layer to the data storage system; and sending an interrupt instruction to the host after the interaction between the host and the data storage system is determined. The application can effectively improve the CUP utilization rate oF the host side on the basis oF ensuring the reliability and the effectiveness oF NVMe-oF data processing, and ensures that the host side can efficiently process core services.
Description
Technical Field
The application relates to the technical field oF computers, in particular to an NVMe-oF semi-unloading processing method and device and hardware acceleration equipment.
Background
The NVMe is a storage interface access protocol formally proposed in 2015, is specially designed for a Solid State Disk (SSD), and has the characteristics of more queues (64K), large depth (64K), no locking and the like compared with the traditional SSD in the form of SATA and the like, and fully utilizes the characteristics of a nonvolatile storage medium NVM, thereby having lower time delay and higher bandwidth. The NVMe-oF (NVMe over Fabric) is an NVMe storage network access protocol formally proposed in 2019, compared with the traditional iSCSI protocol for realizing network storage access, the NVMe-oF protocol reduces the conversion from NVMe to iSCSI protocol, and can realize network storage access based on native NVMe operation, thereby further reducing access delay and improving bandwidth. The access path oF the NVMe-ofs is shown in fig. 1, where nvme_initiator represents an NVMe initiator, and nvme_target represents an NVMe request processing end (connected to a physical NVMe device).
However, in the current manner oF driving access based on the NVMe-oe kernel, for example, NVMe over TCP, running the TCP protocol stack in kernel mode, maintaining the TCP state machine, frequent interrupt processing may cause huge performance overhead oF the host, especially in an application scenario with extremely large data processing capacity such as a data center, in this case, although the NVMe-oe is designed and optimized for the high-speed storage network, the CPU utilization oF the host side is still reduced, so that the host cannot efficiently process the core service.
Disclosure of Invention
In view oF this, embodiments oF the present application provide an NVMe-orf semi-offload processing method and apparatus, and a hardware acceleration device, so as to eliminate or improve one or more drawbacks existing in the prior art.
One aspect oF the application provides an NVMe-orf semi-offload processing method, comprising:
an NVMe equipment simulation unit based on local preset acquires an NVMe request command corresponding to a data processing request oF a data storage system from a host, and converts the NVMe request command into an NVMe-oF request command, wherein the host carries out initialization processing on the NVMe equipment simulation unit based on a preset NVMe-oF semi-unloading rule in advance;
transmitting the NVMe-oF request command from the corresponding network transmission layer to the data storage system;
and sending an interrupt instruction to the host after the interaction between the host and the data storage system is determined.
In some embodiments of the present application, before the sending of the interrupt instruction to the host after determining that the interaction between the host and the data storage system is completed, further comprises:
receiving an NVMe-oF reply command aiming at the data processing request, which is sent by a data storage system;
and converting the NVMe-oF reply command into an NVMe-oF reply command, and sending the NVMe reply command to the host.
In some embodiments of the present application, before the sending of the interrupt instruction to the host after determining that the interaction between the host and the data storage system is completed, further comprises:
receiving NVMe-oF reply data aiming at the data processing request and sent by a data storage system;
and sending the NVMe-oF reply data to the host.
In some embodiments of the present application, before the locally preset-based NVMe device simulation unit obtains an NVMe request command corresponding to a data processing request of a data storage system from a host, the method further includes:
constructing an NVMe device simulation unit based on local PCIe devices;
and sending a constructed message corresponding to the NVMe equipment simulation unit to the host, so that an NVMe driver in the host initializes the NVMe equipment simulation unit according to the constructed message, and creates a corresponding management queue and an IO queue, so that the host generates a corresponding NVMe request command and updates the IO queue after receiving a data processing request for a data storage system.
In some embodiments oF the present application, the sending the NVMe-oh request command from the corresponding network transport layer to the data storage system includes:
selecting one of the multiple network transmission layers as a target network transmission layer;
and based on the target network transmission layer, sending the NVMe-oF request command to the data storage system through a local MAC network outlet.
In some embodiments of the application, the network transport layer comprises: a TCP offload engine and an RDMA module.
Another aspect oF the present application provides an NVMe-orf semi-unloading processing apparatus, comprising:
the system comprises a half-unloading module, a data storage system and a data processing module, wherein the half-unloading module is used for acquiring an NVMe request command corresponding to a data processing request oF the data storage system from a host based on a local preset NVMe equipment simulation unit and converting the NVMe request command into an NVMe-oF request command, and the host is used for initializing the NVMe equipment simulation unit based on a preset NVMe-oF half-unloading rule in advance;
the data sending module is used for sending the NVMe-oF request command from the corresponding network transmission layer to the data storage system;
and the interrupt module is used for sending an interrupt instruction to the host after the interaction between the host and the data storage system is determined.
In some embodiments oF the application, the NVMe-orf semi-offload processing apparatus further comprises:
the data receiving module is used for receiving an NVMe-oF reply command aiming at the data processing request and sent by the data storage system, converting the NVMe-oF reply command into an NVMe-oF reply command and sending the NVMe reply command to the host; and the system is also used for receiving the NVMe-oF reply data sent by the data storage system and aiming at the data processing request, and sending the NVMe-oF reply data to the host.
In another aspect, the present application provides a hardware acceleration device, including a memory, a processor, and a computer program stored in the memory and capable oF running on the processor, where the processor implements the NVMe-orf semi-offload processing method when executing the computer program.
Another aspect oF the present application provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the NVMe-orf semi-offload processing method.
According to the NVMe-oF semi-unloading processing method provided by the application, an NVMe request command corresponding to a data processing request oF a data storage system is obtained from a host based on a local preset NVMe equipment simulation unit, and the NVMe request command is converted into an NVMe-oF request command, wherein the host carries out initialization processing on the NVMe equipment simulation unit in advance based on a preset NVMe-oF semi-unloading rule; transmitting the NVMe-oF request command from the corresponding network transmission layer to the data storage system; after the interaction between the host and the data storage system is determined, an interrupt instruction is sent to the host; by independently setting an NVMe equipment simulation unit outside a host side and enabling the host to initialize the NVMe equipment simulation unit based on a preset NVMe-oF half-offload rule, an NVMe command can be directly carried from the host side to the outside oF the host side for independent processing, a remote storage access function similar to that oF a host side local disk access can be provided for the host side, and the NVMe-oF half-offload processing process can be directly realized by the host without changing NVMe standard drive, without adding an NVMe-oF hardware accelerated use drive and the like; after the interaction between the host and the data storage system is determined, an interrupt instruction is sent to the host, so that huge performance expenditure caused by frequent interrupt processing of the host is avoided; and then can effectively improve the CUP utilization ratio oF the host side on the basis oF guaranteeing NVMe-oF data processing reliability and effectiveness, guarantee that the host side can process its core business with high efficiency.
Additional advantages, objects, and features of the application will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and drawings.
It will be appreciated by those skilled in the art that the objects and advantages that can be achieved with the present application are not limited to the above-described specific ones, and that the above and other objects that can be achieved with the present application will be more clearly understood from the following detailed description.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate and together with the description serve to explain the application. The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the application. Corresponding parts in the drawings may be exaggerated, i.e. made larger relative to other parts in an exemplary device actually manufactured according to the present application, for convenience in showing and describing some parts of the present application. In the drawings:
fig. 1 is a schematic view oF an NVMe-orf access path.
Fig. 2 is a schematic diagram oF an acceleration module requiring acceleration in an NVMe-orf access path.
Fig. 3 is a general flow chart oF an NVMe-orf semi-unloading processing method according to an embodiment oF the application.
Fig. 4 is a schematic flow chart oF an NVMe-orf semi-unloading processing method according to an embodiment oF the present application.
Fig. 5 is a schematic structural diagram oF an NVMe-oh half-unloading treatment apparatus according to another embodiment oF the application.
Fig. 6 is a schematic view oF another structure oF an NVMe-oh half-unloading treatment apparatus according to another embodiment oF the application.
Fig. 7 is an exemplary schematic diagram of a functional layer of a hardware acceleration device provided by an application example of the present application.
Fig. 8 is an exemplary schematic diagram of a functional implementation module of a hardware acceleration device provided by an application example of the present application.
Fig. 9 is an exemplary schematic diagram oF an interaction process for implementing an NVMe-orf semi-offload processing method based on a hardware acceleration device provided by an application example oF the present application.
Detailed Description
The present application will be described in further detail with reference to the following embodiments and the accompanying drawings, in order to make the objects, technical solutions and advantages of the present application more apparent. The exemplary embodiments of the present application and the descriptions thereof are used herein to explain the present application, but are not intended to limit the application.
It should be noted here that, in order to avoid obscuring the present application due to unnecessary details, only structures and/or processing steps closely related to the solution according to the present application are shown in the drawings, while other details not greatly related to the present application are omitted.
It should be emphasized that the term "comprises/comprising" when used herein is taken to specify the presence of stated features, elements, steps or components, but does not preclude the presence or addition of one or more other features, elements, steps or components.
It is also noted herein that the term "coupled" may refer to not only a direct connection, but also an indirect connection in which an intermediate is present, unless otherwise specified.
Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar components, or the same or similar steps.
The processing oF the network transport layer in NVMe-orf consumes CPU resources most, for example, TCP is one oF the most complex network protocols, complex window mechanism, retransmission mechanism, flow control, congestion control, etc. are all tasks to be implemented to accelerate the host-side NVMe-orf protocol, RDMA is a key technology oF current remote storage access, unlike widely used TCP protocol, RDMA protocol is only deployed in a data center (high-speed internet requirement), and also has great complexity, how to accelerate the processing oF the network transport layer in NVMe-orf through hardware is a key problem.
Therefore, the application considers the processing oF the network transmission layer in the NVMe-oF through hardware acceleration, and at the moment, the relation between the NVMe and the NVMe-oF is different from a standard protocol stack, and a cooperative mechanism oF software NVMe and hardware NVMe-oF is required to be designed. Wherein the software modules that need acceleration are shown in fig. 2.
The application provides an NVMe-oF semi-unloading processing method for solving the problem oF lower performance oF a network transmission layer executed by a CPU at a host side in the prior art, which comprises the following steps: the NVMe-oF protocol oF the host side is semi-unloaded by adopting a hardware acceleration mode, the performance oF remote storage access close to local disk access is provided for the host side, and the host side does not need to be changed, and standard drive is directly used.
In one or more embodiments of the application, the host or host side may be a server or client device.
Based on this, an embodiment oF the present application provides an NVMe-orf half-offload processing method that may be implemented by an NVMe-orf half-offload processing apparatus, referring to fig. 3, where the NVMe-orf half-offload processing method specifically includes the following contents:
step 100: and acquiring an NVMe request command corresponding to a data processing request oF the data storage system from a host based on a local preset NVMe equipment simulation unit, and converting the NVMe request command into an NVMe-oF request command, wherein the host performs initialization processing on the NVMe equipment simulation unit based on a preset NVMe-oF semi-unloading rule in advance.
In step 100, an NVMe device simulation unit is used to simulate an NVMe device, which may refer to an NVMe SSD device or the like.
In addition, the preset NVMe-orf semi-offload rule refers to: and initializing the NVMe equipment simulation unit by using an NVMe driver in the host, creating a corresponding management queue and an IO queue, generating a corresponding NVMe request command after the host receives a data processing request aiming at a data storage system, and updating a doorbell register in the IO queue.
The data processing request for the data storage system may be a data processing request for the data storage system, which is received by the host from a user, where the request at least includes a unique identifier oF the data storage system and a data processing type (access or storage, etc.), so that the host generates a corresponding NVMe-oh request command according to the data processing request for the data storage system. In one or more embodiments oF the present application, the NVMe-oh request command or the NVMe-oh reply command is an NVMe-oh command.
It is understood that the data processing request for the data storage system includes a data storage request or a data access request for the data storage system.
Step 200: and sending the NVMe-oF request command from the corresponding network transmission layer to the data storage system.
Step 300: and sending an interrupt instruction to the host after the interaction between the host and the data storage system is determined.
In step 300, the manner in which the interaction between the host and the data storage system is determined may be: after the NVMe-oF request command is sent to the data storage system from the corresponding network transmission layer, the data storage system does not reply any command or data within a preset period, and then the hardware acceleration device judges that the interaction between the host and the data storage system is finished currently; or after the NVMe-ob request command is sent to the data storage system from the corresponding network transport layer, the data storage system sends back an NVMe-ob reply command or NVMe-ob reply data for the data processing request in a preset period, and then the hardware acceleration device determines that the interaction between the host and the data storage system is currently completed after the reply command or reply data needs to be forwarded to the host.
As can be seen from the above description, in the NVMe-orf semi-offload processing method provided by the embodiment oF the present application, by independently setting an NVMe device simulation unit outside the host side, and making the host perform initialization processing on the NVMe device simulation unit in advance based on a preset NVMe-orf semi-offload rule, an NVMe command can be directly carried from the host side to the outside oF the host side for independent processing, a remote storage access function similar to that oF a local disk on the host side can be provided for the host side, so that the host can directly implement the NVMe-orf semi-offload processing process without changing an NVMe standard drive, without adding an NVMe-orf hardware accelerated usage drive, and the like; after the interaction between the host and the data storage system is determined, an interrupt instruction is sent to the host, so that huge performance expenditure caused by frequent interrupt processing of the host is avoided; and then can effectively improve the CUP utilization ratio oF the host side on the basis oF guaranteeing NVMe-oF data processing reliability and effectiveness, guarantee that the host side can process its core business with high efficiency.
In order to further improve the cpu utilization rate oF the host side, in the method for semi-unloading NVMe-orf processing provided by the embodiment oF the present application, referring to fig. 4, the following is specifically included between step 200 and step 300 oF the NVMe-orf processing method:
step 011: receiving an NVMe-oF reply command aiming at the data processing request, which is sent by a data storage system;
step 012: and converting the NVMe-oF reply command into an NVMe-oF reply command, and sending the NVMe reply command to the host.
As can be seen from the above description, in the NVMe-oh f semi-offload processing method provided by the embodiment oF the present application, by receiving an NVMe-oh f reply command sent by a data storage system and converting the NVMe-oh f reply command into an NVMe reply command and sending the NVMe reply command to the host, when the data storage system sends the NVMe-oh f reply command to the host, the NVMe-oh f reply command can be directly carried to the outside oF the host for independent processing, without processing by a CUP oF the host, so that on the basis oF ensuring reliability and effectiveness oF NVMe-oh f data processing, the CUP utilization rate oF the host can be further improved, and the host side can be ensured to efficiently process core services.
In order to further improve the efficiency oF the NVMe-orf semi-unloading processing, in the method for NVMe-orf semi-unloading processing provided in the embodiment oF the present application, referring to fig. 4, the following is specifically included between step 200 and step 300 oF the method for NVMe-orf semi-unloading processing:
step 021: receiving NVMe-oF reply data aiming at the data processing request and sent by a data storage system;
step 022: and sending the NVMe-oF reply data to the host.
It will be appreciated that the steps 011 and 021 may occur sequentially or alternatively depending on the actual situation.
As can be seen from the above description, in the NVMe-orf semi-offload processing method provided by the embodiment oF the present application, by receiving the NVMe-orf reply data sent by the data storage system and directly sending the NVMe-orf reply data to the host after that, when the data storage system sends the NVMe-orf reply data to the host, the NVMe-orf reply data can be directly forwarded to the host side, so as to further improve the efficiency oF NVMe-orf semi-offload processing.
In order to improve the reliability and application reliability oF the simulated NVMe device, in the method for semi-unloading NVMe-orf provided in the embodiment oF the present application, referring to fig. 4, before step 100 oF the method for semi-unloading NVMe-orf, the method specifically further includes the following contents:
step 001: constructing an NVMe device simulation unit based on local PCIe devices;
step 002: and sending a constructed message corresponding to the NVMe equipment simulation unit to the host, so that an NVMe driver in the host initializes the NVMe equipment simulation unit according to the constructed message, and creates a corresponding management queue and an IO queue, so that the host generates a corresponding NVMe request command and updates the IO queue after receiving a data processing request for a data storage system.
Specifically, simulating an NVMe SSD device requires support of the PCIe device, defines a PCIe configuration space according to PCIe register information specified by an NVMe PCIe protocol, and provides a register access interface, a data DMA interface, and an interrupt generation interface. PCIe (peripheral component interconnect express) is a high-speed serial computer expansion bus standard, which belongs to high-speed serial point-to-point dual-channel high-bandwidth transmission, and the connected equipment allocates a single-shared channel bandwidth and does not share bus bandwidth, and mainly supports the functions of active power management, error reporting, end-to-end reliability transmission, hot plug, quality of service (QOS) and the like. PCIe devices also come in a variety of sizes, ranging from PCI Express x1 to PCI Express x32, which can meet the demands of low-speed devices and high-speed devices that occur in a certain period of time in the future. The PCIe interface is a PCIe 3.0 interface with a bit rate of 8Gbps, approximately twice the bandwidth of the previous generation of products, and contains a series of important new functions such as transmitter and receiver equalization, PLL improvement, and clock data recovery for improved data transmission and data protection performance.
As can be seen from the above description, the NVMe-oh semi-offload processing method provided by the embodiment oF the present application is capable oF effectively providing support for simulating an NVMe device by constructing an NVMe device simulation unit by using a PCIe device, defining a PCIe configuration space according to PCIe register information specified by an NVMe PCIe protocol, and providing a register access interface, a data DMA interface and an interrupt generation interface, so that reliability and application reliability oF the simulated NVMe device can be effectively improved, and further, effectiveness and reliability oF an NVMe-oh semi-offload processing process can be further improved.
In order to improve the intelligentization degree and reliability oF sending the NVMe-orf request command to the data storage system, in an NVMe-orf semi-offload processing method provided by the embodiment oF the application, referring to fig. 4, step 200 oF the NVMe-orf semi-offload processing method specifically includes the following steps:
step 210: one of the plurality of network transport layers is selected as a target network transport layer.
Step 220: and based on the target network transmission layer, sending the NVMe-oF request command to the data storage system through a local MAC network outlet.
As can be seen from the above description, the NVMe-orf semi-offload processing method provided by the embodiment oF the present application can improve the intelligentization degree and reliability oF sending the NVMe-orf request command to the data storage system by selecting one oF multiple network transport layers as the target network transport layer.
In order to further improve the intelligentization degree and reliability oF sending the NVMe-orf request command to the data storage system, in the NVMe-orf semi-offload processing method provided by the embodiment oF the application, the network transmission layer includes: a TCP offload engine and an RDMA module.
Specifically, at the transmitting end, the emulated NVMe disk will receive an NVMe command, which needs to be converted into an NVMe-oh command, and then select the corresponding network transport layer (RDMA or TCP) to send out through the MAC. At the receiving end, the control command and data sent by the remote storage system are received, the received effective data are analyzed (for example, TCP is used as a transmission layer, NVMe/TCP PDU is analyzed), the data are put into a host side memory in a DMA mode, and after the DMA is completed, an interrupt is sent.
From the software aspect, the present application further provides an NVMe-oh f half-offload processing apparatus for executing all or part oF the NVMe-oh f half-offload processing method, referring to fig. 5, where the NVMe-oh f half-offload processing apparatus specifically includes the following contents:
the semi-unloading module 10 is configured to obtain, from a host, an NVMe request command corresponding to a data processing request oF the data storage system based on a locally preset NVMe device simulation unit, and convert the NVMe request command into an NVMe-oe request command, where the host performs initialization processing on the NVMe device simulation unit in advance based on a preset NVMe-oe semi-unloading rule;
a data sending module 20, configured to send the NVMe-oh request command from the corresponding network transport layer to the data storage system;
and the interrupt module 30 is used for sending an interrupt instruction to the host after the interaction between the host and the data storage system is determined.
On this basis, in order to further improve the cpu utilization rate oF the host side and ensure that the host side can efficiently process its core service, in an embodiment oF the NVMe-orf semi-offload processing apparatus oF the present application, referring to fig. 6, the apparatus may further include the following contents:
the data receiving module 01 is configured to receive an NVMe-orf reply command for the data processing request sent by the data storage system, convert the NVMe-orf reply command into an NVMe-orf reply command, and send the NVMe reply command to the host; and the system is also used for receiving the NVMe-oF reply data sent by the data storage system and aiming at the data processing request, and sending the NVMe-oF reply data to the host.
The embodiment oF the NVMe-orf half unloading processing device provided by the present application may be specifically used to execute the processing flow oF the embodiment oF the NVMe-orf half unloading processing method in the above embodiment, and the functions thereof are not described herein again, and may refer to the detailed description oF the embodiment oF the NVMe-orf half unloading processing method.
The part oF the NVMe-oh semi-offload processing device for performing the NVMe-oh semi-offload processing may be executed in a server, and in another practical application case, all operations may be completed in the client device. Specifically, the selection may be made according to the processing capability of the client device, and restrictions of the use scenario of the user. The application is not limited in this regard. If all operations are completed in the client device, the client device may further include a processor for specific processing oF the NVMe-oh semi-offload processing.
The client device may have a communication module (i.e. a communication unit) and may be connected to a remote server in a communication manner, so as to implement data transmission with the server. The server may include a server on the side of the task scheduling center, and in other implementations may include a server of an intermediate platform, such as a server of a third party server platform having a communication link with the task scheduling center server. The server may include a single computer device, a server cluster formed by a plurality of servers, or a server structure of a distributed device.
Any suitable network protocol may be used between the server and the client device, including those not yet developed on the filing date of the present application. The network protocols may include, for example, TCP/IP protocol, UDP/IP protocol, HTTP protocol, HTTPS protocol, etc. Of course, the network protocol may also include, for example, RPC protocol (Remote Procedure Call Protocol ), REST protocol (Representational State Transfer, representational state transfer protocol), etc. used above the above-described protocol.
As can be seen from the above description, in the NVMe-oe half-offload processing apparatus provided by the embodiment oF the present application, by independently setting an NVMe device simulation unit outside the host side, and making the host perform initialization processing on the NVMe device simulation unit in advance based on a preset NVMe-oe half-offload rule, an NVMe command can be directly carried from the host side to the outside oF the host side for independent processing, and a remote storage access function similar to local disk access on the host side can be provided for the host side, so that the host can directly implement an NVMe-oe half-offload processing procedure without changing an NVMe standard drive, without adding an NVMe-oe hardware accelerated usage drive, and the like; after the interaction between the host and the data storage system is determined, an interrupt instruction is sent to the host, so that huge performance expenditure caused by frequent interrupt processing of the host is avoided; and then can effectively improve the CUP utilization ratio oF the host side on the basis oF guaranteeing NVMe-oF data processing reliability and effectiveness, guarantee that the host side can process its core business with high efficiency.
The embodiment oF the application also provides a hardware acceleration device (i.e. a computer device), which may include a processor, a memory, a receiver and a transmitter, where the processor is configured to execute the NVMe-oh semi-offload processing method mentioned in the foregoing embodiment, and the processor and the memory may be connected by a bus or other manners, for example, through a bus connection. The receiver may be connected to the processor, memory, by wire or wirelessly. The computer equipment is in communication connection with the NVMe-oF semi-unloading processing device so as to receive real-time motion data from a sensor in the wireless multimedia sensor network and receive an original video sequence from the video acquisition device.
The processor may be a central processing unit (Central Processing Unit, CPU). The processor may also be any other general purpose processor, digital signal processor (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), field programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof.
The memory is used as a non-transitory computer readable storage medium and can be used for storing non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions/modules corresponding to the NVMe-orf half-offload processing method in the embodiment oF the application. The processor executes various functional applications and data processing oF the processor by running non-transitory software programs, instructions and modules stored in the memory, that is, implementing the NVMe-oh semi-offload processing method in the above method embodiment.
The memory may include a memory program area and a memory data area, wherein the memory program area may store an operating system, at least one application program required for a function; the storage data area may store data created by the processor, etc. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory may optionally include memory located remotely from the processor, the remote memory being connectable to the processor through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more modules are stored in the memory that, when executed by the processor, perform the NVMe-orf semi-offload processing method oF the embodiments.
In some embodiments of the present application, a user equipment may include a processor, a memory, and a transceiver unit, which may include a receiver and a transmitter, the processor, the memory, the receiver, and the transmitter may be connected by a bus system, the memory being configured to store computer instructions, the processor being configured to execute the computer instructions stored in the memory to control the transceiver unit to transmit and receive signals.
As an implementation manner, the functions of the receiver and the transmitter in the present application may be considered to be implemented by a transceiver circuit or a dedicated chip for transceiver, and the processor may be considered to be implemented by a dedicated processing chip, a processing circuit or a general-purpose chip.
As another implementation manner, a manner of using a general-purpose computer may be considered to implement the server provided by the embodiment of the present application. I.e. program code for implementing the functions of the processor, the receiver and the transmitter are stored in the memory, and the general purpose processor implements the functions of the processor, the receiver and the transmitter by executing the code in the memory.
The embodiment oF the application also provides a computer readable storage medium, on which a computer program is stored, which when executed by a processor, implements the steps oF the aforementioned NVMe-orf semi-offload processing method. The computer readable storage medium may be a tangible storage medium such as Random Access Memory (RAM), memory, read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disk, a removable memory disk, a CD-ROM, or any other form of storage medium known in the art.
In order to further explain the scheme, the application also provides a specific application example oF implementing the NVMe-oF semi-unloading processing method by applying the hardware acceleration device, referring to fig. 7, the functional layer oF the hardware acceleration device at least includes:
1) PCIe NVMe device emulation;
2) Semi-offload oF NVMe ofs protocol;
3) Network transport layer (TCP/RDMA);
4) MAC network egress.
Firstly, the hardware acceleration device simulates the device of NVMe SSD, provides remote storage access performance close to local disk access for the host side, and can directly use standard drive without any change on the host side. Specifically, at the transmitting end, the emulated NVMe disk will receive an NVMe command, which needs to be converted into an NVMe-oh command, and then select the corresponding network transport layer (RDMA or TCP) to send out through the MAC. At the receiving end, the control command and data sent by the remote storage system are received, the received effective data are analyzed (for example, TCP is used as a transmission layer, NVMe/TCP PDU is analyzed), the data are put into a host side memory in a DMA mode, and after the DMA is completed, an interrupt is sent.
Referring to fig. 8, the function implementation module of the hardware acceleration device includes:
1) PCIe devices;
2) NVMe register;
3)CPU；
4) A data processing unit;
5) TCP offloading engine (TCP offload engine);
6) RDMA (also referred to as RDMA Module);
7)MAC；
8) And an interrupt module.
Firstly, simulating NVMe SSD equipment, requiring support of PCIe equipment, defining a PCIe configuration space according to PCIe register information specified by NVMe PCIe protocol, and providing a register access interface, a data DMA interface and an interrupt generation interface.
The NVMe register part is responsible for simulating the NVMe register, and specific information is configured by the CPU in the acceleration unit.
The CPU in the acceleration unit is responsible for detecting whether an NVMe command exists (through a doorbell register), if so, taking the command out oF the host side, converting the NVMe command into an NVMe-oF command format, and starting a corresponding network transport layer module. Meanwhile, whether the network transmission layer module receives the latest data or not needs to be detected, when the latest data is received, the data is analyzed, the data is converted into an NVMe format, a data processing unit in hardware acceleration equipment is started, and after the execution of the data processing unit is finished, an interrupt request is sent.
The main function of the data processing unit is to carry data, receive the data carrying command issued by the CPU, start corresponding data transmission and give feedback.
The interrupt module is responsible for processing interrupt requests initiated by the CPU.
TCP offloading engine is a TCP offload engine, which is a hardware acceleration method for the network transport layer to use TCP, where the control of TCP is in the CPU, but the data is handled by the TCP offload engine.
The RDMA module is also a hardware acceleration method when NVMe-oF uses RDMA as the network transport layer.
The MAC is responsible for issuing network packets.
Referring to fig. 9, a specific application example oF implementing the NVMe-orf semi-offload processing method based on the above hardware acceleration device includes the following:
the method comprises the steps that an NVMe driver at a host side firstly initializes NVMe SSD equipment simulated by hardware, creates a management queue and an IO queue, when a real access and storage requirement exists, updates a doorbell register in the IO queue, after a CPU in a hardware acceleration device detects an NVMe command, retrieves the NVMe command through a data processing unit, converts the NVMe command into an NVMe-oF command, starts a corresponding network transmission layer, and sends the NVMe-oF command to a TCP offloading engine or RDMA module. When receiving the remote data return, the TCP offloading engine or RDMA module analyzes the message header, if the message header is an NVMe-oF command, informs the CPU, and starts the data processing unit to send to the host side by converting the NVMe-oF command into the NVMe command, if the message header is NVMe-oF data, directly starts the data processing unit to send to the host side, and waits until the data transmission is completed, then an interrupt is sent.
That is, the above application examples of the present application at least include the following 2 key points:
key point 1, simulation of NVMe equipment; the hardware acceleration device is not provided with an NVMe SSD, but in order to enable a host to use the hardware acceleration device without changing (without changing an NVMe standard drive and without adding an NVMe-oF hardware accelerated use drive), the hardware simulation oF the NVMe hard disk is realized, and the hardware simulation is realized mainly through simulating a register in an NVMe PCIe protocol;
the key point is 2, the interaction mode oF the NVMe-oF semi-unloading hardware CPU and other modules is adopted; the network transmission layer is completely realized by hardware, firstly, the simulated NVMe equipment, the hardware acceleration equipment can carry NVMe commands to a CPU oF the hardware acceleration equipment from a host side for processing, secondly, the network transmission is high in complexity, the flexibility oF the network transmission is sacrificed by the hardware realization, and the resources (LUT and BRAM in the FPGA) consumed by the hardware are huge, so that the network transmission layer is divided into a control surface and a data surface, the CPU oF the control surface needs to interact with the data transmission layer oF the data surface, finally, the NVMe data analyzed from the NVMe-oF protocol needs to be controlled by the CPU, the data is carried to the host side, and corresponding interruption is sent after the data carrying is completed.
In summary, by implementing the half-offload oF the NVMe-orf at the host, in the host perspective, the remote NVMe SSD or other storage devices (HDD, distributed storage system) can implement a manner similar to local disk access, and the network transport layer is completely handed over to the hardware acceleration device, so that consumption oF the host CPU is reduced, and the utilization rate oF the host CPU is improved.
Those of ordinary skill in the art will appreciate that the various illustrative components, systems, and methods described in connection with the embodiments disclosed herein can be implemented as hardware, software, or a combination of both. The particular implementation is hardware or software dependent on the specific application of the solution and the design constraints. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application. When implemented in hardware, it may be, for example, an electronic circuit, an Application Specific Integrated Circuit (ASIC), suitable firmware, a plug-in, a function card, or the like. When implemented in software, the elements of the application are the programs or code segments used to perform the required tasks. The program or code segments may be stored in a machine readable medium or transmitted over transmission media or communication links by a data signal carried in a carrier wave.
It should be understood that the application is not limited to the particular arrangements and instrumentality described above and shown in the drawings. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method processes of the present application are not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between steps, after appreciating the spirit of the present application.
In this disclosure, features that are described and/or illustrated with respect to one embodiment may be used in the same way or in a similar way in one or more other embodiments and/or in combination with or instead of the features of the other embodiments.
The above description is only of the preferred embodiments of the present application and is not intended to limit the present application, and various modifications and variations can be made to the embodiments of the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (9)
1. An NVMe-ofe semi-unloading processing method, comprising:
constructing an NVMe device simulation unit based on local PCIe devices;
sending a constructed message corresponding to an NVMe equipment simulation unit to a host, so that an NVMe driver in the host initializes the NVMe equipment simulation unit according to the constructed message, and creates a corresponding management queue and IO queue;
the method comprises the steps that based on an NVMe equipment simulation unit which is independently arranged outside a host, an NVMe request command corresponding to a data processing request oF a data storage system is obtained from the host, and the NVMe request command is converted into an NVMe-oF request command, wherein the host carries out initialization processing on the NVMe equipment simulation unit in advance based on a preset NVMe-oF semi-unloading rule, so that the host generates the corresponding NVMe request command after receiving the data processing request oF the data storage system, and updates a doorbell register in an IO queue, and the NVMe request command is directly carried from the host side to the outside oF the host side for independent processing;
transmitting the NVMe-oF request command from the corresponding network transmission layer to the data storage system;
and sending an interrupt instruction to the host after the interaction between the host and the data storage system is determined.
2. The NVMe-orf semi-offload processing method oF claim 1, further comprising, prior to the sending an interrupt instruction to the host after determining that interaction between the host and the data storage system is complete:
receiving an NVMe-oF reply command aiming at the data processing request, which is sent by a data storage system;
and converting the NVMe-oF reply command into an NVMe reply command, and sending the NVMe reply command to the host.
3. The NVMe-orf semi-offload processing method oF claim 1, further comprising, prior to the sending an interrupt instruction to the host after determining that interaction between the host and the data storage system is complete:
receiving NVMe-oF reply data aiming at the data processing request and sent by a data storage system;
and sending the NVMe-oF reply data to the host.
4. The NVMe-orf semi-offload processing method oF claim 1, wherein the sending the NVMe-orf request command from the corresponding network transport layer to the data storage system comprises:
selecting one of the multiple network transmission layers as a target network transmission layer;
and based on the target network transmission layer, sending the NVMe-oF request command to the data storage system through a local MAC network outlet.
5. The NVMe-orf semi-offload processing method oF claim 4, wherein the network transport layer comprises: a TCP offload engine and an RDMA module.
6. An NVMe-orf semi-offload processing apparatus, wherein the NVMe-orf semi-offload processing apparatus is configured to perform the following:
constructing an NVMe device simulation unit based on local PCIe devices;
sending a constructed message corresponding to an NVMe equipment simulation unit to a host, so that an NVMe driver in the host initializes the NVMe equipment simulation unit according to the constructed message, and creates a corresponding management queue and IO queue;
the NVMe-oF semi-unloading treatment device further comprises:
the system comprises a half-unloading module, a storage host and a storage host, wherein the half-unloading module is used for acquiring an NVMe request command corresponding to a data processing request oF a data storage system from the host based on an NVMe equipment simulation unit which is independently arranged outside the host, and converting the NVMe request command into an NVMe-oF request command, wherein the host carries out initialization processing on the NVMe equipment simulation unit in advance based on a preset NVMe-oF half-unloading rule, so that the host generates the corresponding NVMe request command after receiving the data processing request oF the data storage system and updates a doorbell register in an IO (input/output) queue so as to directly carry the NVMe request command from the host side to the outside oF the host side for independent processing;
the data sending module is used for sending the NVMe-oF request command from the corresponding network transmission layer to the data storage system;
and the interrupt module is used for sending an interrupt instruction to the host after the interaction between the host and the data storage system is determined.
7. The NVMe-orf semi-offloaded processing apparatus oF claim 6, further comprising:
the data receiving module is used for receiving an NVMe-oF reply command aiming at the data processing request and sent by the data storage system, converting the NVMe-oF reply command into an NVMe reply command and sending the NVMe reply command to the host; and the system is also used for receiving the NVMe-oF reply data sent by the data storage system and aiming at the data processing request, and sending the NVMe-oF reply data to the host.
8. A hardware acceleration device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the NVMe-orf semi-offload processing method oF any one oF claims 1-5 when executing the computer program.
9. A computer-readable storage medium, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the NVMe-orf semi-offload processing method according to any one oF claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211049777.8A CN115543474B (en) | 2022-08-30 | 2022-08-30 | NVMe-oF semi-unloading processing method and device and hardware acceleration equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211049777.8A CN115543474B (en) | 2022-08-30 | 2022-08-30 | NVMe-oF semi-unloading processing method and device and hardware acceleration equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115543474A CN115543474A (en) | 2022-12-30 |
| CN115543474B true CN115543474B (en) | 2023-10-20 |
Family
ID=84726111
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211049777.8A Active CN115543474B (en) | 2022-08-30 | 2022-08-30 | NVMe-oF semi-unloading processing method and device and hardware acceleration equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115543474B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116880759A (en) * | 2023-07-13 | 2023-10-13 | 北京大禹智芯科技有限公司 | NVMe system based on DPU and starting method thereof |
| CN116909484A (en) * | 2023-08-02 | 2023-10-20 | 中科驭数（北京）科技有限公司 | Data processing method, device, equipment and computer readable storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108829353A (en) * | 2018-06-15 | 2018-11-16 | 郑州云海信息技术有限公司 | A kind of networking storage system and method based on NVMe |
| CN113934366A (en) * | 2020-07-13 | 2022-01-14 | 三星电子株式会社 | System and method for data storage and storage device |
| CN114417373A (en) * | 2021-12-28 | 2022-04-29 | 联想(北京)有限公司 | Data access method and device for NVMe-oF user mode client |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200319812A1 (en) * | 2020-06-03 | 2020-10-08 | Intel Corporation | Intermediary for storage command transfers |
-
2022
- 2022-08-30 CN CN202211049777.8A patent/CN115543474B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108829353A (en) * | 2018-06-15 | 2018-11-16 | 郑州云海信息技术有限公司 | A kind of networking storage system and method based on NVMe |
| CN113934366A (en) * | 2020-07-13 | 2022-01-14 | 三星电子株式会社 | System and method for data storage and storage device |
| CN114417373A (en) * | 2021-12-28 | 2022-04-29 | 联想(北京)有限公司 | Data access method and device for NVMe-oF user mode client |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115543474A (en) | 2022-12-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115543474B (en) | NVMe-oF semi-unloading processing method and device and hardware acceleration equipment | |
| US11016911B2 (en) | Non-volatile memory express over fabric messages between a host and a target using a burst mode | |
| US10868767B2 (en) | Data transmission method and apparatus in optoelectronic hybrid network | |
| US10623257B1 (en) | Wireless network slice distributed ledger | |
| CN106951388B (en) | PCIe-based DMA data transmission method and system | |
| US11146362B2 (en) | Internet of things data transmission method, device and system | |
| JP2021501407A (en) | Methods, devices, and systems for offloading algorithms | |
| US10461986B2 (en) | Data transmission method, apparatus, and system | |
| WO2019200716A1 (en) | Fog computing-oriented node computing task scheduling method and device thereof | |
| US9906581B2 (en) | Information processing apparatus, control method, and storage medium | |
| US9081905B2 (en) | Low latency interconnect bus protocol | |
| EP4195760A1 (en) | Roce network congestion control method and related apparatus | |
| US10609125B2 (en) | Method and system for transmitting communication data | |
| CN103885840A (en) | FCoE protocol acceleration engine IP core based on AXI4 bus | |
| Kissel et al. | Evaluating high performance data transfer with rdma-based protocols in wide-area networks | |
| KR101717005B1 (en) | Apparatus, system and method of supporting streaming over a protocol adaptation layer (pal) | |
| WO2019062725A1 (en) | Method and device for uplink data transmission | |
| CN111404842B (en) | Data transmission method, device and computer storage medium | |
| WO2015055008A1 (en) | Storage controller chip and disk packet transmission method | |
| KR102487367B1 (en) | Synchronization method and apparatus | |
| US20180239729A1 (en) | Increasing media agnostic universal serial bus (ma usb) throughput using multiple parallel tcp connections | |
| Kundu et al. | Hardware acceleration for open radio access networks: A contemporary overview | |
| WO2019114586A1 (en) | Information transmission method and device | |
| JP6084204B2 (en) | Wireless communication apparatus, wireless communication method, and wireless communication control program | |
| CN108121587B (en) | Data acceleration method and virtual accelerator |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |