CN107346265B

CN107346265B - Method and device for realizing QoS

Info

Publication number: CN107346265B
Application number: CN201610514453.5A
Authority: CN
Inventors: 路向峰
Original assignee: Beijing Memblaze Technology Co Ltd
Current assignee: Beijing Memblaze Technology Co Ltd
Priority date: 2016-05-05
Filing date: 2016-06-30
Publication date: 2021-05-18
Anticipated expiration: 2036-06-30
Also published as: CN107346265A

Abstract

A method and apparatus for implementing QoS are provided. QoS is implemented in processing IO commands. A method of processing IO commands, comprising: in response to populating a commit queue with a first IO command, reducing a quota for a first namespace, wherein the first IO command corresponds to the first namespace; and in response to a message indicating that a second IO command is completed, obtaining a second IO command from a completion queue, and increasing a quota for a second namespace, wherein the second IO command corresponds to the second namespace, and the second IO command carries a mark indicating the quota.

Description

Method and device for realizing QoS

Technical Field

The present invention relates to generation and processing for processing NVMe commands, and in particular to providing Quality of Service (QoS) over multiple NMVe spaces.

Background

It is common for devices to exchange commands or messages through a queue. The command transmitting side inserts the command into the tail of the submission queue, the command receiving side takes out the command from the tail of the queue and processes the command, the processing result is inserted into the tail of the completion queue, and the command transmitting side takes out the command processing result from the head of the completion queue. By exchanging commands through the queue, multiple concurrent commands can be maintained directly at the sending and receiving sides, and the processing speed between commands does not have to be consistent. The NVMe protocol (see "NVM Express review 1.2", 11/3/2014, which is incorporated by reference in its entirety in this specification) defines the command and data structures (as shown in fig. 1) for exchanging data between the Host (Host) and the Device (Device). Multiple queues are maintained in the host, each queue being populated with multiple commands. The host and the device exchange data based on the queue. The host computer fills the command C into the submission queue, the equipment acquires the command C from the submission queue, processes the command C according to the content of the command C, and fills the processing result into the completion queue. The host obtains the results of the processing of command C from the completion queue to know that the processing of command C is complete.

According to the NVMe protocol, head and tail pointers of a plurality of queues are maintained at the equipment end. Both the host and the device update the head of line and the tail of line pointers. When the host sends a command to the device, the host inserts the command into the tail of the submission queue and updates the tail pointer. And the equipment terminal compares the head pointer and the tail pointer of the submitted queue, and determines the commands to be processed in the queue when the head pointer and the tail pointer are different. And after the equipment terminal takes out the command from the queue, updating the queue head pointer. The host end can determine whether the queue has space to accommodate a new command or not based on the head pointer and the tail pointer of the queue.

Queues are also used to exchange messages or commands between software components, integrated circuit components, and between software and hardware.

Namespaces (namespaces) are also defined in the NVMe protocol. A namespace of size n is a set of logical blocks with logical block addresses from 0 to n-1. Namespaces can be uniquely identified by a namespace ID (Namespace ID). The namespace can be shared between the host/NVMe controllers. The hosts can access different namespaces through a single NVMe controller. Different hosts may also access the same namespace through multiple NVMe controllers.

Disclosure of Invention

QoS needs to be provided for namespaces, then a flow restriction mechanism needs to be provided for each namespace, and a waiting queue is provided for each namespace. It has been realized that contention of memory resources between CPU cores will severely impact the performance of the host and thus tends to enforce binding between CPU cores and memory resources. In an NVMe system that provides 32 command spaces and 32 host CPU cores, 32 × 32(1024) NVMe commit queues need to be provided to meet the requirements of CPU core binding and providing current limiting for namespaces. And providing 1024 commit queues consumes a significant amount of controller resources.

According to a first aspect of the present invention, there is provided a first method for processing an IO command, including: in response to populating a commit queue with a first IO command, reducing a quota for a first namespace, wherein the first IO command corresponds to the first namespace; and in response to a message indicating that a second IO command is completed, obtaining a second IO command from a completion queue, and increasing a quota for a second namespace, wherein the second IO command corresponds to the second namespace, and the second IO command carries a mark indicating the quota.

According to the first method for processing IO commands of the first aspect of the present invention, there is provided the second method for processing IO commands of the first aspect of the present invention, further comprising: if the quota for the first namespace is insufficient, the IO command corresponding to the first namespace is inserted into the first wait queue without filling the commit queue.

According to the first or second method for processing an IO command of the first aspect of the present invention, there is provided a third method for processing an IO command of the first aspect of the present invention, further comprising: and in response to obtaining a message indicating that a third IO command is completed from a completion queue, maintaining a quota for a second space unchanged, where the third IO command corresponds to a second namespace, and the third IO command does not carry a flag indicating a quota or the third IO command carries a flag indicating that a quota is zero.

According to one of the first to third methods for processing an IO command of the first aspect of the present invention, there is provided the fourth method for processing an IO command of the first aspect of the present invention, further comprising: in response to obtaining an empty message from the completion queue, the quota for the second namespace is increased, wherein the empty message indicates the second namespace and the empty message does not indicate completion of any IO commands.

According to one of the first to fourth methods of processing an IO command of the first aspect of the present invention, there is provided a fifth method of processing an IO command of the first aspect of the present invention, further comprising: if the quota for the first namespace is sufficient, taking out a fourth IO command from the first waiting queue; the fourth IO command is populated to the commit queue and a quota for the first namespace is reduced.

According to one of the first to fifth methods for processing an IO command of the first aspect of the present invention, there is provided the sixth method for processing an IO command of the first aspect of the present invention, further comprising: if the first waiting queue is empty, taking out the IO request from the IO request queue and generating a fifth IO command; and filling the fifth IO command into a submission queue, and reducing the quota for the first namespace.

According to one of the first to sixth methods of processing an IO command of the first aspect of the present invention, there is provided the seventh method of processing an IO command of the first aspect of the present invention, wherein the first namespace shares a quota with the second namespace.

According to a second aspect of the present invention, there is provided a first method for processing an IO command according to the second aspect of the present invention, including: in response to processing the first IO request for the first namespace, allocating a first quota for the first namespace; generating a first command indicating the first quota based on the first IO request, and filling the first command to a submission queue; and in response to obtaining a message indicating completion of the second command from the completion queue, releasing a second quota for the second namespace, wherein the message indicating completion of the second command corresponds to the second namespace, and the message indicating completion of the second command indicates the second quota.

According to a second aspect of the present invention, there is provided a method of processing an IO command according to the second aspect of the present invention, wherein in response to allocating a first quota for a first namespace, a quota for the first namespace is decremented by the value of the first quota; and in response to releasing the second quota for the second namespace, incrementing the quota for the second namespace by the value of the second quota.

According to the second aspect of the present invention, there is provided a method for processing an IO command according to the third aspect of the present invention, wherein if allocation of the first quota for the first namespace fails, the first IO request is inserted into a first waiting queue for the first namespace.

According to one of the first to third methods of processing an IO command of the second aspect of the present invention, there is provided the fourth method of processing an IO command of the second aspect of the present invention, wherein if the message indicating completion of the second command indicates that the second quota is empty, the quota for the second namespace is kept unchanged.

According to one of the first to fourth methods for processing an IO command of the second aspect of the present invention, there is provided a fifth method for processing an IO command of the second aspect of the present invention, where the first command further includes a first flag, and the first flag is used to indicate that the first command carries the first quota.

According to one of the first to fifth methods for processing an IO command of the second aspect of the present invention, there is provided the sixth method for processing an IO command of the second aspect of the present invention, wherein the message indicating completion of the second command further includes a second flag, and the second flag is used to indicate that the message indicating completion of the second command carries the second quota.

According to one of the first to sixth methods of processing an IO command of the second aspect of the present invention, there is provided the seventh method of processing an IO command of the second aspect of the present invention, wherein if the message indicating completion of the second command does not include the second flag, a quota for the second namespace is kept unchanged.

According to one of the first to seventh methods of processing an IO command of the second aspect of the present invention, there is provided the eighth method of processing an IO command of the second aspect of the present invention, wherein the third quota for the second namespace is released in response to acquiring an empty message from the completion queue, wherein the empty message indicates the second namespace and the third quota, and the empty message does not indicate completion of any command.

According to one of the first to eighth methods of processing an IO command of the second aspect of the present invention, there is provided the ninth method of processing an IO command of the second aspect of the present invention, further comprising: in response to processing the second IO request for the second namespace, allocating a first quota for the second namespace; generating a third command indicating the first quota based on a second IO request, and filling the third command to a submission queue; the first quota for the first namespace is released in response to obtaining a message from the completion queue indicating completion of the first command.

According to one of the methods of processing IO commands of the second aspect of the present invention, there is provided the method of processing IO commands of the tenth aspect of the present invention, wherein the first namespace shares a quota with the second namespace.

According to a third aspect of the present invention, there is provided a method for processing an IO command according to the first aspect of the present invention, including: acquiring a first IO command from a submission queue, wherein the first IO command is used for a first namespace; accumulating a quota indicated by the first command for the first namespace; and generating a first message indicating that the second IO command is completed, setting a mark indicating a quota for the first namespace in the first message, correspondingly reducing the quota accumulated for the first namespace, and filling the second message into a completion queue.

According to the first method for processing IO commands of the third aspect of the present invention, there is provided the second method for processing IO commands of the third aspect of the present invention, further comprising: generating an empty message, setting a mark indicating quota in the empty message, filling the empty message into a completion queue, and correspondingly reducing the quota accumulated for the first name space; wherein the null message is for a first namespace and the null message does not indicate completion of any IO commands.

According to the first or second method for processing an IO command of the third aspect of the present invention, there is provided a third method for processing an IO command of the third aspect of the present invention, further comprising: and if the QoS level corresponding to the first namespace is not lower than the first threshold, generating a second message indicating that the third IO command is completed, and setting a quota indicating mark in the second message, wherein the quota indicating mark indicates that the quota is zero.

According to one of the first to third methods of processing an IO command of the third aspect of the present invention, there is provided a fourth method of processing an IO command of the third aspect of the present invention, further including: and generating a third message indicating that the fourth IO command is completed, setting a mark indicating a quota in the third message, wherein the indicated quota is larger than the quota indicated by the first command, and correspondingly reducing the quota accumulated for the first namespace.

According to a fourth aspect of the present invention, there is provided a first method for processing an IO command according to the fourth aspect of the present invention, including: acquiring a first IO command from a submission queue, wherein the first IO command is used for a first namespace; processing the first IO command and generating a first message indicating completion of the first IO command; and setting a mark indicating quota in the first message, and filling the first message into the completion queue.

The first method for processing IO commands according to the fourth aspect of the present invention provides the second method for processing IO commands according to the fourth aspect of the present invention, further comprising: generating an empty message, setting a mark indicating quota in the empty message, and filling the empty message into a completion queue; wherein the null message is for a first namespace and the null message does not indicate completion of any IO commands.

According to a fourth aspect of the present invention, there is provided a method for processing an IO command according to the third aspect of the present invention, wherein the quota-indicating flag indicates that the quota is zero.

According to one of the methods of processing IO commands of the fourth aspect of the present invention, there is provided the method of processing IO commands of the fourth aspect of the present invention, wherein if the QoS level corresponding to the first namespace is lower than the first threshold, a flag indicating a quota is set in the first message according to the quota indicated by the first command.

According to one of the methods of processing an IO command of the fourth aspect of the present invention, there is provided the method of processing an IO command of the fifth aspect of the present invention, wherein if the QoS level corresponding to the first namespace is not lower than the first threshold, a flag indicating a quota is set in the first message, the flag indicating a quota indicating that the quota is zero.

According to a fifth method for processing IO commands of the fourth aspect of the present invention, there is provided the sixth method for processing IO commands of the fourth aspect of the present invention, wherein if the QoS level corresponding to the first namespace is not lower than the first threshold, the quota indicated by the first command is also accumulated for the first namespace.

According to a fifth method for processing IO commands of the fourth aspect of the present invention, there is provided the seventh method for processing IO commands of the fourth aspect of the present invention, further comprising: generating an empty message, setting a mark indicating quota in the empty message, filling the empty message into a completion queue, and correspondingly reducing the quota accumulated for the first name space; wherein the null message is for a first namespace, and the null message does not indicate completion of any IO commands.

According to one of the first to seventh methods of processing an IO command of the fourth aspect of the present invention, there is provided the eighth method of processing an IO command of the fourth aspect of the present invention, further comprising: setting a flag indicating a quota in the first message, wherein the indicated quota is larger than the quota indicated by the first command, and correspondingly reducing the quota accumulated for the first namespace.

According to one of the first to eighth methods of processing an IO command of the fourth aspect of the present invention, there is provided the ninth method of processing an IO command of the fourth aspect of the present invention, further comprising: acquiring a second IO command, wherein the second IO command is used for the first namespace; processing the second IO command and generating a second message indicating completion of the second IO command; and judging the service quality level of the first command space, setting a mark indicating quota in the second message, and filling the first message into a completion queue.

According to a fifth aspect of the present invention, there is provided an apparatus for processing an IO command, comprising: the quota applying module is used for responding to filling of a first IO command in a submission queue and reducing a quota for a first namespace, wherein the first IO command corresponds to the first namespace; and the quota releasing module is used for responding to a message which indicates that a second IO command is completed and is obtained from the completion queue, and increasing a quota for a second namespace, wherein the second IO command corresponds to the second namespace, and the second IO command carries a mark indicating the quota.

According to a sixth aspect of the present invention, there is provided an apparatus for processing an IO command according to the sixth aspect of the present invention, including: a quota allocation module, configured to allocate a first quota for a first namespace in response to processing a first IO request for the first namespace; the command submission module is used for generating a first command indicating the first quota based on the first IO request and filling the first command into a submission queue; and the quota releasing module is used for releasing the second quota used for the second namespace in response to the message which indicates the completion of the second command and is acquired from the completion queue, wherein the message which indicates the completion of the second command corresponds to the second namespace, and the message which indicates the completion of the second command indicates the second quota.

According to a seventh aspect of the present invention, an apparatus for processing an IO command includes: the device comprises an IO command acquisition module, a first storage module and a second storage module, wherein the IO command acquisition module is used for acquiring a first IO command, and the first IO command is used for a first name space; the quota accumulation module is used for accumulating the quota indicated by the first command for the first namespace; and the message generation module is used for generating a first message indicating the completion of the second IO command, setting a mark indicating a quota for the first namespace in the first message, correspondingly reducing the quota accumulated for the first namespace, and filling the second message into the completion queue.

According to an eighth aspect of the present invention, there is provided an apparatus for processing an IO command according to the eighth aspect of the present invention, including: the IO command acquisition module is used for acquiring a first IO command from the submission queue, wherein the first IO command is used for a first name space; the IO command processing module is used for processing the first IO command and generating a first message indicating the completion of the first IO command; and the message filling module is used for setting a mark indicating quota in the first message and filling the first message into the completion queue.

Drawings

The invention, as well as a preferred mode of use and further objectives and advantages thereof, will best be understood by reference to the following detailed description of an illustrative embodiment when read in conjunction with the accompanying drawings, wherein:

FIG. 1 is a diagram illustrating a prior art method for exchanging commands between a host and a device via a queue;

FIG. 2 shows a block diagram of an NVMe system according to an embodiment of the present invention;

FIG. 3A is a schematic view of namespace flow restriction according to an embodiment of the invention;

FIG. 3B is a schematic view of namespace flow restriction according to another embodiment of the invention;

FIG. 4A is a flow diagram of processing NVMe commands according to an embodiment of the present invention; and

fig. 4B is a flow diagram of processing an indicate NVMe command complete message according to an embodiment of the invention.

Detailed Description

Fig. 2 is a diagram illustrating a command exchange between a host and a device through a queue in the prior art. In the embodiment according to fig. 2, the host and the device exchange commands through the queue according to the NVMe protocol. A controller supporting NVMe protocol is included in the device. The device may be a solid state disk supporting NVMe protocol, for example.

The host includes a plurality of applications (e.g., application 1, application 2, and application 3), issues NVMe protocol to the device through a commit queue (SQ) defined by the NVMe protocol, and receives a processing result of the NVMe command from a Completion Queue (CQ). The controller provides a plurality of namespaces. In order to provide QoS (Quality of Service) for each command space, in an embodiment according to the present invention, a namespace flow restriction mechanism is also provided. The namespace flow restriction mechanism may be implemented as a software module or as an intermediary layer. In the host, the quotas (202, 204, 206) provided for each namespace are recorded. By way of example, a quota is the number of IO commands that a host can issue to each namespace. Upon initialization, each namespace can be assigned the same quota. Or, as another example, a quota is assigned to each namespace based on different attributes of each command space.

The IO requests issued by the application are converted to NVMe commands by the namespace current limit module. The quota for the namespace accessed is carried in the NVMe command. The namespace throttling module only issues the NVMe command to access the namespace NS1 when there is a remaining quota for namespace NS1 in the host, otherwise, the NVMe command is temporarily stored in the host.

To implement namespace current limiting, a command space current limiting module examines an IO request issued by an application, identifies a command space to be accessed by the IO request, and applies for a quota for the namespace. For example, to access namespace NS1, the namespace current limit module checks the remaining values of the quota (202) for namespace NS1 recorded in the host. If the remaining value (202) of the quota of namespace NS1 is greater than 0, a fit is set in the generated NVMe command (e.g., quota value 1) and the generated NVMe command is filled into the commit queue while the quota (202) of namespace NS1 recorded in the host is decremented (e.g., minus the quota value marked in the NVMe command, to 1). If the remaining value of the quota (202) for namespace NS1 is 0, indicating that the quota for namespace NS1 has been exhausted, to achieve QoS, no NVMe commands are temporarily issued to the controller (commit queue) to access namespace NS 1. Optionally, a queue is also provided in the namespace throttling module in the host to buffer NVMe commands that temporarily cannot fill the commit queue due to throttling. It is beneficial to provide a queue (non-commit queue) in the host that caches NVMe commands. When the QoS quota of a namespace is exhausted, the controller is likely to lack resources to process more NVMe commands to access the namespace. If these NVMe commands appear in the submission queue, they may cause congestion in the controller due to inefficient processing, and may even affect the processing rate of the submission queue, and thus affect the processing of NVMe commands that access other namespaces (even if the QoS quotas for the other namespaces are still sufficient).

In an embodiment according to the invention, the current limiting is implemented per namespace. Even if the quota (202) for namespace NS1 is exhausted, and the quotas (204, 206) for namespace NS2 or NS3 remain, IO requests by applications to access namespace NS2 or NS3 will still fill the commit queue.

Upon completion of NVMe command processing, the controller populates a Completion Queue (CQ) with a message indicating completion of NVMe command processing and indicates the returned quota in the message. The namespace throttling module of the host fetches the message from the completion queue, identifies the namespace (e.g., NS1) to which the message corresponds, and identifies whether the message carries a quota value. If the quota value is carried in the message, the quota (202) of namespace NS1 is updated (e.g., by adding 1). Further, after the quota of namespace NS1 is greater than 0, a quota can be allocated for NVMe commands for namespace NS1 cached in the host and filled into the commit queue (SQ).

In an embodiment according to the invention, the host provides QoS (throttling through namespaces) to each namespace through quotas, while the controller participates in namespace throttling. The controller may check or measure the QoS provided for each namespace and when it is desired to limit the QoS of, for example, namespace NS1, the controller withholds the quota for namespace NS 1. The controller may choose to provide or not provide quotas in messages that are populated to a Completion Queue (CQ). The controller traps the coordination for namespace NS1 in the controller by not providing a quota in the completion queue message. In this case, with access to namespace NS1, the remaining value of the quota of namespace NS1 in the host is stepped down and not restored. When the NS1 quota is exhausted, the namespace current limit module no longer processes IO requests for namespace NS1, thereby enabling the current limit of namespaces.

It is noted that in the case where the controller does not implement trapping, if the processing rates of the host and controller do not match, the quota for one or more namespaces is also exhausted due to too many outstanding completed NVMe commands being cached in the commit queue (SQ), the Completion Queue (CQ), and the device (controller). In the embodiment of the invention, the controller carries out interception, so that the process of using up the namespace quota is accelerated.

In an embodiment according to the present invention, the messages that the controller populates the Completion Queue (CQ) include at least three types. (1) A quota is carried and indicates that 1 NVMe command process is complete. When the namespace current limit module acquires such messages from a Completion Queue (CQ), the quota corresponding to the namespace of the NVMe command is updated accordingly to complete the return of the quota. The quota carried by the message may be indicated by a 1 bit, flag, or value in the message. In 1 example, the quota value indicated by a 1 bit or flag is a unit value (e.g., 1). In another example, a quota value greater than 1 may be returned in the message, and the returned quota value is indicated by a value in the message. (2) (as another message) carries no quota and indicates that 1 message of NVMe command processing is complete. By issuing such a message, the controller implements a trap on the quota of the namespace. The namespace throttling module does not need to update the quota for the namespace corresponding to the NVMe command when it obtains such messages from the Completion Queue (CQ). To indicate such a message, there may be no quota flag in the message, or there may be an invalid quota flag, or there may be a quota indication, but the indicated quota value is 0. (3) (as yet another message) an empty message. The empty message does not correspond to any NVMe commands that have been submitted. The null message is actively generated by the controller. The empty message carries the quota and also indicates the namespace to which it is directed. The quota carried in the null message may be a unit quota (e.g., 1) or may be a quota value. In response to retrieving the empty message from the Completion Queue (CQ), the namespace quota limit module updates (e.g., increments) the namespace quota remaining value in the host with the quota value indicated by the empty message. Other message types are possible, for example, in a message indicating completion of processing of 1 NVMe command, the return of the namespace corresponding to the quota is indicated by an extra bit.

In the above example, the quota is an integer value (e.g., 100), and to issue 1 NVMe command, the required quota value is 1. When the NS1 quota (202) is greater than 0, meaning that the quota of namespace NS1 is sufficient, NVMe commands to access namespace NS1 are assigned a quota value and fill the commit queue (SQ). If the NS1 quota (202) is not greater than 0, meaning that the quota of namespace NS1 is insufficient, the NVMe commands to access namespace NS1 are temporarily cached to wait for the quota. As another example, the number of quotas required is different for different NVMe commands. For example, NVMe commands that access larger amounts of data require more quota. By way of example, 1 NVMe command to access 512 bytes of data requires a quota value of 1, while 1 NVMe command to access 4K bytes of data requires a quota value of 8. When the NS1 quota is 1, for NVMe commands accessing 4 kbytes of data, the quota is insufficient and only 1 NVMe command accessing 512 bytes can be issued. In addition to the length of data accessed by the NVMe command, the type of NVMe command also affects the quota it requires. For example, erase commands, Trim (Trim) commands require longer time and/or more resources than read commands, and more quota is required to execute these commands. As another example, the host may adjust the quotas required to issue 1 NVMe command, and/or adjust the quotas for the respective namespaces.

It is understood that the specific value or manifestation of a quota can vary and the invention is not limited thereto.

FIG. 3A is a schematic view of namespace flow restriction according to an embodiment of the invention. Fig. 3A illustrates the execution of multiple NVMe commands in a host and a device exchanging data via the NVMe protocol at a time. The running applications 1, 2 and 3 are included in the host. The host includes a commit queue (SQ) and a Completion Queue (CQ) according to the NVMe protocol. The host also comprises a namespace current limiting module according to the embodiment of the invention. The namespace current limit module receives IO requests from applications and decides whether to fill IO commands to a commit queue (SQ).

The controller in the device provides 3 namespaces (NS1, NS2, and NS 3). Also recorded in the host are the quotas (NS1 quota 302, NS2 quota 304, and NS3 quota 306) provided for each namespace. As shown in FIG. 3A, the quota value for current namespace NS1 is 15, the quota value for namespace NS2 is 20 and the quota value for namespace NS3 is 0. In the example of FIG. 3, the quotas for namespace NS1 and namespace NS2 are sufficient with respect to IO requests to both spaces, while the quota for namespace NS3 is insufficient to currently issue IO requests to namespace NS 3. IO command buffers (NS1 waitlist 312, NS2 waitlist 314, and NS3 waitlist 316) are also provided in the host for each namespace for temporarily storing IO commands for namespaces with insufficient quotas when the quotas for the namespaces are insufficient.

With continued reference to FIG. 3A, 5 NVMe commands that fill the commit queue one after the other are currently included in the commit queue (SQ). CMD5-NS1(1) represents the NVMe command with command ID CMD5 accessing namespace NS1 and carries a quota value of 1; CMD4-NS3(1) represents the NVMe command with command ID CMD4 accessing namespace NS3 and carries a quota value of 1; CMD3-NS1(1) represents the NVMe command with command ID CMD3 accessing namespace NS1 and carries a quota value of 1. In FIG. 3A, below the commit queue (SQ) is the head of the queue and above is the tail of the queue, from which NVMe commands are inserted into the queue and removed from the head of the queue.

The controller of the device of fig. 3A fetches the NVMe command from the head of the commit queue (SQ), processes the NVMe command, and populates a Completion Queue (CQ) with a message indicating completion of command execution. The controller measures the quality of service for each namespace and enforces a trap on the quota in the NVMe command retrieved from the commit queue (SQ). In FIG. 3A, the controller has trapped a quota of value 4 for command space NS1 and a quota of value 18 for the namespace. By way of example, the controller fetches NVMe commands CMD5-NS1(1) from the head of the commit queue (SQ). The controller determines that the quality of service provided for namespace NS1 has exceeded a desired level and decides to enforce a cutback for namespace NS 1. To implement trapping, the controller retains the quota value carried by NVMe commands CMD5-NS1(1) within the controller. Within the controller, the quota value retained for each namespace is recorded. By way of example, the quota value for namespace NS1 recorded within the controller is incremented (plus 1) by a quota value of "1" carried by NVMe command CMD5-NS1 (1).

Further, to implement the trap, when the controller fills a Completion Queue (CQ) with a message, the message is marked as not carrying a quota. By way of example, referring to fig. 3, messages CMD9-NS1(0) indicate a message indicating that processing of the NVMe command with command ID CMD9 is complete, the command space accessed by the NVMe command with command ID CMD9 is NS1, and the "(0)" in the message indicates that the message does not carry a quota (in other words, carries a quota value of 0). That is, to enforce the trap on namespace NS1, the controller traps the quota carried in the NVMe command without returning the quota to the host. Thus, the quota for namespace NS1 in the host is gradually consumed but not replenished as namespace NS1 is accessed. After the quota for namespace NS1 in the host is exhausted, the host no longer issues NVMe commands to the commit queue (SQ) to access namespace NS 1. It is noted that the NVMe command carrying the quota need not correspond to the message returning the quota. In the example of fig. 3, NVMe commands CMD5-NS1(1) carry a quota for namespace NS1, while messages CMD7-NS1(1) return a quota for namespace NS1, and messages CMD7-NS1(1) are messages for indicating completion of execution of NVMe commands with command ID CMD7, but not for indicating completion of execution of NVMe commands with command ID CMD 5.

The controller may determine that no trap on namespace NS1 is needed. In this case, the controller marks, in a message indicating completion of execution of the NVMe command to access namespace NS1, that the message carries a quota to indicate returning the quota value to the host. The quota indicates the number of NVMe commands that the controller can simultaneously process for each namespace (or, given the differences between NVMe commands, the quota corresponds to the maximum capacity the controller provides for each namespace to simultaneously process NVMe commands). In one embodiment, the controller marks a quota in a message for namespace NS1 added to a Completion Queue (CQ) and decrements the quota value recorded in the controller for namespace NS 1. In other words, the controller does not generate a quota value, but rather returns a quota value for the specified command space, which is trapped from the NVMe command, to the host.

With continued reference to fig. 3A, at the end of the Completion Queue (CQ) is a message indicated by 0- (NS1) (1), which does not correspond to any valid NVMe commands, but rather is a special message generated by the controller (referred to as an "empty message"). In the null message, namespace (NS1) is indicated along with the returned quota value (1). By filling the completion queue with empty messages, the controller actively returns the trapped quota to the host. In the example of fig. 3A, the NVMe commands in the commit queue (SQ) carry quota values of 1, while the messages in the Completion Queue (CQ) carry quota values of 1 or 0. Since the NVMe command corresponds to a message indicating that the command is complete, after the controller implements quota hold, an additional message is needed to return quota to the host. To this end, the controller generates an empty message and indicates the namespace and the returned quota value in the empty message. In another embodiment, the quota carried in the NVMe command and/or message may have different values, such that by modifying the quota value, the trapped quota may be returned to the host in the message without using an empty message.

In the example of FIG. 3A, the host provides a quota for each namespace and the controller traps quotas for each namespace. As yet another example, the controller shares or "borrows" quotas among multiple namespaces. For example, the controller withholds a quota of value 20 for namespace NS1 and a quota of value 10 for namespace NS 2. The controller may return the quota trapped for namespace NS2 to namespace NS1 as needed. Alternatively, the controller may generate an additional quota value and "borrow" the namespace (e.g., NS1), such that the host discovers that the controller is (temporarily) promoting the quality of service level for namespace NS 1. In yet another example, multiple namespaces share a quota. For example, namespace NS1 shares a quota with namespace NS 2. In the host, a single namespace quota is recorded for namespace NS1 and namespace NS 2. This single namespace quota is used whether the NVMe command to access namespace NS1 or the NVMe command to access namespace NS2 is filled into the commit queue (SQ). And the controller may or may not be aware that namespace NS1 shares a quota with namespace NS 2. With the controller aware that namespace NS1 shares quota with namespace NS2, the controller records a single quota cutoff for both namespaces and measures the overall quality of service level of both namespaces. In the case where the controller is unaware that namespace NS1 shares a quota with namespace NS2, the controller records a quota cutoff value for each of the two namespaces and measures the quality of service level for each of the two namespaces.

The controller controls whether quotas are returned for the namespace and the manner in which quotas are returned. In one example, the controller measures a quality of service level or level for each namespace. The quality of service level for a namespace can be indicated by the amount of service provided to the namespace over a period of time (the number of NVMe commands processed, or the resources used for NVMe commands (e.g., memory space, CPU time, data transfer bandwidth, energy, etc.)). Each namespace has a quality of service threshold. The controller measures the quality of service level of each namespace, does not trap quota of namespace NS1 when the quality of service level of a command space (e.g., NS1) is below a threshold, returns quota carried in NVMe command accessing namespace NS1 to the host through a message corresponding to namespace NS1, or carries quota in each message corresponding to namespace NS 1. When the quality of service level of a certain command space (e.g., NS1) is above or below a threshold, a trap is applied to the quota of namespace NS1, with no quota (or a carried quota value of 0) being carried in the message corresponding to namespace NS 1. Further, the controller returns the quota by filling the completion queue with an empty message, if necessary. For example, the controller detects that the quality of service of namespace NS1 has been below the threshold for a long time, and there is a quota value trapped in the controller for namespace NS1, and the controller returns the quota value for namespace NS1 to the host via a null message. In another example, a second threshold is provided that is lower than the previous threshold, and when the controller detects that the quality of service of namespace NS1 has fallen below the second threshold, the quota value for namespace NS1 is returned to the host via a null message. In yet another example, the host can adjust thresholds and/or second thresholds for quality of service for respective namespaces in the controller and/or policies that measure quality of service levels.

In the example of FIG. 3A, the namespace NS1 quota (15) (302) in the host indicates that the remainder of the namespace NS1 quota is 15, the NS2 quota (20) (304) indicates that the remainder of the namespace NS2 quota is 20, and the NS3 quota (0) (306) indicates that the remainder of the namespace NS3 quota is 0, i.e., its quota has been exhausted. In this case, if an access request to namespace NS3 is received again, the namespace throttling module generates an NVMe command according to the access request to namespace NS3 and puts it into the NS3 wait list (316) to temporarily store the NVMe command. If an access request to the namespace NS1 or NS2 is received, because quotas of the namespaces NS1 and NS2 are sufficient, an NVMe command is generated according to the access request, the carried quotas are marked in the NVMe command, the NVMe command is filled into a commit queue (SQ), and the namespace NS1 quotas (302) and the NS2 quotas (304) recorded in the host are correspondingly decremented.

After the quota of the namespace NS3 is sufficient, the namespace current limiting module takes out the cached NVMe command from the NS3 waiting list (316), marks the carried quota in the NVMe command, and fills the Submission Queue (SQ).

FIG. 3B is a schematic view of namespace flow restriction according to another embodiment of the invention. In contrast to the embodiment shown in fig. 3A, the embodiment of fig. 3 also includes wait queues (322, 324, and 326). In the example of fig. 3, a wait queue is provided for each namespace corresponding thereto. Wait queue 322 is used to store IO requests to namespace NS1, wait queue 324 is used to store IO requests to namespace NS2, and wait queue 326 is used to store IO requests to namespace NS 3. Application access requests to the namespace are filled into a wait queue. And the namespace current limiting module takes out the IO request from the waiting queue and processes the IO request.

The namespace current limit module may implement a plurality of scheduling policies according to the namespace quota. For example, the namespace current limit module implements round robin scheduling for each namespace. The namespace current limit module checks whether the quota of namespace NS1 is sufficient (having a quota value for processing NVMe commands that access the namespace), and when the quota is sufficient, fetches the IO request from wait queue 322, generates an NVMe command, populates the commit queue (SQ), sets the quota in the NVMe command, and subtracts the quota value set to the NVMe command (e.g., 1) from NS1 quota (302). Next, the namespace current limit module processes the IO requests of the namespaces NS2 and NS3 in turn.

In another embodiment according to the present invention, a different number of wait queues, e.g., 1 wait queue, is provided, including IO requests to access various namespaces. The namespace current limiting module takes out the IO request from the waiting queue, identifies the accessed namespace, and checks whether the quota of the accessed namespace is sufficient. When the quota is sufficient, generating an NVMe command for the IO request, filling the command into a commit queue (SQ), setting a quota value in the NVMe command, and subtracting the quota value set to the NVMe command from the quota value of the corresponding namespace recorded in the host. And when the quota value is insufficient, generating an NVMe command for the IO request, and filling the NVMe command into a waiting list corresponding to the namespace.

In yet another embodiment according to the invention, the namespace current limit module preferentially fetches NVMe commands from the namespace wait list (312, 314, or 316) and fills the commit queue (SQ) when the quota value is sufficient (as compared to the

wait queue

322, 324, or 326).

Further, the wait queues (322, 324, and 326) are also bound to the CPU core. So that for a wait queue, operations on the wait queue are processed by the particular CPU core to which it is bound.

Fig. 4A is a flow diagram of processing NVMe commands according to an embodiment of the invention. The host issues NVMe commands to the device by populating the commit queue with NVMe commands. To obtain the NVMe command, in the example of fig. 3A, the namespace throttling module of the host receives an access request from an application to the device. In the example of FIG. 3B, the namespace throttling module retrieves access requests to devices from the wait queue. By way of example, an access request to a device accesses a first namespace (e.g., NS 1). The NVMe commands are generated according to access requests to the storage device. At step 410, the namespace current limit module checks whether the quota for the first namespace is sufficient. For example, it is checked whether the NS1 quota recorded in the host (see fig. 3A, NS1 quota 302) is greater than 0. If the quota of the first namespace (NS1) is insufficient, the generated NVMe command is populated into the waiting list for the first namespace (see FIG. 3A or 3B, NS1 waiting list 312) (412). Upon detecting that the quota of namespace NS1 is sufficient, the NVMe command can be taken from NS1 wait list (312) and filled into the commit queue (SQ). At step 410, if the quota of the first namespace (NS1) is reorganized, the generated NVMe command is filled into a commit queue (SQ) (414). In one example, a flag is set in the NVMe command to indicate that a quota is carried in the NVMe command. Typically, the default value of the quota carried in the NVMe command is 1. The default value of the quota may be other values. Other values of quotas may also be carried in the NVM commands. In another example, an NVMe command to access a specified namespace carries a quota by default, without being indicated by an additional flag. At step 416, the NS1 quota recorded in the host is updated (FIG. 3A, NS1 quota 302). For example, the value of the NS1 quota 302 is reduced by an amount equivalent to the quota value carried in the NVMe command.

In response to the host populating the commit queue with NVMe commands (SQ) at step 401, the device retrieves NVMe commands from the commit queue (420). The device identifies and traps the quota carried in the NVMe command (422). In one example, when a quota flag is provided in the NVMe command, the device withholds a default value of quota from the NVMe command. In another example, a value for the quota is provided in an NVMe command from which the device withholds the quota value provided in the NVMe command. In yet another example, a specified namespace is accessed according to the NVMe command, the NVMe command is accessed to carry a quota by default, and the device withholds the default value of the quota. The trapped quota value is recorded in the device for each namespace. In response to intercepting the quota value, the intercepted quota value is increased by the intercepted quota value recorded in the device for the respective namespace. In step 424, (the controller of) the device processes the NVMe commands. The order of execution of

steps

422 and 424 is not limiting and the steps may be performed in parallel or in any order.

Fig. 4B is a flow diagram of processing an indicate NVMe command complete message according to an embodiment of the invention. The device generates a message for the NVMe command that processing is complete in step 424 (see fig. 4A) indicating to the host that the corresponding NVMe command processing is complete. In one example, the device process completes an NVMe command to access a first namespace (e.g., NS1), for which a message is generated. At step 450, the controller of the device checks whether a quality of service (QoS) level recorded in the device for the first namespace (NS1) is below a threshold. If the quality of service level for the first namespace (NS1) is below a threshold, the controller sets a quota in the generated message to indicate that the quota for the first namespace (NS1) is returned to the host via the message (452). In one example, a quota flag is set in the message to indicate a quota to return a default value to the first namespace (NS 1). In another example, a configuration value is set in the message to indicate that the set quota value is returned to the first namespace (NS 1). The controller also reduces a quota (454) trapped in the device for the first namespace (NS1) accordingly based on the returned quota value. The order of execution of

steps

452 and 454 is not limited and the two steps may be performed in parallel or in any order. At step 450, if the quality of service level for the first namespace (NS1) is not below the threshold, to indicate that the host is restricting access to the first namespace (NS1), the controller sets an invalid quota in the generated message (456) to indicate that no quota was returned to the host in the message. In one example, an invalid quota flag is set in a message to indicate that no quota is carried in the message. In another example, a quota with a value of 0 is set in a message to indicate that no quota is carried in the message.

The host retrieves the message from the completion queue (460). The host (e.g., namespace throttling module) checks the retrieved message type. If the message is a completion message (462) for the committed NVMe command, it is also checked whether the message carries a quota (463). If the message carries the quota, a namespace corresponding to the message (the namespace accessed by the NVMe command corresponding to the message) is obtained, and a quota value carried in the message is returned to the corresponding namespace (464). For example, the namespace corresponding to the message is NS3, and the quota value carried by the message is 1, then the NS3 quota recorded in the host is incremented (see fig. 3B, 306). If it is determined in step 463 that the message does not carry a quota or the carried quota value is 0, the quota corresponding to the namespace does not need to be modified.

If the message is judged to be an empty message (the message does not correspond to any valid NVMe command) (462, 464), a namespace corresponding to the empty message and a quota value carried by the message are obtained, and the quota value is returned to the corresponding namespace (466). By way of example, a field is included in the null message to indicate a namespace to which the null message corresponds.

In response to returning the quota to the namespace (e.g., NS3) at step 464, the value of the NS3 quota (see FIG. 3B, 306) recorded in the host becomes greater than 0, such that the current limit to namespace NS3 is temporarily cancelled, and the namespace current limit module populates the commit queue (SQ) with NVMe commands that access namespace S3. Referring back to FIG. 4A, at step 410, a check is made to find the quota of the first namespace (e.g., NS3) is sufficient, filling the submission queue with NVMe commands (414). The NVMe command is from the waitlist for namespace NS3 (NS 3 waitlist 316 of fig. 3A or 3B), or from an application accessing namespace NS3 (application 1, application 2, or application 3 of fig. 3A), or a waiting queue (waiting queue 322, waiting queue 324, or waiting queue 326 of fig. 3B).

One skilled in the art will appreciate that embodiments consistent with the invention may also be applied to other situations where a device having multiple namespaces is accessed based on a commit queue and a completion queue, in addition to information that is exchanged for data according to the NVMe protocol.

Embodiments of the present invention also provide a program comprising program code that, when loaded into a host CPU and executed in the CPU, causes the CPU to perform the above method performed by the host.

Embodiments of the present invention also provide a program comprising program code which, when loaded into a storage device and executed thereon, causes a processor of the storage device to perform one of the methods described above as being performed by a device.

It will be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, respectively, can be implemented by various means including computer program instructions. These computer program instructions may be loaded onto a general purpose computer, special purpose computer, or other programmable data control apparatus to produce a machine, such that the instructions which execute on the computer or other programmable data control apparatus create means for implementing the functions specified in the flowchart block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data control apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including computer-readable instructions for implementing the function specified in the flowchart block or blocks. The computer program instructions may also be loaded onto a computer or other programmable data control apparatus to cause a series of operational operations to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide operations for implementing the functions specified in the flowchart block or blocks.

Accordingly, blocks of the block diagrams and flowchart illustrations support combinations of means for performing the specified functions, combinations of operations for performing the specified functions and program instruction means for performing the specified functions. It will also be understood that each block of the block diagrams and flowchart illustrations, and combinations of blocks in the block diagrams and flowchart illustrations, can be implemented by special purpose hardware-based computer systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Although the present invention has been described with reference to examples, which are intended to be illustrative only and not to be limiting of the invention, changes, additions and/or deletions may be made to the embodiments without departing from the scope of the invention.

Many modifications and other embodiments of the inventions set forth herein will come to mind to one skilled in the art to which these embodiments pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the inventions are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

1. A method of processing IO commands, comprising:

the method comprises the steps that a host responds to a submission queue filled with a first IO command, and a quota used for a first name space is reduced, wherein the first IO command corresponds to the first name space and indicates the first quota;

the equipment acquires the first IO command from a submission queue;

the device accumulates the first quota indicated by the first command for the first namespace;

the device generates a second message indicating that the second IO command is completed, sets a flag indicating a quota for the second namespace in the second message, the second message indicating a second quota, correspondingly reduces the value of the second quota to a quota accumulated by the second namespace, and fills the second message in a completion queue, wherein the second IO command corresponds to the second namespace;

the host increases the quota for the second namespace by the value of the second quota in response to obtaining the second message from the completion queue.

2. The method of claim 1, further comprising:

if the quota for the first namespace is insufficient, the host inserts the IO command corresponding to the first namespace into the first wait queue without filling the commit queue.

3. The method according to one of claims 1-2, further comprising:

the host responds to a message which indicates that a third IO command is completed and is obtained from a completion queue, and the quota for the second space is kept unchanged, wherein the third IO command corresponds to the second namespace, and the third IO command does not carry a mark indicating the quota or the third IO command carries a mark indicating that the quota is zero.

4. The method of claim 1 or 2, further comprising:

the host increases a quota for the first namespace in response to obtaining an empty message from the completion queue, wherein the empty message indicates the first namespace and the empty message does not indicate completion of any IO commands.

5. The method of claim 1 or 2, further comprising:

if the quota for the first namespace is sufficient, the host takes out a fourth IO command from the first waiting queue;

the host populates the commit queue with the fourth IO command and reduces the quota for the first namespace.

6. The method of claim 1 or 2, further comprising:

the device generates an empty message, sets a mark indicating quota in the empty message, fills the empty message into a completion queue, and correspondingly reduces the quota accumulated for the first namespace; wherein the null message is for a first namespace and the null message does not indicate completion of any IO commands.

7. The method of claim 6, further comprising:

if the QoS level corresponding to the first namespace is not lower than the first threshold, the device generates a third message indicating that the third IO command is completed, and sets a quota-indicating flag in the third message, where the quota-indicating flag indicates that the quota is zero.

8. The method of claim 7, further comprising:

the device generates a fourth message indicating that the fourth IO command is completed, sets a flag indicating a quota in the fourth message, where the indicated quota is greater than the first quota indicated by the first command, and accordingly reduces the quota accumulated for the first namespace.

9. An apparatus to process IO commands, comprising:

the host quota applying module is used for responding to filling of a first IO command in a submission queue and reducing a quota for a first namespace, wherein the first IO command corresponds to the first namespace, and the first IO command indicates the first quota;

the equipment IO command acquisition module is used for acquiring the first IO command from the submission queue;

a device quota accumulation module, configured to accumulate the first quota indicated by the first command for the first namespace;

the device message generation module is configured to generate a second message indicating that the second IO command is completed, set a flag indicating a quota used for the second namespace in the second message, where the second message indicates a second quota, correspondingly reduce a value of the second quota to a quota accumulated in the second namespace, and fill the second message in a completion queue, where the second IO command corresponds to the second namespace; and a host quota releasing module, configured to increase a quota for the second namespace by a value of the second quota in response to obtaining the second message from the completion queue.