WO2020034819A1 - 分布式存储系统中服务质量保障方法、控制节点及系统 - Google Patents
分布式存储系统中服务质量保障方法、控制节点及系统 Download PDFInfo
- Publication number
- WO2020034819A1 WO2020034819A1 PCT/CN2019/097560 CN2019097560W WO2020034819A1 WO 2020034819 A1 WO2020034819 A1 WO 2020034819A1 CN 2019097560 W CN2019097560 W CN 2019097560W WO 2020034819 A1 WO2020034819 A1 WO 2020034819A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- unit time
- storage
- node
- requests
- storage node
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0631—Configuration or reconfiguration of storage systems by allocating resources to storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0673—Single storage device
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Definitions
- the invention relates to the technical field of distributed data storage, in particular to a method, a control node and a system for guaranteeing service quality in a distributed storage system.
- QoS Quality of service
- SLA service level agreement
- the storage system needs to perform scheduling or flow control processing before sending I / O requests to the storage nodes, so as to ensure that the service quality of I / O is different for different applications.
- the main popular metrics of QoS are upper limit flow control (limit), Lower limit reservation (reservation), proportional guarantee (share), etc.
- limit limit
- Reservation Limit
- share proportional guarantee
- QoS is to ensure the performance of the application, and to allocate the limited storage system processing power to users for reasonable allocation.
- dmClock algorithm is a fine-grained QoS control algorithm, which can support three characteristics of lower limit guarantee, upper limit flow control and proportional guarantee.
- the dmClock algorithm calculates the tag for each request in a distributed storage system, and needs to obtain the number of processing requests on other storage nodes. The communication overhead between storage nodes is large.
- Embodiments of the present invention provide a service quality guarantee method, a control node, and a system in a distributed storage system, which can reduce network communication overhead.
- an embodiment of the present invention provides a method for guaranteeing quality of service in a distributed storage system, which is applied to a control node side.
- the method includes: the control node according to the remaining input and output I / O requests of the target storage node in a unit time Number of storage resources, the number of I / O requests required to reach the lower limit guarantee in unit time, and the number of I / O requests that the storage resource needs to be processed by the target storage node within the unit time.
- the distributed storage system includes N storage nodes that provide storage resources.
- the target storage node is one of N storage nodes, where N is a positive integer.
- the control node allocates the lower limit number of tokens for the storage resource to the target storage node within the unit time according to the lower limit I / O request number that the target storage node can handle for the storage resource in the unit time.
- the control node sends the lower limit token number to the target storage node.
- the control node can allocate a storage resource token for the storage node to the storage node, so the storage node only needs to allocate according to the control node
- the tokens corresponding to I / O requests with a token are scheduled and processed, and I / O requests without a token are not processed by the storage node for the time being.
- the embodiment of the present invention only needs to focus on the target when calculating the number of tokens of the storage resource on the target storage node.
- the capacity of the storage node and the requirements of the storage resource on the target storage node are sufficient. It is not necessary to pay attention to the capabilities of other storage nodes, so the network communication overhead can be reduced. With the large scale of the storage node, the reduction of communication overhead will be more significant; It also reduces the computational complexity.
- the control node is based on the number of remaining I / O requests of the target storage node in a unit time, the number of I / O requests required for the storage resource to reach the upper limit of flow control in a unit time, and the storage resources in a unit time.
- the control node allocates storage resources on the target storage node and the maximum number of tokens in the unit time according to the maximum number of I / O requests that the target storage node can process for the storage resource in a unit time.
- the control node sends the upper limit token number to the target storage node in turn.
- control node is based on the number of input / output I / O requests remaining in the unit time of the target storage node, the number of I / O requests required for the storage resource to reach the lower limit guarantee in the unit time, and the storage resources in the unit time.
- the number of I / O requests that need to be processed by the target storage node to obtain the lower limit number of I / O requests that the target storage node can process for storage resources in a unit time including: the control node according to the maximum flow algorithm, The remaining number of input / output I / O requests within the unit, the number of I / O requests required to reach the lower limit of the storage resource in a unit time, and the number of I / O requests that the target storage node needs to process in a unit time.
- control node is based on the number of remaining I / O requests of the target storage node in a unit time, the number of I / O requests required for the storage resource to reach the upper limit of flow control in a unit time, and the storage resources in a unit time.
- the number of O requests includes: the control node calculates the first value, the first value is the difference between the second value and the third value, and the second value is the I /
- the number of O requests, the third value is the sum of the lower limit I / O requests that the N storage nodes can process for the storage resource in a unit time; the control node calculates a fourth value, and the fourth value is The difference between a fifth value and a third value, the fifth value is the number of I / O requests remaining in the target storage node in a unit time; the control node calculates a sixth value, and the sixth value is the seventh value and the third value
- the seventh value is the number of I / O requests that the storage resource needs to process by the target storage node in unit time; the control node obtains the target storage node in unit time according to the first value, the fourth value, and the sixth value.
- the maximum number of I / O requests that can be processed for storage resources is the control node calculates the first value, the first value is the difference
- control node allocates storage resources on the target storage node and the maximum number of tokens in the unit time according to the maximum number of I / O requests that the target storage node can handle for the storage resource in a unit time, including: control The node allocates the over-allocated value of the storage resource on the target storage node and within the unit time according to the following formula; the storage resource on the target storage node and within the unit time
- the eighth value is the maximum number of I / O requests that the target storage node can process for the storage resource in unit time
- the ninth value is the number of N storage nodes capable of processing in unit time.
- the sum of the upper limit I / O requests processed by the storage resource, and the tenth value is the sum of the number of I / O requests that the storage resource needs to process by the N storage nodes in a unit time;
- the control node allocates, to the storage resource, the storage resource on the target storage node and the maximum number of tokens in a unit time according to the following formula: the storage resource is on the target storage node and in a unit time
- the upper limit number of tokens is the sum of the over-allocated value of the storage resource on the target storage node within a unit time and the lower limit number of tokens of the storage resource on the target storage node within a unit time.
- control node obtains, based on the first value, the fourth value, and the sixth value, the maximum number of I / O requests that the target storage node can process for the storage resource in a unit time, include:
- the control node calculates, based on a maximum flow algorithm, the first value, the fourth value, and the sixth value, the upper limit I / O requests that the target storage node can process for the storage resource in a unit time. number.
- the above method further includes: the control node receives from the target storage node the number of I / O requests remaining in the target storage node within a unit time and the number of I / O requests that the storage resource needs to be processed by the target storage node in the unit time .
- an embodiment of the present invention provides a method for ensuring service quality in a distributed storage system, which is applied to a target storage node side.
- the distributed storage system includes N storage nodes that provide storage resources.
- the N storage nodes include a target storage node, and N is a positive integer.
- the method includes: the target storage node receives a lower limit number of tokens of the first storage resource on the target storage node and within a unit time from the control node.
- the target storage node obtains a first I / O request in the I / O request queue, and the first I / O request is an I / O request that requires the target storage node to process the first storage resource. If the lower limit token number of the first storage resource on the target storage node is greater than 0, the target storage node processes the first I / O request.
- the control node can allocate tokens for storage resources on each target storage node to each target storage node, so the target storage node only needs to schedule and process the tokens corresponding to the tokens allocated by the control node. Brand I / O requests, I / O requests without tokens are not processed by the target storage node for the time being, which can improve control accuracy, reduce network communication overhead, and computing overhead.
- the method further includes: if the lower limit token number of the first storage resource and other storage resources except the first storage resource on the target storage node are equal to zero, the target storage node obtains The second I / O request, the second I / O request is an I / O request that the second storage resource needs to be processed by the target storage node. If the upper limit token number of the second storage resource on the target storage node is greater than 0, the target storage node processes the second I / O request, and the upper limit token number is received by the target storage node from the control node.
- the target storage node receives the first storage resource from the control node on the target storage node and before the lower limit number of tokens per unit time
- the method further includes: the target storage node sends to the control node the target storage node remaining in the unit time Of I / O requests and the number of I / O requests that the first storage resource needs to be processed by the target storage node within unit time, the number of remaining I / O requests of the target storage node within the unit time, and the number of first storage resources in The number of I / O requests that need to be processed by the target storage node per unit time is used to control the number of tokens that the node allocates the first storage resource to the target storage node for the first storage resource within the unit time.
- an embodiment of the present invention provides a control node, and the control node includes a unit for implementing the foregoing first aspect or any one of the possible implementation manners described in the first aspect.
- the unit may be implemented by hardware, for example, a processor, etc., may also be implemented by hardware executing corresponding software, or may be implemented by software.
- an embodiment of the present invention provides a storage node having a unit that implements the foregoing second aspect or any one of the possible implementation manners described in the second aspect.
- the unit may be implemented by hardware, for example, a processor, etc., may also be implemented by hardware executing corresponding software, or may be implemented by software.
- an embodiment of the present invention provides a control node for performing a method of guaranteeing service quality in the distributed storage system described in the first aspect.
- the control node includes a processor and a communication interface, and the processor is in communication with the communication interface, and the processor is configured to execute the method provided by the first aspect, or any one of the possible implementation manners of the first aspect. The provided method.
- an embodiment of the present invention provides a storage node, which is configured to execute a method of guaranteeing service quality in the distributed storage system described in the second aspect.
- the control node may include a processor and a communication interface, and the processor is in communication with the communication interface, and the processor is configured to execute the method provided by the second aspect, or any one of the possible implementation manners of the second aspect. Methods provided.
- an embodiment of the present invention provides a distributed storage system including a control node and a storage node, where the control node is configured to execute the method provided in the first aspect, or any of the possible implementation manners in the first aspect. A provided method.
- an embodiment of the present invention provides a readable storage medium, where the readable storage medium stores instructions, and when the instructions stored in the readable storage medium are run on a device, the device executes the first A method for guaranteeing quality of service in a distributed storage system according to one aspect or the second aspect.
- an embodiment of the present invention provides a computer program product, and when the computer program product is run on a computer, the computer is caused to perform service quality guarantee in the distributed storage system according to the first aspect or the second aspect. method.
- FIG. 1 is a schematic architecture diagram of a distributed storage system according to an embodiment of the present invention
- FIG. 2 is a schematic diagram of a method for guaranteeing a service quality in a distributed storage system according to an embodiment of the present invention
- FIG. 3 is a schematic diagram of a computer device according to an embodiment of the present invention.
- FIG. 4 is a flowchart of a method for guaranteeing service quality in a distributed storage system according to an embodiment of the present invention
- FIG. 5 is a schematic diagram of a directed graph provided by an embodiment of the present invention.
- FIG. 6 is a structural block diagram of a distributed storage system, a control node, and a storage node according to an embodiment of the present invention.
- FIG. 1 is a schematic diagram of an architecture of a distributed storage system according to an embodiment of the present invention.
- the distributed storage system 100 includes one or more storage nodes 102 and a control node 103.
- the client may access the distributed storage system shown in FIG. 1, where the client 101 may be one or more.
- the storage node 102 may be a server.
- the control node 103 may be a server.
- the client 101 may refer to a device used by a tenant / client / user, such as a computer.
- the client 101 may apply for a storage resource to the storage node 102 for storing data.
- the storage resource may be a logical unit number (LUN) in block storage, a file system in file storage, or a bucket in an object storage system.
- the client 101 may also send an input / output (input / output (I / O)) request to the storage node 102 for writing data to or reading data from the storage resource.
- Storage nodes are used to provide storage resources in a distributed storage system.
- the storage node 102 may also send the demand information of the storage resource on the storage node 102 and the capability of the storage node 102 to the control node 103.
- the demand information may be the number of I / O requests that the storage resource needs to be processed by the storage node 102 in a unit time (for example, every second).
- the capacity may also refer to the number of I / O requests that the storage node 102 can process in a unit time based on the remaining storage resources and computing resources in a unit time.
- the control node 103 collects the demand information reported by the storage node 102 and the capabilities of the storage node 102. In addition, the control node 103 also needs to obtain the number of I / O requests that need to be completed for the storage resource to reach the lower limit in a unit time, and the number of I / O requests that the storage resource can complete in the unit time to meet the upper limit flow control. . For example, when each client requests storage resources from the storage node 102, a lower limit guarantee value and an upper limit flow control value may be set in a unit time.
- the lower guarantee value per unit time can be understood as the minimum number of I / O requests that the storage node needs to process in the unit time (for example, every second), such as Input Output per second. second, IOPS).
- the upper limit flow control value in unit time can be understood as the maximum number of I / O requests that the storage node needs to process per unit time (for example, every second).
- the number of I / O requests that need to be completed when the storage resource reaches the lower limit guarantee in unit time can also be understood as the difference between the lower limit guarantee value set by the client or the tenant and the number of I / O completed by the storage resource in unit time. .
- the number of I / O requests that need to be completed when the storage resource reaches the lower limit guarantee in unit time is also called the number of I / O requests that the storage resource needs to reach the lower limit guarantee in unit time.
- the number of I / O requests that can be completed by the storage resource within a unit of time to meet the upper limit flow control can also be understood as the upper limit guarantee value set by the client or the tenant and the I / O completed by the storage resource in the unit time.
- the gap between the numbers For example, when client 1 applies for storage resource 1, the lower guarantee value is set to 200 I / O requests per second. For example, the control node needs to calculate that storage node 1 needs to be 1 second to 2 seconds.
- the number of I / O requests processed by storage resource 1 needs to be calculated in advance.
- the control node 103 is based on the demand information per unit time reported by each storage node 102, the capacity of the storage node 102 within the unit time, the number of I / O requests required for the storage resource to reach the lower limit within the unit time, and the storage resources in the unit time.
- the number of I / O requests within the distance that meets the upper limit flow control needs to calculate the quality of service (QoS) control parameter value of the storage resource on the storage node 102 and within a unit time, the control parameter value includes the storage resource in the storage The minimum number of tokens and the maximum number of tokens on the node 102 in a unit time.
- QoS quality of service
- the control node 103 calculates the storage resource according to the demand information reported by the storage node 102 in a unit time, the capacity of the storage node 102 in a unit time, and the number of I / O requests required for the storage resource to reach the lower limit within the unit time.
- the control node 103 calculates the storage resources based on the demand information reported by the storage node 102 in a unit time, the capacity of the storage node 102 in a unit time, and the number of I / O requests for the storage resources that meet the upper limit flow control requirements in a unit time.
- the minimum number of tokens per unit time refers to the minimum number of I / O requests that the storage node 102 can handle for tenants or storage resources per unit time in order to achieve the minimum guaranteed value of the storage resources within the unit time. Therefore, the lower limit of the number of tokens per unit time is also referred to as the lower limit of the number of tokens per unit time, or the lower limit (minimum value) of the number of I / O requests processed by tenants or storage resources within the unit time.
- the maximum number of tokens per unit time refers to the maximum number of I / O requests that the storage node 102 can handle for the storage resources in a unit time in order to meet the maximum flow control value of the storage resources in a unit time.
- the upper limit of the number of tokens is also referred to as the upper limit of the number of tokens per unit time, or the upper limit (maximum value) of the number of I / O requests processed by tenants or storage resources within the unit time.
- the control node 103 distributes the control parameter value to each storage node 102 after the control parameter value is assigned. Each storage node 102 achieves the QoS guarantee target according to the received control parameter values.
- the remaining number of I / O requests per unit time of the storage node refers to the number of I / O requests that the storage node can also process / execute.
- FIG. 2 it is a schematic diagram of a method for guaranteeing quality of service in a distributed storage system according to an embodiment of the present invention.
- the client sends an I / O request to the storage node 102 for reading / writing data.
- the storage node 102 stores or reads data according to the I / O request sent by the client.
- the letter A represents the I / O request sent by the client A to the storage node 102
- the letter B represents the I / O request sent by the client B to the storage node 102.
- Each storage node 102 will queue the received I / O requests according to the received order.
- the I / O queue of storage node 1 is sorted as A0, A1, B0, A2, A3, B1, B2, A4, where the earlier I / O request represents the earlier the request received by storage node 1, the more The later I / O request represents the later the request received by storage node 1.
- A1-A4 are I / O requests sent by client A to storage node 1
- B0-B2 are I / O requests sent by client B to storage node 1.
- the I / O queues of storage node 2 are sorted as A5, B3, B4, B5, A6, A7, B6, and B7.
- A5-A7 are I / O requests sent by client A to storage node 2
- B3-B7 are I / O requests sent by client B to storage node 2.
- the I / O queues of storage node 3 are sorted as B8, B9, B10, B11, A8, A9.
- A8-A9 are I / O requests sent by client A to storage node 3
- B8-B11 are I / O requests sent by client B to storage node 3.
- Storage node 1 sends to control node 103 the number of I / O requests that client A needs to process in storage unit 1 within a unit time (for example, 1 second), and client B needs I / O requests that storage node 1 processes in a unit time.
- the remaining number of I / O requests of a storage node in a unit time refers to the number of I / O requests that the storage node can process in a unit time.
- storage node 2 sends to control node 103 the number of I / O requests that client A needs to process in storage unit 1 in a unit time, and the number of I / O requests that client B needs to process in storage unit 2 in a unit time. And the number of I / O requests remaining in storage node 2 in a unit time.
- Storage node 3 sends to control node 103 the number of I / O requests that client A needs to process in storage unit 3 in a unit of time, the number of I / O requests that client B needs to process in storage unit 3 in a unit of time, and the storage. The number of I / O requests remaining on node 3 in a unit time.
- control node 103 also needs to obtain the storage resources of the client A within a unit of time to reach the lower limit.
- the control node 103 is based on the number of I / O requests that the storage node 1 needs to process in the unit time of the client A, the number of I / O requests remaining in the unit time of the storage node 1, and the storage resources of the client A in the unit time.
- the number of I / O requests that need to be completed before the distance reaches the lower limit guarantee calculates the lower limit token number of the client A on the storage node 1 within a unit time.
- the control node 103 is based on the number of I / O requests that the storage node 1 needs to process in the unit time of the client A, the number of I / O requests that the storage node 1 can process in the unit time, and the storage resources of the client A in the unit.
- the number of I / O requests remaining when the distance reaches the upper limit in time guarantees that the maximum number of tokens for client A on storage node 1 within a unit time.
- the control node 103 according to the number of I / O requests that the storage node 1 needs to process in the unit time of the client B, the number of I / O requests remaining in the storage node 1 in the unit time, and the storage resources of the client B in The number of I / O requests that need to be completed when the distance reaches the lower limit in unit time is to calculate the lower limit token number of client B on storage node 1 and in unit time.
- the control node 103 is based on the number of I / O requests that the storage node 1 needs to process in the unit time of the client B, the number of I / O requests remaining in the unit time of the storage node 1 and the storage resources of the client B in the unit time.
- the number of I / O requests remaining after the distance reaches the upper limit guarantee calculates the upper limit token number of the client B on the storage node 1 and within a unit time.
- control node 103 calculates the lower limit token number of the client A or the client B on the storage node 2 or the storage node 3 and the control node 103 calculates the lower limit token number of the client A or the client B on the storage node 1
- the control node 103 calculates the maximum number of tokens of client A or client B on storage node 2 or storage node 3
- the control node 103 calculates the maximum number of tokens of client A or client B on storage node 1.
- the method is similar and will not be repeated here.
- the control node 103 sends the lower limit token number to the storage node 1 after calculating the lower limit token number of the client A or the client B on the storage node 1 within a unit time.
- the control node 103 sends the lower limit token number to the storage node 2 after calculating the lower limit token number of the client A or the client B on the storage node 2 within a unit time.
- the control node 103 sends the lower limit token number to the storage node 3 after calculating the lower limit token number of the client A or the client B on the storage node 3 within a unit time.
- the service quality guarantee method in the embodiment of the present invention reduces communication between storage nodes, thereby reducing communication overhead.
- control node sends the upper limit token number to the corresponding storage node after calculating the upper limit token number of the client A or the client B on the storage node within a unit time.
- the storage node On the storage node, for the same storage resource, there are a lower limit token bucket and an upper limit token bucket corresponding to the storage resource.
- the tokens in the lower token bucket are the lower tokens
- the tokens in the upper token bucket are the upper tokens.
- the scheduler of storage node 1 points the next (next) pointer to the first (that is, the first) I / O request in the I / O request queue according to the first-in-first-out (FIFO) policy (that is, the figure).
- A0 in 2 to determine whether A0 has a lower limit token.
- the judging method may be judging whether the number of tokens in the lower limit token bucket of the client A on the storage node 1 is greater than 0.
- the scheduler requests A0 to dequeue, process (or execute) the A0 request, and the scheduler subtracts 1 from the token of client A in the lower limit token bucket on storage node 1. For example, if A0 request is used to request to read data 1 of client A, the scheduler obtains data 1 of client A from the database and returns data 1 to client A. After executing the A0 request, the scheduler points the next pointer to the next request in the queue, which is A1 in Figure 2.
- the scheduler points the next pointer to the next request in the queue, which is A1 in Figure 2. Then, it is determined whether A1 has a lower limit token. The judgment method is similar to A0, and is not repeated here. After that, the scheduler points the next pointer to the next request in the queue, that is, B0 in Fig. 2, and determines whether B0 has a lower limit token. For example, the judging method may be judging whether the lower limit token number of the client B on the storage node 1 is greater than 0. If it is greater than 0, it indicates that the request of B0 has a lower limit token. If it is equal to 0, it indicates that the request of B0 is not. Has a lower limit token.
- the scheduler requests B0 to dequeue, processes (or executes) the B0 request, and the scheduler reduces the number of lower limit tokens of client B on storage node 1 by one. After executing the B0 request, the scheduler points the next pointer to the next request in the queue. If B0 does not have a lower limit token, the scheduler points the next pointer to the next request in the queue.
- the scheduler of storage node 1 redirects the next (next) pointer to the first (that is, the first) I / O request in the I / O request queue (assuming B0 in Figure 2) to determine whether B0 Has an upper limit token. For example, the judgment method can be to determine whether the number of tokens in the upper limit token bucket of client B on storage node 1 is greater than 0. If greater than 0, it indicates that the request for B0 has an upper limit token. Equal to 0, it means that the request of B0 does not have a cap token.
- the scheduler requests B0 to dequeue, processes (or executes) the B0 request, and the scheduler subtracts 1 from the token of client B in the upper limit token bucket on storage node 1. After that, the scheduler points the next pointer to the next request in the queue until it traverses the entire I / O request queue. After the scheduler has traversed the I / O request queue, all I / O requests with the upper limit token have been processed, and I / O requests without the upper limit token number will not be processed for the time being.
- the scheduler determines the number of I / O requests performed by the storage resource according to the lower limit token number and the upper limit token number of the storage resource on the storage node, thereby implementing I / O request scheduling.
- the computer device 300 includes at least one Central Processing Unit (CPU) 301, a communication bus 302, a memory 303, and at least one communication interface 304.
- the server shown in FIG. 3 also includes a storage device, such as a mechanical hard disk and / or a solid state disk.
- the CPU 301 executes computer instructions in the memory 303 for implementing the embodiments of the present invention.
- Field Programmable Gate Array (FPGA) or other hardware can also be used to perform all operations of CPU301 in the embodiment of the present invention, or FPGA or other hardware and CPU301 are used for The operations of the CPU 301 according to the embodiment of the present invention are performed.
- the CPU 301 and the memory 303 are referred to as a processor, or the FPGA and other hardware replacing the CPU 301 are referred to as a processor, or the combination of the FPGA and other hardware replacing the CPU 201 and the CPU 301 are collectively referred to as a processor.
- the processor is in communication with the communication interface 304.
- the communication interface 304 communication may be a network interface card (NIC), a host bus adapter (Host Bus Adaptor, HBA), or the like.
- NIC network interface card
- HBA host bus adapter
- the communication bus 302 may include a path for transmitting information between the aforementioned components.
- the communication interface 304 uses any device such as a transceiver to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), and wireless local area networks (WLAN). )Wait.
- RAN radio access network
- WLAN wireless local area networks
- the memory 303 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (random access memory, RAM), or other types that can store information and instructions
- the dynamic storage device can also be electrically erasable programmable read-only memory (EEPROM-ready-only memory (EEPROM)), compact disc (read-only memory (CD-ROM)) or other optical disk storage, optical disk storage (Including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store desired program code in the form of instructions or data structures and can be used by a computer Any other media accessed, but not limited to this.
- the memory may exist independently and be connected to the processor through a bus. The memory can also be integrated with the processor.
- the processor 301 takes a CPU as an example, and may include one or more CPUs, such as CPU0 and CPU1 in FIG. 3.
- the storage node 102 shown in FIG. 2 may be the device shown in FIG. 3.
- the memory of the storage node 102 stores one or more software modules (such as an interaction module and a processing module).
- the control node 103 in FIG. 2 may also be the device shown in FIG. 3.
- One or more software modules (such as an interaction module and a processing module) are stored in the memory of the control node 103.
- the storage node or the control node can implement software modules through the processor and the program code in the memory to achieve the quality of service guarantee in the distributed storage system.
- an embodiment of the present invention provides a method for guaranteeing service quality in a distributed storage system.
- FIG. 4 A flowchart of a method for guaranteeing quality of service in a distributed storage system provided by the embodiment includes the following steps.
- the number of I / O requests processed by the target storage node obtains the lower limit number of I / O requests that the target storage node can process for storage resources in a unit time; wherein the distributed storage system includes N storage nodes that provide storage resources; the target The storage node is one of N storage nodes, where N is a positive integer.
- the storage node j of the N storage nodes is used as an example for description, and other storage nodes are similar.
- the remaining processable I / O request number of the storage node j within a unit time (assuming 1 second to 2 seconds) is represented by RC j .
- the number of I / O requests required for the storage resource i to reach the lower limit guarantee within the unit time is represented by RR i .
- D ij be the number of I / O requests that storage resource i needs to process in storage unit j in this unit time.
- the number of lower-limit I / O requests that the storage node j can process for the storage resource i in this unit time is represented by f ij .
- the maximum flow algorithm, RC j , RR i , and D ij may be used to calculate the lower limit I / O requests that the storage node j can process for the storage resource i in the unit time.
- the directed graph shown in FIG. 5 is constructed, where the directed graph is composed of 4 types of nodes, where the source node and the destination node are virtual nodes, respectively representing the starting node and the ending node of the directed graph, and the resource node represents the storage resource.
- the weight of the edge from the source node to the resource node represents the corresponding QoS parameter of the resource node in unit time
- the weight of the edge of the resource node to the storage node represents the demand value of the storage resource for the storage node in unit time.
- the storage node The weight of the edge to the terminating node represents the node capacity of the storage node in unit time.
- u, v represent nodes in the constructed directed graph
- u represents the start node of an edge
- v represents the end node of an edge.
- f (u, v) represents the amount of edge traffic from node u to node v, that is, the number of tokens (or token speed) allocated.
- a breadth-first search algorithm is used to obtain a path from the source node to the terminating node.
- the source node ⁇ storage resource i ⁇ storage node j ⁇ terminating node; calculate the remaining traffic between each adjacent node on the path cf (u, v) c (u, v) -f (u, v), where c (u, v) is the weight of the edge initially set, for example, the weight of the source node to the resource node i is RRi, and the weight of the resource node i to the storage node j The weight of the edge is RCij, and the weight of the storage node j to the terminating node is Dj.
- the traffic from the edge of the resource node i to the edge of the resource node j is the lower limit number of I / O requests that the storage node j can process for the storage resource i per unit time (for example, 1 second).
- the control node allocates a lower limit number of tokens for the storage resource on the target storage node according to the number of lower limit I / O requests that the target storage node can process for the storage resource in a unit time.
- the lower limit token number of the storage resource on the target storage node in a unit time is equal to the number of I / O requests that the target storage node can process for the storage resource in the unit time.
- the minimum number of tokens of the storage resource on the target storage node in a unit time is equal to the number of I / O requests that the target storage node can process for the storage resource in a unit time plus or minus a predetermined number.
- the predetermined value may be a value preset by the control node.
- the control node sends the lower limit token number of the storage resource on the target storage node and in the unit time to the target storage node, and the target storage node receives the lower limit order of the storage resource on the target storage node and in the unit time from the control node. Number of cards.
- control node sends the lower limit token number of the storage resource 1 on the storage node 1 within a unit time to the storage node 1, and sends the lower limit token number of the storage resource 1 on the storage node 2 within the unit time to
- the storage node 2 sends the storage resource 1 on the storage node 3 and the lower limit token number in a unit time to the storage node 3, and so on.
- the storage node obtains the first I / O request in the I / O request queue.
- the first I / O request is an I / O request that the storage resource needs to process by the target storage node.
- the storage node obtains the first I / O request in the I / O request queue and determines whether the I / O request has a lower limit token. If so, execute step S405; if not, the storage node points the next pointer to the I / O request queue. The next I / O request continues to determine whether the I / O request has a lower limit token. Until all I / O requests in the I / O request queue are traversed.
- the manner in which the storage node determines whether the I / O request has a lower limit token may be: determining whether the lower limit number of storage resources corresponding to the I / O request on the storage node is greater than 0, and if it is greater than 0, it indicates that the I / O The O request has a lower limit token. If it is equal to 0, it indicates that the I / O request does not have a lower limit token.
- the storage resource corresponding to the I / O request means that the I / O request is a request that the storage resource needs to process by the storage node.
- control node may also allocate the upper limit number of tokens of the storage resource on the target storage node for the storage resource.
- the control node is based on the number of remaining I / O requests (RC j ) of the target storage node in a unit time and the number of I / O requests (RL i ) required for the storage resource to reach the upper limit of flow control in a unit time.
- the target storage node storage resources required processing per unit time I / O request number (D ij) and the destination node is stored in a unit time can be treated as storage resources / O requests and the number ( ⁇ j f ij ) Calculate the maximum number of I / O requests (g ij ) that the target storage node can process for the storage resource in a unit time.
- the control node allocates storage resources on the target storage node and the maximum number of tokens in the unit time according to the maximum number of I / O requests that the target storage node can process for the storage resource in a unit time.
- the control node sends the storage resource on the target storage node and the maximum number of tokens per unit time to the target storage node.
- the target storage node receives from the control node the upper limit token number of the storage resource on the target storage node within a unit time.
- the number of remaining I / O requests that can be processed by storage node j in a unit time is represented by RC j .
- the number of I / O requests required to reach the upper limit of flow control in a unit time from the storage resource i is represented by RL j .
- D ij be the number of I / O requests that storage resource i needs to process in storage unit j in a unit time.
- the number of lower-limit I / O requests that the storage node j can process for the storage resource i in a unit time is represented by f ij .
- the sum of the lower I / O requests that N storage nodes can handle for storage resource i in a unit time is equal to
- the number of upper-limit I / O requests that the storage node j can process for the storage resource i in a unit time is represented by g ij .
- the manner in which the control node calculates the maximum number of I / O requests that the storage node j can process for the storage resource i in a unit time may be:
- the number of upper-limit I / O requests that the storage node j can process for the storage resource i in a unit time is calculated according to the first value, the fourth value, and the sixth value.
- the method of calculating the maximum number of I / O requests that the storage node j can process for the storage resource i in a unit time according to the first value, the fourth value, and the sixth value may be: according to the maximum flow
- the algorithm, the first value, the fourth value, and the sixth value calculate the number of lower-limit I / O requests that the storage node j can process for the storage resource i in a unit time.
- control node allocates the storage resource i on the storage node j and the storage resource i in the unit time according to the maximum number of I / O requests that the storage node j can process for the storage resource i in a unit time.
- Maximum number of tokens including:
- the control node allocates an over-allocation value of the storage resource i on the storage node j and the storage resource i in a unit time according to the following formula: Among them, h ij represents an over-allocation value of the storage resource i on the storage node j within a unit time. g ij represents the maximum number of I / O requests that storage node j can process for storage resource i in a unit time. Represents the sum of the upper limit I / O requests that N storage nodes can process for storage resource i in a unit time.
- D ij indicates the number of I / O requests that the storage resource i needs to process in the storage node j in a unit time, Indicates that the storage resource i requires the sum of the number of I / O requests processed by N storage nodes in a unit time.
- the control node allocates the maximum number of tokens for storage resource i on storage node j and within unit time according to the following formula:
- the upper limit of the number of tokens of the storage resource i on the storage node j in a unit time h ij + f ij .
- f ij represents the lower limit number of I / O requests that the storage node j can process for the storage resource i in a unit time.
- the foregoing RC j and D ij may be sent by the storage node j to the control node, and the control node may collect the demand information reported by each storage node and the capability of the storage node.
- the quality of service guarantee method in a distributed storage system further includes the following steps S406 and S407.
- the first I / O request and the second I / O request may be requirements of the target storage node by two different storage resources.
- the first I / O request is a request from storage resource 1 to storage node 1
- the second I / O request is a request from storage resource 2 to storage node 1.
- the first I / O request and the second I / O request may also be requirements of the same storage resource on the target storage node.
- the target storage node After the target storage node obtains the lower limit token number and the upper limit token number of the storage resource from the control node, it executes the following I / O scheduling process according to a round-robin mechanism:
- the number of lower limit tokens of storage resource i on storage node j is reduced by 1 and next The pointer points to the next I / O request in the queue and jumps to step 1. Up to all I / O requests in the queue do not have a lower limit token.
- I / O request After the I / O request is dequeued, assuming that the I / O request is a request that storage node i needs to be processed by storage node j, reduce the maximum number of tokens for storage resource i on storage node j by one, and next The pointer points to the next I / O request in the queue and jumps to step 1. Up to all I / O requests in the queue do not have a cap token.
- the lower-limit guarantee requirements can be fulfilled preferentially, and the upper-limit flow control can be implemented in accordance with the upper-limit token bucket when the lower-limit guarantee request is satisfied.
- the allocation of the token by the control node may be triggered by the timer at a fixed time, for example, once every 5s. That is to say, the token is generated by the control node at regular intervals, and issued to each storage node.
- the control node can control the generation speed of the token, so as to achieve the QoS target of each storage resource. For example, for storage resource 1, the control node generates 1000 tokens per second for it, of which 300 tokens are sent to storage node 1, 300 tokens are sent to storage node 2, and the remaining 400 tokens are sent to Storage node 3.
- each storage node executes the I / O scheduling process according to the aforementioned round-robin mechanism.
- control node can also configure the token generation speed (for example, several lower limit tokens per second or several upper limit tokens per second), and then each storage node is responsible for generating the corresponding speed according to the generation speed configured by the control node.
- the lower or upper token For example, the control node notifies the storage node 1 to generate 300 lower limit tokens and 100 upper limit tokens for the storage resource 1 every second, and the control node notifies the storage node 2 to generate 300 lower limit tokens and 100 per second for the storage resource 1.
- Upper limit token The control node informs storage node 3 to generate 400 lower limit tokens and 100 upper limit tokens for storage resource 1 every second.
- storage node 1 generates a lower limit order for storage resource 1 at a rate of 300 tokens / second.
- the card and the speed of 100 tokens / second generate an upper limit token for the storage resource 1 and execute the I / O scheduling process according to the aforementioned round-robin mechanism.
- Storage node 2 generates a lower limit token for storage resource 1 at a rate of 300 tokens / second and a maximum limit token for storage resource 1 at a rate of 100 tokens / second, and performs I / according to the aforementioned round-robin mechanism O scheduling process.
- Storage node 3 generates a lower limit token for storage resource 1 at a rate of 400 tokens / sec.
- 100 tokens per second generates an upper limit token for storage resource 1 and performs I / O according to the aforementioned round-robin mechanism Scheduling process.
- a control node can allocate a storage resource token to each storage node. Therefore, the storage node only needs to schedule and process the I / O request with the token according to the token allocated by the control node, without the token. I / O requests are not processed by the storage node. Realized the lower guarantee performance in the distributed storage system. Further, a storage resource capping flow control mechanism can also be introduced in a distributed storage system. Different from the calculation of the tag for each request in the prior art, which needs to obtain information on the number of processing requests on other nodes, the embodiment of the present invention only needs to focus on the target when calculating the number of tokens of the storage resource on the target storage node.
- the capacity of the storage node and the requirements of the storage resource on the target storage node are sufficient. It is not necessary to focus on the capabilities of other storage nodes, so network communication overhead can be reduced.
- the embodiment of the present invention can periodically implement token update, for example, every 5s, and the prior art solution needs to synchronize every I / O.
- the storage node in the embodiment of the present invention performs I / O request scheduling, it does not need to sort the request tag values, which saves the computational cost of sorting.
- FIG. 6 shows a structural block diagram of a distributed storage system, a control node, and a storage node according to an embodiment of the present invention.
- the distributed storage system 600 includes a control node 700 and a storage node 800.
- the control node 700 may be the control node 103 in the embodiment of FIG. 1, and accordingly, the storage node 800 may be the storage node 102 in the embodiment of FIG. 1, and the distributed storage system 600 may be the distributed storage system described in FIG. 1. 100. They are described separately below.
- control node 700 may include a calculation unit 701, an allocation unit 702, and a sending unit 703.
- the calculation unit 701 is configured to calculate the number of input / output I / O requests remaining in the unit time of the target storage node in a unit time, the number of I / O requests required for the storage resource to reach the lower limit guarantee in the unit time, and the The number of I / O requests that the storage resource needs to process by the target storage node within the unit time to obtain the lower limit number of I / O requests that the target storage node can process for the storage resource within the unit time;
- the distributed storage system includes N storage nodes providing the storage resources; the target storage node is one of the N storage nodes, where N is a positive integer;
- An allocating unit 702 configured to allocate the storage resources on the target storage node according to the number of lower-limit I / O requests that the target storage node can process for the storage resource in the unit time and on the target storage node The minimum number of tokens per unit time;
- a sending unit 703 is configured to send the lower limit token number to the target storage node.
- the calculation unit 701 is further configured to calculate the number of I / O requests remaining in the unit time of the target storage node and the distance required for the storage resource to reach the upper limit flow control in the unit time.
- the number of I / O requests, the number of I / O requests that the storage resource needs to process by the target storage node in the unit time, and the target storage node can process the storage resource in the unit time
- the lower limit I / O request number to obtain the upper limit I / O request number that the target storage node can process for the storage resource in the unit time;
- the allocating unit 702 is further configured to allocate the storage resource on the target storage node according to the maximum number of I / O requests that the target storage node can process for the storage resource in the unit time.
- the sending unit 703 is further configured to sequentially send the upper limit token number to the target storage node.
- calculation unit 701 is specifically configured to:
- calculation unit 701 is specifically configured to:
- the first value is a difference between a second value and a third value
- the second value is the number of I / O requests required for the storage resource to reach the upper limit flow control within the unit time.
- the third value is the sum of the lower limit I / O requests that the N storage nodes can process for the storage resource in the unit time;
- the allocation unit 702 is specifically configured to:
- Allocate an over-allocation value of the storage resource on the target storage node and within the unit time according to the following formula: the storage resource is on the target storage node and within the unit time of
- the eighth value is the maximum number of I / O requests that the target storage node can process for the storage resource in the unit time
- the ninth value is the number of N storage nodes in the unit.
- the sum of the maximum number of I / O requests that can be processed for the storage resource per unit time, and the tenth parameter is the I / O requests that the storage resource needs to process by the N storage nodes in the unit time
- the upper limit token number of the storage resource on the target storage node and in the unit time is equal to the over-allocated value of the storage resource on the target storage node and in the unit time and the storage The sum of the lower limit tokens of the resources on the target storage node and within the unit time.
- calculation unit 701 is specifically configured to:
- the maximum number of I / O requests that the target storage node can process for the storage resource in the unit time is calculated according to the maximum flow algorithm, the first value, the fourth value, and the sixth value.
- control node 700 further includes:
- a receiving unit configured to receive, from the target storage node, the number of I / O requests remaining in the target storage node in the unit time and the storage resources that need to be processed by the target storage node in the unit time Number of I / O requests.
- control node 700 For specific implementation of each functional unit included in the control node 700, reference may be made to the foregoing embodiment, and details are not described herein again.
- the storage node 800 may include a receiving unit 801, an obtaining unit 802, and a processing unit 803.
- the receiving unit 801 is configured to receive a lower limit token number of a first storage resource on the storage node and within a unit time from a control node;
- the obtaining unit 802 is configured to obtain a first I / O request in an I / O request queue, where the first I / O request is an I / O request that the first storage resource needs to be processed by the storage node;
- the processing unit 803 is configured to process the first I / O request if the lower limit token number of the first storage resource on the storage node is greater than 0.
- the obtaining unit 802 is further configured to obtain the I if the lower limit token number of the first storage resource and other storage resources except the first storage resource on the storage node are equal to zero.
- the processing unit 803 is further configured to process the second I / O request if the upper limit token number of the second storage resource on the storage node is greater than 0, and the upper limit token number is the The storage node receives from the control node.
- the storage node 800 further includes:
- a sending unit configured to: before the receiving unit 801 receives a first storage resource from the control node on the storage node and before the lower limit number of tokens in the unit time, send the storage node the The number of I / O requests remaining in a unit time and the number of I / O requests that the first storage resource needs to be processed by the storage node in the unit time.
- the number of I / O requests and the number of I / O requests that the first storage resource needs to process by the storage node within the unit time are used by the control node to allocate the first storage resource for the first storage resource The number of tokens of storage resources on the storage node and within the unit time.
- control node 700 and the storage node 800 shown in FIG. 6 may be implemented with reference to the description in FIG. 3.
- Each unit may be a software module or hardware.
- the embodiment of the present invention does not limit this.
- a readable storage medium stores computer-executable instructions.
- a device which can be a single-chip microcomputer, a chip, or the like
- a processor can call the readable storage medium
- Computer execution instructions are stored therein to execute the steps of the control node or the storage node in the method of guaranteeing service quality in the distributed storage system provided in FIG. 4.
- the foregoing readable storage medium may include: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk, or an optical disk.
- a computer program product in another embodiment, includes computer-executable instructions stored in a computer-readable storage medium; at least one processor of the device may be The read storage medium reads the computer execution instruction, and at least one processor executes the computer execution instruction to cause the device to implement the steps of the control node or the storage node in the method of guaranteeing quality of service in the distributed storage system provided in FIG. 4.
- a distributed storage node including a control node and a storage node.
- the control node may be the control node provided in FIG. 6, and is used to execute the steps of the control node in the method of guaranteeing service quality in the distributed storage system provided in FIG. 4; and / or, the storage node may be provided in FIG. 6.
- the storage node is configured to perform the steps of the storage node in the method of guaranteeing quality of service in the distributed storage system provided in FIG. 4.
- the above embodiments it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof.
- software it may be implemented in whole or in part in the form of a computer program product.
- the computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.
- the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
- the computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be from a website site, computer, server, or data center Transmission by wire (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) to another website site, computer, server, or data center.
- the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, and the like that includes one or more available medium integration.
- the usable medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)).
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
本发明实施例公开了分布式存储系统中服务质量保障方法、控制节点及系统,用于降低网络通信开销。控制节点根据目标存储节点在单位时间内剩余的I/O请求个数、存储资源在单位时间内距离达成下限保障所需的I/O请求个数以及存储资源在单位时间内需要目标存储节点处理的I/O请求个数计算目标存储节点在单位时间内能够为存储资源处理的下限I/O请求个数,根据目标存储节点在单位时间内能够为存储资源处理的下限I/O请求个数为存储资源分配在目标存储节点上且在单位时间内的下限令牌数,将下限令牌数发送给目标存储节点。目标存储节点获取I/O请求队列中第一存储资源的第一I/O请求,若第一存储资源在目标存储节点上的下限令牌数大于0,则处理第一I/O请求。
Description
本发明涉及分布式数据存储技术领域,尤其涉及分布式存储系统中服务质量保障方法、控制节点及系统。
服务质量(quality of service,QoS),顾名思义,是为用户提供的“服务质量”,有时也称为服务等级(service level agreement,SLA)。存储系统在将I/O请求下发到存储节点前需要进行调度或流控的处理,从而保证不同应用I/O的服务质量不同,QoS现在流行的主要衡量指标有上限流控(limit)、下限保障(reservation)、比例保障(share)等。简而言之,QoS就是保障应用的性能,将有限的存储系统处理能力供用户进行合理的分配。
dmClock算法是一种细粒度的QoS控制算法,能够支持下限保障、上限流控和比例保障三种特性。然而dmClock算法为在分布式存储系统中针对每个请求的tag计算,都需要获取其他存储节点上处理请求个数,存储节点之间通信的开销大。
发明内容
本发明实施例提供了一种分布式存储系统中服务质量保障方法、控制节点及系统,能够降低网络通信开销。
第一方面,本发明实施例提供了一种分布式存储系统中服务质量保障方法,应用于控制节点侧,该方法包括:控制节点根据目标存储节点在单位时间内剩余的输入输出I/O请求个数、存储资源在单位时间内距离达成下限保障所需的I/O请求个数以及存储资源在单位时间内需要目标存储节点处理的I/O请求个数获得目标存储节点在单位时间内能够为存储资源处理的下限I/O请求个数。其中,分布式存储系统包含提供存储资源的N个存储节点。目标存储节点为N个存储节点中的一个,其中,N为正整数。控制节点根据目标存储节点在单位时间内能够为存储资源处理的下限I/O请求个数为存储资源分配在目标存储节点上且在单位时间内的下限令牌数。控制节点将下限令牌数发送给目标存储节点。
实施第一方面所描述的方法,对分布式存储系统中每一个存储节点,可以由控制节点为该存储节点分配存储资源在该存储节点上的令牌,因此该存储节点只需要按照控制节点分配的令牌相应调度并处理具有令牌的I/O请求,不具有令牌的I/O请求则该存储节点对其暂不处理。与现有技术中针对每个请求的tag计算,都需要获取其他节点上处理请求个数的信息不同,本发明实施例在计算存储资源在目标存储节点上的令牌数时,只需要关注目标存储节点的能力以及存储资源对目标存储节点的需求即可,无需关注其他存储节点的能力,因此可以降低网络通信开销,在存储节点规模较大的情况下,通信开销的降低会更加显著;同时也会降低计算的复杂度。
可选的,控制节点根据目标存储节点在单位时间内剩余的I/O请求个数、存储资源在单位时间内距离达成上限流控所需的I/O请求个数、存储资源在单位时间内需要目标存储节点处理的I/O请求个数以及目标存储节点在单位时间内能够为存储资源处理的下限I/O请 求个数获得目标存储节点在单位时间内能够为存储资源处理的上限I/O请求个数。控制节点根据目标存储节点在单位时间内能够为存储资源处理的上限I/O请求个数为存储资源分配在目标存储节点上且在单位时间内的上限令牌数。控制节点将上限令牌数依次发送给目标存储节点。
可选的,控制节点根据目标存储节点在单位时间内剩余的输入输出I/O请求个数、存储资源在单位时间内距离达成下限保障所需的I/O请求个数以及存储资源在单位时间内需要目标存储节点处理的I/O请求个数获得目标存储节点在单位时间内能够为存储资源处理的下限I/O请求个数,包括:控制节点根据最大流算法、目标存储节点在单位时间内剩余的输入输出I/O请求个数、存储资源在单位时间内距离达成下限保障所需的I/O请求个数以及需要目标存储节点在单位时间内处理的I/O请求个数计算目标存储节点在单位时间内能够为存储资源处理的下限I/O请求个数。
可选的,控制节点根据目标存储节点在单位时间内剩余的I/O请求个数、存储资源在单位时间内距离达成上限流控所需的I/O请求个数、存储资源在单位时间内需要目标存储节点处理的I/O请求个数以及目标存储节点在单位时间内能够为存储资源处理的下限I/O请求个数获得目标存储节点在单位时间内能够为存储资源处理的上限I/O请求个数,包括:控制节点计算第一数值,第一数值为第二数值与第三数值之差,第二数值为所述存储资源在单位时间内距离达成上限流控所需的I/O请求个数,所述第三数值为所述N个存储节点在单位时间内能够为所述存储资源处理的下限I/O请求个数之和;控制节点计算第四数值,第四数值为第五数值与第三数值之差,第五数值为所述目标存储节点在单位时间内剩余的I/O请求个数;控制节点计算第六数值,第六数值为第七数值与第三数值之差,第七数值为所述存储资源在单位时间内需要目标存储节点处理的I/O请求个数;控制节点根据第一数值、第四数值和第六数值获得目标存储节点在单位时间内能够为存储资源处理的上限I/O请求个数。
可选的,控制节点根据目标存储节点在单位时间内能够为存储资源处理的上限I/O请求个数为存储资源分配在目标存储节点上且在单位时间内的上限令牌数,包括:控制节点根据如下公式为存储资源分配在目标存储节点上且在单位时间内的超额分配值;存储资源在目标存储节点上且在单位时间内的
其中,所述第八数值为所述目标存储节点在单位时间内能够为所述存储资源处理的上限I/O请求个数,所述第九数值为所述N个存储节点在单位时间内能够为所述存储资源处理的上限I/O请求个数之和,所述第十数值为所述存储资源在单位时间内需要所述N个存储节点处理的I/O请求个数之和;所述控制节点根据如下公式为所述存储资源分配所述存储资源在所述目标存储节点上且在单位时间内的上限令牌数:所述存储资源在所述目标存储节点上且在单位时间内的上限令牌数为所述存储资源在所述目标存储节点上且在单位时间内的超额分配值与所述存储资源在所述目标存储节点上且在单位时间内的下限令牌数之和。
可选的,控制节点根据所述第一数值、所述第四数值和所述第六数值获得所述目标存 储节点在单位时间内能够为所述存储资源处理的上限I/O请求个数,包括:
所述控制节点根据最大流算法、所述第一数值、所述第四数值和所述第六数值计算所述目标存储节点在单位时间内能够为所述存储资源处理的上限I/O请求个数。
可选的,上述方法还包括:控制节点从目标存储节点接收目标存储节点在单位时间内剩余的I/O请求个数以及存储资源在单位时间内需要目标存储节点处理的I/O请求个数。
第二方面,本发明实施例提供了一种分布式存储系统中服务质量保障方法,应用于目标存储节点侧,分布式存储系统包含提供存储资源的N个存储节点。N个存储节点包括目标存储节点,N为正整数,该方法包括:目标存储节点从控制节点接收第一存储资源在目标存储节点上且在单位时间内的下限令牌数。目标存储节点获取I/O请求队列中的第一I/O请求,第一I/O请求为第一存储资源需要目标存储节点处理的I/O请求。若第一存储资源在目标存储节点上的下限令牌数大于0,则目标存储节点处理第一I/O请求。
实施第二方面所描述的方法,可以由控制节点为各个目标存储节点分配存储资源在各个目标存储节点上的令牌,因此目标存储节点只需要按照控制节点分配的令牌相应调度并处理具有令牌的I/O请求,不具有令牌的I/O请求则目标存储节点对其暂不处理,能够提高控制精度、降低网络通信开销以及计算开销。
可选的,上述方法还包括:若第一存储资源以及除第一存储资源以外的其他存储资源在目标存储节点上的下限令牌数均等于零,则目标存储节点获取I/O请求队列中的第二I/O请求,第二I/O请求为第二存储资源需要目标存储节点处理的I/O请求。若第二存储资源在目标存储节点上的上限令牌数大于0,则目标存储节点处理第二I/O请求,上限令牌数为目标存储节点从控制节点接收的。
可选的,目标存储节点从控制节点接收第一存储资源在目标存储节点上且在单位时间内的下限令牌数之前,还包括:目标存储节点向控制节点发送目标存储节点在单位时间内剩余的I/O请求个数以及第一存储资源在单位时间内需要目标存储节点处理的I/O请求个数,目标存储节点在单位时间内剩余的I/O请求个数以及第一存储资源在单位时间内需要目标存储节点处理的I/O请求个数用于控制节点为第一存储资源分配第一存储资源在目标存储节点上且在单位时间内的令牌数。
第三方面,本发明实施例提供了一种控制节点,该控制节点包括用于实现上述第一方面或第一方面任一种可能的实施方式描述的单元。单元可以通过硬件实现,例如,处理器等,也可以通过硬件执行相应的软件实现,也可以由软件实现。
第四方面,本发明实施例提供了一种存储节点,具有实现上述第二方面或第二方面任一种可能的实施方式描述的单元。单元可以通过硬件实现,例如,处理器等,也可以通过硬件执行相应的软件实现,也可以由软件实现。
第五方面,本发明实施例提供了一种控制节点,用于执行第一方面描述的分布式存储系统中服务质量保障方法。所述控制节点包括:处理器和通信接口,所述处理器和所述通信接口通信,所述处理器用于执行第一方面所提供的方法,或者第一方面可能的实施方式中的任意一种所提供的方法。
第六方面,本发明实施例提供了一种存储节点,用于执行第二方面描述的分布式存储系统中服务质量保障方法。所述控制节点可包括:处理器和通信接口,所述处理器和所述 通信接口通信,所述处理器用于执行第二方面所提供的方法,或者第二方面可能的实施方式中的任意一种所提供的方法。
第七方面,本发明实施例提供了一种分布式存储系统,包括控制节点和存储节点,其中,控制节点用于执行第一方面所提供的方法,或者第一方面可能的实施方式中的任意一种所提供的方法。
第八方面,本发明实施例提供一种可读存储介质,所述可读存储介质中存储有指令,当所述可读存储介质中存储的指令在设备上运行时,使得所述设备执行第一方面或第二方面所述的分布式存储系统中服务质量保障方法。
第九方面,本发明实施例提供一种计算机程序产品,当所述计算机程序产品在计算机上运行时,使得所述计算机执行第一方面或第二方面所述的分布式存储系统中服务质量保障方法。
为了更清楚地说明本发明实施例或背景技术中的技术方案,下面将对本发明实施例或背景技术中所需要使用的附图进行说明。
图1是本发明实施例提供的一种分布式存储系统的架构示意图;
图2是本发明实施例提供的分布式存储系统中服务质量保障方法的原理示意图;
图3是本发明实施例提供的计算机设备示意图;
图4是本发明实施例提供的一种分布式存储系统中服务质量保障方法的流程图;
图5是本发明实施例提供的有向图的示意图;
图6是本发明实施例提供的一种分布式存储系统、控制节点及存储节点的结构框图。
下面结合本发明实施例中的附图对本发明实施例进行描述。
请参见图1,图1是本发明实施例提供的一种分布式存储系统的架构示意图,该分布式存储系统100包括一个或多个存储节点102以及控制节点103。客户端可以访问图1所示的分布式存储系统,其中,客户端101可以为一个或多个。
存储节点102可以为服务器。控制节点103也可以为服务器。客户端101可以是指租户/客户/用户使用的设备,例如计算机。客户端101可以向存储节点102申请存储资源,用于存储数据。存储资源可以为块存储中的逻辑单元号(logical unit number,LUN)、文件存储中的文件系统等或对象存储系统中的桶等。客户端101还可以向存储节点102发送输入/输出(input output,I/O)请求,用于向存储资源写入数据或从存储资源读取数据。存储节点用于提供分布式存储系统中的存储资源。
存储节点102还可以向控制节点103发送存储资源在存储节点102上的需求信息以及存储节点102的能力。这里,需求信息可以是存储资源在单位时间内(例如每秒钟)需要存储节点102处理的I/O请求个数。能力还可以是指存储节点102基于单位时间内剩余存储资源、计算资源等在单位时间内还能处理的I/O请求个数。
控制节点103收集存储节点102上报的需求信息以及存储节点102的能力。此外,控 制节点103还需要获取存储资源在单位时间内距离达到下限保障还需完成的I/O请求个数以及存储资源在单位时间内距离满足上限流控还能完成的I/O请求个数。例如,每个客户端在向存储节点102申请存储资源时,可以设置单位时间内的下限保障值和上限流控值。本发明实施例中,单位时间内的下限保障值可以理解为存储资源在单位时间内(例如每秒钟)需要存储节点处理的I/O请求最小个数,如每秒钟输入输出(Inputoutput per second,IOPS)的最小值。相应的,单位时间内的上限流控值可以理解为存储资源单位时间内(例如每秒钟)需要存储节点处理的I/O请求最大个数。存储资源在单位时间内距离达到下限保障还需完成的I/O请求个数还可以理解为客户端或租户设置的下限保障值与单位时间内存储资源已经完成的I/O数量之间的差距。存储资源在单位时间内距离达到下限保障还需完成的I/O请求个数也称为存储资源在单位时间内距离达成下限保障所需的I/O请求个数。同理,存储资源在单位时间内距离满足上限流控还能完成的I/O请求个数还可以理解为客户端或租户设置的上限保障值与在单位时间内存储资源已经完成的I/O数量之间的差距。例如,客户端1申请存储资源1时,设置的下限保障值为每秒钟200个I/O请求,例如控制节点需要计算存储节点1在第1秒到第2秒这1秒时间内需要为存储资源1处理的I/O请求个数,首先需要预先计算存储资源1在当前时刻距离达成下限保障所需的I/O请求个数,假设当前时刻为第1.5秒,控制节点统计出其他存储节点,假设为存储节点2和存储节点3已经为存储资源1处理了100个I/O请求,则控制节点计算出存储资源1在第1-第2秒之间距离达到下限保障还需完成的I/O请求个数等于200-100=100。又例如,以5秒为例,假设客户端1在申请存储资源1时,设置的下限保障值为每秒钟200个I/O请求,则在5秒钟内需要最少为存储资源1处理200*5=1000个I/O请求。例如控制节点需要计算存储节点1在第2秒到第3秒这1秒时间内需要为存储资源1处理的I/O请求个数,首先需要预先计算存储资源1在当前时刻距离达成下限保障所需的I/O请求个数,假设当前时刻为第2秒,控制节点统计出其他存储节点,假设为存储节点2和存储节点3已经为存储资源1处理了300个I/O请求,则控制节点计算出存储资源1在第2-第3秒之间距离达到下限保障还需完成的I/O请求个数等于1000-300=700。
控制节点103根据各个存储节点102上报的单位时间内的需求信息、存储节点102单位时间内的能力、存储资源在单位时间内距离达到下限保障需要的I/O请求个数以及存储资源在单位时间内距离满足上限流控需要的I/O请求个数计算存储资源在存储节点102上且在单位时间内的服务质量(quality of service,QoS)控制参数值,该控制参数值包括存储资源在存储节点102上且在单位时间内的下限令牌数以及上限令牌数。例如,控制节点103根据存储节点102上报的在单位时间内的需求信息、存储节点102在单位时间内的能力以及存储资源在单位时间内距离达到下限保障需要的I/O请求个数计算存储资源在存储节点102上且在单位时间内的下限令牌数。控制节点103根据存储节点102上报的在单位时间内的需求信息、存储节点102在单位时间内的能力以及存储资源在单位时间内距离满足上限流控需要的I/O请求个数计算存储资源在存储节点102上且在单位时间内的上限令牌数。其中,单位时间内的下限令牌数是指:为实现存储资源在单位时间内达到下限保障值,存储节点102在单位时间内最少还能为租户或存储资源处理的I/O请求个数,因此单位时间内的下限令牌数也称为在单位时间内令牌数的下限,或者在单位时间内租户或存储 资源处理的I/O请求个数的下限(最小值)。单位时间内的上限令牌数是指:为满足存储资源在单位时间内达到上限流控值,存储节点102在单位时间内最多还能为存储资源处理的I/O请求个数,因此单位时间内的上限令牌数也称为在单位时间内令牌数的上限,或者在单位时间内租户或存储资源处理的I/O请求个数的上限(最大值)。控制节点103在完成控制参数值的分配后,将控制参数值分发给各个存储节点102。每个存储节点102根据接收到的控制参数值,达成QoS保障目标。
需要说明的是,存储节点单位时间内剩余的I/O请求个数是指存储节点还能够处理/执行的I/O请求个数。
例如,参见图2所示,是本发明实施例提供的分布式存储系统中服务质量保障方法的原理示意图。如图2所示,客户端向存储节点102发送I/O请求,用于读/写数据。存储节点102根据客户端发送的I/O请求存储数据或读取数据。图2中,字母A代表客户端A向存储节点102发送的I/O请求,字母B代表客户端B向存储节点102发送的I/O请求。每个存储节点102会将接收到的I/O请求按照接收的先后顺序进行队列排序。例如,存储节点1的I/O队列排序为A0、A1、B0、A2、A3、B1、B2、A4,其中,越靠前的I/O请求代表存储节点1越早接收到的请求,越靠后的I/O请求代表存储节点1越晚接收到的请求。A1-A4是客户端A向存储节点1发送的I/O请求,B0-B2是客户端B向存储节点1发送的I/O请求。同理,存储节点2的I/O队列排序为A5、B3、B4、B5、A6、A7、B6、B7。A5-A7是客户端A向存储节点2发送的I/O请求,B3-B7是客户端B向存储节点2发送的I/O请求。存储节点3的I/O队列排序为B8、B9、B10、B11、A8、A9。A8-A9是客户端A向存储节点3发送的I/O请求,B8-B11是客户端B向存储节点3发送的I/O请求。
存储节点1会向控制节点103发送客户端A在单位时间内(例如1秒钟)需要存储节点1处理的I/O请求个数、客户端B在单位时间内需要存储节点1处理的I/O请求个数以及存储节点1在单位时间内剩余的I/O请求个数。存储节点在单位时间内剩余的I/O请求个数是指存储节点在单位时间内还能处理的I/O请求个数。同样的,存储节点2会向控制节点103发送客户端A在单位时间内需要存储节点1处理的I/O请求个数、客户端B在单位时间内需要存储节点2处理的I/O请求个数以及存储节点2在单位时间内剩余的I/O请求个数。存储节点3会向控制节点103发送客户端A在单位时间内需要存储节点3处理的I/O请求个数、客户端B在单位时间内需要存储节点3处理的I/O请求个数以及存储节点3在单位时间内剩余的I/O请求个数。此外,控制节点103还需要获取客户端A的存储资源在单位时间内距离达到下限保障还需完成的I/O请求个数、客户端A的存储资源在单位时间内距离满足上限流控还能完成的I/O请求个数、客户端B的存储资源在单位时间内距离达到下限保障还需完成的I/O请求个数以及客户端B的存储资源在单位时间内距离满足上限流控还能完成的I/O请求个数。控制节点103根据客户端A在单位时间内需要存储节点1处理的I/O请求个数、存储节点1在单位时间内剩余的I/O请求个数以及客户端A的存储资源在单位时间内距离达到下限保障还需完成的I/O请求个数计算客户端A在存储节点1上且在单位时间内的下限令牌数。控制节点103根据客户端A在单位时间内需要存储节点1处理的I/O请求个数、存储节点1在单位时间内还能处理的I/O请求个数以及客户端A的存储资源在单位时间内距离达到上限保障剩余的I/O请求个数计算客户端A在存储节点1 上且在单位时间内的上限令牌数。相应的,控制节点103根据客户端B在单位时间内需要存储节点1处理的I/O请求个数、存储节点1在单位时间内剩余的I/O请求个数以及客户端B的存储资源在单位时间内距离达到下限保障还需完成的I/O请求个数计算客户端B在存储节点1上且在单位时间内的下限令牌数。控制节点103根据客户端B在单位时间内需要存储节点1处理的I/O请求个数、存储节点1在单位时间内剩余的I/O请求个数以及客户端B的存储资源在单位时间内距离达到上限保障剩余的I/O请求个数计算客户端B在存储节点1上且在单位时间内的上限令牌数。控制节点103计算客户端A或客户端B在存储节点2或存储节点3上的下限令牌数的方式与控制节点103计算客户端A或客户端B在存储节点1上的下限令牌数的方式类似,控制节点103计算客户端A或客户端B在存储节点2或存储节点3上限令牌数的方式与控制节点103计算客户端A或客户端B在存储节点1上的上限令牌数的方式类似,此次不再赘述。
控制节点103在计算出客户端A或客户端B在存储节点1上且在单位时间内的下限令牌数后,将下限令牌数发送给存储节点1。控制节点103在计算出客户端A或客户端B在存储节点2上且在单位时间内的下限令牌数后,将下限令牌数发送给存储节点2。控制节点103在计算出客户端A或客户端B在存储节点3上且在单位时间内的下限令牌数后,将下限令牌数发送给存储节点3。本发明实施例服务质量保障方法,减少了存储节点之间的通信,从而降低了通信开销。
同理,控制节点在计算出客户端A或客户端B在存储节点上且在单位时间内的上限令牌数后,将上限令牌数发送给相应的存储节点。
在存储节点上,针对同一存储资源,存在与该存储资源对应的下限令牌桶和上限令牌桶。下限令牌桶中的令牌即为下限令牌,上限令牌桶中的令牌即为上限令牌。存储节点1的调度器按照先入先出策略(first input first output,FIFO)将下一个(next)指针指向I/O请求队列中的第一个(即排在最前)I/O请求(即图2中的A0),判断A0是否具有下限令牌。例如,判断方式可以是判断客户端A在存储节点1上的下限令牌桶中的令牌数是否大于0,若大于0,则表明A0这一请求具有下限令牌,若等于0,则表明A0这一请求不具有下限令牌。若A0具有下限令牌,则调度器请求A0出列,处理(或执行)该A0请求,并且调度器将客户端A在存储节点1上的下限令牌桶中的令牌减1。例如,A0请求用于请求读取客户端A的数据1,则调度器从数据库中获取客户端A的数据1,并返回数据1给客户端A。调度器执行A0请求后,将next指针指向队列中的下一个请求,即图2中的A1。若A0不具有下限令牌,则调度器将next指针指向队列中的下一个请求,即图2中的A1。之后,判断A1是否具有下限令牌。判断方式与A0类似,此处不再赘述。之后,调度器将next指针指向队列中的下一个请求,即图2中的B0,判断B0是否具有下限令牌。例如,判断方式可以是判断客户端B在存储节点1上的下限令牌数是否大于0,若大于0,则表明B0这一请求具有下限令牌,若等于0,则表明B0这一请求不具有下限令牌。若B0具有下限令牌,则调度器请求B0出列,处理(或执行)该B0请求,并且调度器将客户端B在存储节点1上的下限令牌数减1。调度器执行B0请求后,将next指针指向队列中的下一个请求。若B0不具有下限令牌,则调度器将next指针指向队列中的下一个请求。
调度器遍历完I/O请求队列后,具有下限令牌的I/O请求已全部进行处理,不具有下限 令牌数的I/O请求则暂不处理。之后,存储节点1的调度器将下一个(next)指针重新指向I/O请求队列中的第一个(即排在最前)I/O请求(假设为图2中的B0),判断B0是否具有上限令牌,例如,判断方式可以是判断客户端B在存储节点1上的上限令牌桶中的令牌数是否大于0,若大于0,则表明B0这一请求具有上限令牌,若等于0,则表明B0这一请求不具有上限令牌。若B0具有上限令牌,则调度器请求B0出列,处理(或执行)该B0请求,并且调度器将客户端B在存储节点1上的上限令牌桶中的令牌减1。之后,调度器将next指针指向队列中的下一个请求,直至遍历完整个I/O请求队列。调度器遍历完I/O请求队列后,具有上限令牌的I/O请求已全部进行处理,不具有上限令牌数的I/O请求则暂不处理。
调度器根据存储节点上存储资源的下限令牌数和上限令牌数决定该存储资源执行I/O请求的数量,从而实现I/O请求调度。
实施上述分布式存储系统中服务质量保障方法,在每存储个节点上,通过部署令牌控制机制和调度器实现I/O请求的调度,进行分布式存储系统下限保障目标,从而实现QoS保障。同时也可以实现上限流控。以下实施例将具体阐述。
如图2中的控制节点和存储节点102可以参考图3所示的服务器结构。计算机设备300包括至少一个中央处理单元(Central Processing Unit,CPU)301,通信总线302,存储器303以及至少一个通信接口304。图3所示的服务器还包括存储设备,如机械硬盘和/或固态硬盘。
CPU301执行存储器303中的计算机指令用于实现本发明实施例。另外,为节省CPU301的计算资源,现场可编程门阵列(Field Programmable Gate Array,FPGA)或其他硬件也可以用于执行本发明实施例中CPU301全部操作,或者,FPGA或其他硬件与CPU301分别用于执行本发明实施例CPU301的操作。为方便描述,本发明实施例将CPU301与存储器303称为处理器、或将FPGA及其他替代CPU301的硬件称为处理器,或将FPGA及其他替代CPU201的硬件与CPU301的组合统称为处理器,处理器与通信接口304通信。通信接口304通信可以为网络接口卡(Networking Interface Card,NIC)、主机总线适配器(Host Bus Adaptor,HBA)等。
通信总线302可包括一通路,在上述组件之间传送信息。所述通信接口304,使用任何收发器一类的装置,用于与其他设备或通信网络通信,如以太网,无线接入网(radio access network,RAN),无线局域网(wireless local area networks,WLAN)等。
存储器303可以是只读存储器(read-only memory,ROM)或可存储静态信息和指令的其他类型的静态存储设备,随机存取存储器(random access memory,RAM)或者可存储信息和指令的其他类型的动态存储设备,也可以是电可擦可编程只读存储器(electrically erasable programmable read-only memory,EEPROM)、只读光盘(compact disc read-only memory,CD-ROM)或其他光盘存储、光碟存储(包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质,但不限于此。存储器可以是独立存在,通过总线与处理器相连接。存储器也可以和处理器集成在一起。
在具体实现中,作为一种实施例,处理器301以CPU为例,可以包括一个或多个CPU, 例如图3中的CPU0和CPU1。
如图2中的存储节点102可以为图3所示的设备,存储节点102的存储器中存储了一个或多个软件模块(如交互模块和处理模块)。如图2中的控制节点103也可以为图3所示的设备,控制节点103的存储器中存储了一个或多个软件模块(如交互模块和处理模块)。存储节点或者控制节点可以通过处理器以及存储器中的程序代码来实现软件模块,实现分布式存储系统中服务质量保障。
基于上述图1所述的分布式存储架构、图3所示存储节点、控制节点的结构示意图,本发明实施例提供了一种分布式存储系统中服务质量保障方法,参见图4,为本发明实施例提供的一种分布式存储系统中服务质量保障方法的流程图,该方法包括如下步骤。
S401:控制节点根据目标存储节点在单位时间内剩余的输入输出I/O请求个数、存储资源在单位时间内距离达成下限保障所需的I/O请求个数以及存储资源在单位时间内需要目标存储节点处理的I/O请求个数获得目标存储节点在单位时间内能够为存储资源处理的下限I/O请求个数;其中,分布式存储系统包含提供存储资源的N个存储节点;目标存储节点为N个存储节点中的一个,其中,N为正整数。
为了便于描述,以N个存储节点中的存储节点j作为目标存储节点为例进行描述,其他存储节点类似。为方便描述,将存储节点j在单位时间(假设为第1-第2秒这1秒钟内)内剩余可处理的I/O请求个数用RC
j表示。将存储资源i在该单位时间内距离达成下限保障所需的I/O请求个数用RR
i表示。将存储资源i在该单位时间内需要存储节点j处理的I/O请求个数用D
ij表示。将存储节点j在该单位时间内能够为存储资源i处理的下限I/O请求个数用f
ij表示。
在一种实现方式中,可以采用最大流算法、RC
j、RR
i、D
ij计算存储节点j在该单位时间内能够为存储资源i处理的下限I/O请求个数。
例如,基于如下算法计算存储节点j在该单位时间内能够为存储资源i处理的下限I/O请求个数:
构造图5所示有向图,其中,有向图由4类节点组成,其中,源节点和目的节点为虚拟节点,分别表示有向图的起始节点和终止节点,资源节点表示存储资源。从源节点到资源节点的边的权重表示该资源节点在单位时间内对应的QoS参数,资源节点到存储节点的边的权重表示该存储资源在单位时间内对该存储节点的需求值,存储节点到终止节点的边的权重表示该存储节点在单位时间内的节点能力。
对于有向图中的每一条边(u,v),设置默认流量f(u,v)=0。其中,u,v表示的是构造的有向图中的节点,u表示一条边的起始节点,v表示一条边的终止节点。对应到图5中,如果u是源节点,那么v就表示资源节点;如果u表示资源节点,那么v表示存储节点。f(u,v)表示从节点u到节点v的边流量大小,也就是分配的令牌数量(或令牌速度)。采用广度优先搜索算法,得到一条从源节点到终止节点的路径,源节点→存储资源i→存储节点j→终止节点;计算该路径上每个相邻节点间剩余流量cf(u,v)=c(u,v)-f(u,v),其中,c(u,v)为初始设置的边的权重,例如源节点到资源节点i的权重为RRi,资源节点i到存储节点j的边的权重为RCij,存储节点j到终止节点的权重为Dj;如果找到了这么一条路径,而且每个cf(u,v)都大于0,则计算cf(p)等于这些cf(u,v)中的最小值,然后计算每 条边的f(u,v)=f(u,v)+cf(p)。
如果没有搜索到路径,输出从资源节点i到资源节点j的边的流量f(资源节点i,存储节点j)=f(u,v),算法结束。
其中,资源节点i到资源节点j的边的流量即为存储节点j单位时间内(例如1秒钟)能够为存储资源i处理的下限I/O请求个数。
S402:控制节点根据目标存储节点在单位时间内能够为存储资源处理的下限I/O请求个数为存储资源分配存储资源在目标存储节点上且在单位时间内的下限令牌数。
在一种实现方式中,存储资源在目标存储节点上且在单位时间内的下限令牌数等于目标存储节点在单位时间内能够为存储资源处理的I/O请求个数。
在另一种实现方式中,存储资源在目标存储节点上且在单位时间内的下限令牌数等于目标存储节点在单位时间内能够为存储资源处理的I/O请求个数加或减去预定值,该预定值可以是控制节点预设的值。
S403:控制节点将存储资源在目标存储节点上且在单位时间内的下限令牌数发送给目标存储节点,目标存储节点从控制节点接收存储资源在目标存储节点上且在单位时间内的下限令牌数。
例如,控制节点将存储资源1在存储节点1上且在单位时间内的下限令牌数发送给存储节点1,将存储资源1在存储节点2上且在单位时间内的下限令牌数发送给存储节点2,将存储资源1在存储节点3上且在单位时间内的下限令牌数发送给存储节点3,等等。
S404:存储节点获取I/O请求队列中的第一I/O请求,第一I/O请求为存储资源需要目标存储节点处理的I/O请求。
存储节点获取I/O请求队列排在最前的I/O请求,判断该I/O请求是否具有下限令牌,若是,执行步骤S405,若否,存储节点将next指针指向I/O请求队列的下一个I/O请求,继续判断I/O请求是否具有下限令牌。直至遍历完I/O请求队列中的所有I/O请求。其中,存储节点判断I/O请求是否具有下限令牌的方式可以是:判断I/O请求对应的存储资源在存储节点上的下限令牌数是否大于0,若大于0,则表明该I/O请求具有下限令牌,若等于0,则表明该I/O请求不具有下限令牌。其中,I/O请求对应的存储资源意思是指:该I/O请求是该存储资源需要存储节点处理的请求。
S405:若第一I/O请求具有下限令牌数,则目标存储节点处理第一I/O请求。
可选的,除了下限令牌数,控制节点还可以为存储资源分配存储资源在目标存储节点上的上限令牌数。具体的,控制节点根据目标存储节点在单位时间内剩余的I/O请求个数(RC
j)、存储资源在单位时间内距离达成上限流控所需的I/O请求个数(RL
i)、存储资源在单位时间内需要目标存储节点处理的I/O请求个数(D
ij)以及目标存储节点在单位时间内能够为存储资源处理的下限I/O请求个数之和(∑
jf
ij)计算目标存储节点在单位时间内能够为存储资源处理的上限I/O请求个数(g
ij)。控制节点根据目标存储节点在单位时间内能够为存储资源处理的上限I/O请求个数为存储资源分配存储资源在目标存储节点上且在单位时间内的上限令牌数。控制节点将存储资源在目标存储节点上且在单位时间内的上限令牌数发送给目标存储节点。目标存储节点从控制节点接收存储资源在目标存储节点上且在单位时间内的上限令牌数。
为了便于描述,将存储节点j在单位时间内剩余可处理的I/O请求个数用RC
j表示。将存储资源i在单位时间内距离达成上限流控所需的I/O请求个数用RL
j表示。将存储资源i在单位时间内需要存储节点j处理的I/O请求个数用D
ij表示。将存储节点j在单位时间内能够为存储资源i处理的下限I/O请求个数用f
ij表示。N个存储节点在单位时间内能够为存储资源i处理的下限I/O请求个数之和等于
将存储节点j在单位时间内能够为存储资源i处理的上限I/O请求个数用g
ij表示。
在一种实现方式中,控制节点计算存储节点j在单位时间内能够为存储资源i处理的上限I/O请求个数的方式可以是:
计算第一数值,
计算第四数值,
计算第六数值,
之后,根据第一数值、第四数值和第六数值计算存储节点j在单位时间内能够为存储资源i处理的上限I/O请求个数。可选的,根据第一数值、第四数值和第六数值计算存储节点j在单位时间内能够为存储资源i处理的上限I/O请求个数的方式可以是:根据最大流(max flow)算法、第一数值、第四数值和第六数值计算存储节点j在单位时间内能够为存储资源i处理的下限I/O请求个数。
在一种实现方式中,控制节点根据存储节点j在单位时间内能够为存储资源i处理的上限I/O请求个数为存储资源i分配存储资源i在存储节点j上且在单位时间内的上限令牌数,包括:
控制节点根据如下公式为存储资源i分配存储资源i在存储节点j上且在单位时间内的超额分配值:
其中,h
ij表示存储资源i在存储节点j上且在单位时间内的超额分配值。g
ij表示存储节点j在单位时间内能够为存储资源i处理的上限I/O请求个数。
表示N个存储节点在单位时间内能够为存储资源i处理的上限I/O请求个数之和。D
ij表示存储资源i在单位时间内需要存储节点j处理的I/O请求个数,
表示存储资源i需要N个存储节点在单位时间内处理的I/O请求个数之和。
控制节点根据如下公式为存储资源i分配存储资源i在存储节点j上且在单位时间内的上限令牌数:
存储资源i在存储节点j上且在单位时间内的上限令牌数=h
ij+f
ij。f
ij表示存储节点j在单位时间内能够为存储资源i处理的下限I/O请求个数。
可选的,上述RC
j和D
ij可以是存储节点j发送给控制节点的,控制节点可以收集各个存储节点上报的需求信息以及存储节点的能力。
可选的,分布式存储系统中服务质量保障方法还包括如下步骤S406和S407。
S406:若全部存储资源在目标存储节点上的下限令牌数均等于零,则目标存储节点获取I/O请求队列中的第二I/O请求。
S407:若第二I/O请求具有上限令牌,则目标存储节点处理第二I/O请求,所述上限令牌的个数为所述目标存储节点从所述控制节点接收的。
其中,第一I/O请求和第二I/O请求可以是两个不同的存储资源对目标存储节点的需求。 例如,第一I/O请求是存储资源1对存储节点1的请求,第二I/O请求是存储资源2对存储节点1的请求。第一I/O请求和第二I/O请求还可以是同一存储资源对目标存储节点的需求。
例如,目标存储节点在从控制节点获取存储资源的下限令牌数和上限令牌数后,按照循环(round-robin)机制执行如下I/O调度过程:
1、将next指针指向队列开始遍历,找到队列中还有下限令牌的I/O请求,使其请求出队,执行该I/O请求;如果没有找到满足要求的I/O请求,则跳转到步骤3。
2、I/O请求出队后,假设该I/O请求是存储资源i需要存储节点j处理的请求,则将存储资源i在存储节点j上的下限令牌个数减1,并且将next指针指向队列中的下一个I/O请求,跳转到步骤1。直至队列中的所有I/O请求都不具有下限令牌。
3、按照round-robin机制将next指针指向队列开始遍历,找到队列中还有上限令牌的I/O请求,使其请求出队,执行该I/O请求;如果没有找到满足要求的I/O请求,则跳转到步骤5。
4、I/O请求出队后,假设该I/O请求是存储资源i需要存储节点j处理的请求,则将存储资源i在存储节点j上的上限令牌个数减1,并且将next指针指向队列中的下一个I/O请求,跳转到步骤1。直至队列中的所有I/O请求都不具有上限令牌。
5、结束。
采用该遍历方式,可以优先满足下限保障要求,在满足下限保障的请求下,再按照上限令牌桶实现上限流控。
采用本发明实施例,控制节点对令牌的分配可以由定时器定时触发,例如5s分配1次。也即是说,由控制节点定时生成令牌,并下发给各个存储节点。控制节点可以控制令牌的生成速度,从而达成各个存储资源的QoS目标。例如,针对存储资源1,控制节点每秒钟为其生成1000个令牌,其中,300个令牌发送给存储节点1,300个令牌发送给存储节点2,剩余的400个令牌发送给存储节点3。各个存储节点在收到控制节点为存储资源分配的令牌后,按照前述循环(round-robin)机制执行I/O调度过程。
此外,还可以由控制节点配置令牌生成速度(例如每秒钟生成若干个下限令牌或者每秒钟生成若干个上限令牌),然后各个存储节点负责按照控制节点配置的生成速度来生成相应的下限令牌或者上限令牌。例如,控制节点通知存储节点1每秒钟为存储资源1生成300个下限令牌以及100个上限令牌,控制节点通知存储节点2每秒钟为存储资源1生成300个下限令牌以及100个上限令牌,控制节点通知存储节点3每秒钟为存储资源1生成400个下限令牌以及100个上限令牌,则存储节点1按照300个令牌/秒的速度为存储资源1生成下限令牌以及100个令牌/秒的速度为存储资源1生成上限令牌,按照前述循环(round-robin)机制执行I/O调度过程。存储节点2按照300个令牌/秒的速度为存储资源1生成下限令牌以及100个令牌/秒的速度为存储资源1生成上限令牌,按照前述循环(round-robin)机制执行I/O调度过程。存储节点3按照400个令牌/秒的速度为存储资源1生成下限令牌100个令牌/秒的速度为存储资源1生成上限令牌,按照前述循环(round-robin)机制执行I/O调度过程。
采用本发明实施例,可以由控制节点为各个存储节点分配存储资源的令牌,因此存储节点只需要按照控制节点分配的令牌相应调度并处理具有令牌的I/O请求,不具有令牌的 I/O请求则存储节点对其暂不处理。在分布式存储系统中实现了下限保障性能。进一步地,还可以在分布式存储系统中引入存储资源上限流控机制。与现有技术中针对每个请求的tag计算,都需要获取其他节点上处理请求个数的信息不同,本发明实施例在计算存储资源在目标存储节点上的令牌数时,只需要关注目标存储节点的能力以及存储资源对目标存储节点的需求即可,无需关注其他存储节点的能力,因此可以降低网络通信开销。此外,本发明实施例可以定期实现令牌更新,例如每5s更新一次,而现有技术方案需要每I/O都同步。本发明实施例的存储节点在进行I/O请求调度时,不需要做请求tag值的排序,节省了排序的计算开销。
参见图6,图6示出了本发明实施例提供的一种分布式存储系统、控制节点及存储节点的结构框图。分布式存储系统600包括:控制节点700和存储节点800。其中,控制节点700可以为图1实施例中的控制节点103,相应的,存储节点800可以为图1实施例中的存储节点102,分布式存储系统600可以是图1描述的分布式存储系统100。下面分别描述。
如图6所示,控制节点700可包括:计算单元701、分配单元702和发送单元703。
其中,计算单元701,用于根据目标存储节点单位时间内剩余的输入输出I/O请求个数、存储资源在所述单位时间内距离达成下限保障所需的I/O请求个数以及所述存储资源在所述单位时间内需要所述目标存储节点处理的I/O请求个数获得所述目标存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数;其中,所述分布式存储系统包含提供所述存储资源的N个存储节点;所述目标存储节点为所述N个存储节点中的一个,其中,N为正整数;
分配单元702,用于根据所述目标存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数为所述存储资源分配在所述目标存储节点上且在所述单位时间内的下限令牌数;
发送单元703,用于将所述下限令牌数发送给所述目标存储节点。
可选的,计算单元701,还用于根据所述目标存储节点在所述单位时间内剩余的I/O请求个数、所述存储资源在所述单位时间内距离达成上限流控所需的I/O请求个数、所述存储资源在所述单位时间内需要所述目标存储节点处理的I/O请求个数以及所述目标存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数获得所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数;
分配单元702,还用于根据所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数为所述存储资源分配在所述目标存储节点上且在所述单位时间内的上限令牌数;
发送单元703,还用于将所述上限令牌数依次发送给所述目标存储节点。
可选的,所述计算单元701具体用于:
根据最大流算法、所述目标存储节点在所述单位时间内剩余的输入输出I/O请求个数、所述存储资源在所述单位时间内距离达成下限保障所需的I/O请求个数以及在所述单位时间内需要所述目标存储节点处理的I/O请求个数计算所述目标存储节点在所述单位时间内 能够为所述存储资源处理的下限I/O请求个数。
可选的,所述计算单元701具体用于:
计算第一数值,所述第一数值为第二数值与第三数值之差,所述第二数值为所述存储资源在所述单位时间内距离达成上限流控所需的I/O请求个数,所述第三数值为所述N个存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数之和;
计算第四数值,所述第四数值为第五数值与所述第三数值之差,所述第五数值为所述目标存储节点在所述单位时间内剩余的I/O请求个数;
计算第六数值,所述第六数值为第七数值与所述第三数值之前,所述第七数值为所述存储资源需要所述目标存储节点在所述单位时间内处理的I/O请求个数;
根据所述第一数值、所述第四数值和所述第六数值计算所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数。
可选的,所述分配单元702具体用于:
根据如下公式为所述存储资源分配所述存储资源在所述目标存储节点上且在所述单位时间内的超额分配值:所述存储资源在所述目标存储节点上且在所述单位时间内的
其中,所述第八数值为所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数,所述第九数值为所述N个存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数之和,所述第十参数为所述存储资源在所述单位时间内需要所述N个存储节点处理的I/O请求个数之和;其中,“-”表示减法运算,即第二数值与第九数值之前;“*”表示乘法运算;
根据如下公式为所述存储资源分配所述存储资源在所述目标存储节点上且在所述单位时间内的上限令牌数:
所述存储资源在所述目标存储节点上且在所述单位时间内的上限令牌数等于所述存储资源在所述目标存储节点上且在所述单位时间内的超额分配值与所述存储资源在所述目标存储节点上且在所述单位时间内的下限令牌之和。
可选的,所述计算单元701具体用于:
根据最大流算法、所述第一数值、所述第四数值和所述第六数值计算所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数。
可选的,控制节点700还包括:
接收单元,用于从所述目标存储节点接收所述目标存储节点在所述单位时间内剩余的I/O请求个数以及所述存储资源在所述单位时间内需要所述目标存储节点处理的I/O请求个数。
关于控制节点700包括的各个功能单元的具体实现,可以参考前述实施例,这里不再赘述。
如图6所示,存储节点800可包括:接收单元801、获取单元802和处理单元803。
其中,接收单元801,用于从控制节点接收第一存储资源在所述存储节点上且在单位 时间内的下限令牌数;
获取单元802,用于获取I/O请求队列中的第一I/O请求,所述第一I/O请求为所述第一存储资源需要所述存储节点处理的I/O请求;
处理单元803,用于若所述第一存储资源在所述存储节点上的下限令牌数大于0,则处理所述第一I/O请求。
可选的,获取单元802,还用于若所述第一存储资源以及除所述第一存储资源以外的其他存储资源在所述存储节点上的下限令牌数均等于零,则获取所述I/O请求队列中的第二I/O请求,所述第二I/O请求为第二存储资源需要所述存储节点处理的I/O请求;
所述处理单元803,还用于若所述第二存储资源在所述存储节点上的上限令牌数大于0,则处理所述第二I/O请求,所述上限令牌数为所述存储节点从所述控制节点接收的。
可选的,存储节点800还包括:
发送单元,用于在所述接收单元801从控制节点接收第一存储资源在所述存储节点上且在所述单位时间内的下限令牌数之前,向控制节点发送所述存储节点在所述单位时间内剩余的I/O请求个数以及所述第一存储资源在所述单位时间内需要所述存储节点处理的I/O请求个数,所述存储节点在所述单位时间内剩余的I/O请求个数以及所述第一存储资源在所述单位时间内需要所述存储节点处理的I/O请求个数用于所述控制节点为所述第一存储资源分配所述第一存储资源在所述存储节点上且在所述单位时间内的令牌数。
关于存储节点800包括的各个功能单元的具体实现,可以参考前述实施例,这里不再赘述。
图6所示的控制节点700和存储节点800中的各单元,具体实现可以参考图3的描述。其中,各单元可以是软件模块,也可以是硬件。本发明实施例对此不作限定。
在本发明的另一实施例中,还提供一种可读存储介质,可读存储介质中存储有计算机执行指令,当一个设备(可以是单片机,芯片等)或者处理器可以调用可读存储介质中存储有计算机执行指令来执行图4所提供的分布式存储系统中服务质量保障方法中控制节点或者存储节点的步骤。前述的可读存储介质可以包括:U盘、移动硬盘、只读存储器、随机存取存储器、磁碟或者光盘等各种可以存储程序代码的介质。
在本发明的另一实施例中,还提供一种计算机程序产品,该计算机程序产品包括计算机执行指令,该计算机执行指令存储在计算机可读存储介质中;设备的至少一个处理器可以从计算机可读存储介质读取该计算机执行指令,至少一个处理器执行该计算机执行指令使得设备实施图4所提供的分布式存储系统中服务质量保障方法中控制节点或者存储节点的步骤。
在本发明的另一实施例中,还提供一种分布式存储节点,包括控制节点和存储节点。其中,控制节点可以为图6所提供的控制节点,且用于执行图4所提供的分布式存储系统中服务质量保障方法中控制节点的步骤;和/或,存储节点可以为图6所提供的存储节点,且用于执行图4所提供的分布式存储系统中服务质量保障方法中存储节点的步骤。
在上述实施例中,可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时,可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时,全部或部 分地产生按照本发明实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中,或者从一个计算机可读存储介质向另一个计算机可读存储介质传输,例如,所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质,(例如,软盘、硬盘、磁带)、光介质(例如,DVD)、或者半导体介质(例如固态硬盘(solid state disk,SSD))等。
Claims (16)
- 一种分布式存储系统中服务质量保障方法,其特征在于,包括:控制节点根据目标存储节点单位时间内剩余的输入输出I/O请求个数、存储资源在所述单位时间内距离达成下限保障所需的I/O请求个数以及所述存储资源在所述单位时间内需要所述目标存储节点处理的I/O请求个数获得所述目标存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数;其中,所述分布式存储系统包含提供所述存储资源的N个存储节点;所述目标存储节点为所述N个存储节点中的一个,其中,N为正整数;所述控制节点根据所述目标存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数为所述存储资源分配在所述目标存储节点上且在所述单位时间内的下限令牌数;所述控制节点将所述下限令牌数发送给所述目标存储节点。
- 根据权利要求1所述的方法,其特征在于,还包括:所述控制节点根据所述目标存储节点在所述单位时间内剩余的I/O请求个数、所述存储资源在所述单位时间内距离达成上限流控所需的I/O请求个数、所述存储资源在所述单位时间内需要所述目标存储节点处理的I/O请求个数以及所述目标存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数获得所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数;所述控制节点根据所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数为所述存储资源分配在所述目标存储节点上且在所述单位时间内的上限令牌数;所述控制节点将所述上限令牌数依次发送给所述目标存储节点。
- 根据权利要求1所述的方法,其特征在于,所述控制节点根据目标存储节点在单位时间内剩余的输入输出I/O请求个数、存储资源在所述单位时间内距离达成下限保障所需的I/O请求个数以及所述存储资源在所述单位时间内需要所述目标存储节点处理的I/O请求个数获得所述目标存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数,包括:所述控制节点根据最大流算法、所述目标存储节点在所述单位时间内剩余的输入输出I/O请求个数、所述存储资源在所述单位时间内距离达成下限保障所需的I/O请求个数以及所述存储资源在所述单位时间内需要所述目标存储节点处理的I/O请求个数计算所述目标存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数。
- 根据权利要求2所述的方法,其特征在于,所述控制节点根据所述目标存储节点在所述单位时间内剩余的I/O请求个数、所述存储资源在所述单位时间内距离达成上限流控所需的I/O请求个数、所述存储资源在所述单位时间内需要所述目标存储节点处理的I/O请求个数以及所述目标存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求 个数获得所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数,包括:所述控制节点计算第一数值,所述第一数值为第二数值与第三数值之差,所述第二数值为所述存储资源在所述单位时间内距离达成上限流控所需的I/O请求个数,所述第三数值为所述N个存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数之和;所述控制节点计算第四数值,所述第四数值为第五数值与所述第三数值之差,所述第五数值为所述目标存储节点在所述单位时间内剩余的I/O请求个数;所述控制节点计算第六数值,所述第六数值为第七数值与所述第三数值之差,所述第七数值为所述存储资源在所述单位时间内需要所述目标存储节点处理的I/O请求个数;所述控制节点根据所述第一数值、所述第四数值和所述第六数值获得所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数。
- 根据权利要求4所述的方法,其特征在于,所述控制节点根据所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数为所述存储资源分配所述存储资源在所述目标存储节点上且在所述单位时间内的上限令牌数,包括:所述控制节点根据如下公式为所述存储资源分配在所述目标存储节点上且在所述单位时间内的超额分配值:所述存储资源在所述目标存储节点上且 其中,所述第八数值为所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数,所述第九数值为所述N个存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数之和,所述第十数值为所述存储资源在所述单位时间内需要所述N个存储节点处理的I/O请求个数之和;所述控制节点根据如下公式为所述存储资源分配所述存储资源在所述目标存储节点上且在所述单位时间内的上限令牌数:所述存储资源在所述目标存储节点上且在所述单位时间内的上限令牌数为所述存储资源在所述目标存储节点上且在所述单位时间内的超额分配值与所述存储资源在所述目标存储节点上且在所述单位时间内的下限令牌数之和。
- 根据权利要求4或5所述的方法,其特征在于,所述控制节点根据所述第一数值、所述第四数值和所述第六数值获得所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数,包括:所述控制节点根据最大流算法、所述第一数值、所述第四数值和所述第六数值计算所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数。
- 根据权利要求1至6任一项所述的方法,其特征在于,还包括:所述控制节点从所述目标存储节点接收所述目标存储节点在所述单位时间内剩余的I/O请求个数以及所述存储资源在所述单位时间内需要所述目标存储节点处理的I/O请求个数。
- 一种控制节点,其特征在于,包括:计算单元,用于根据目标存储节点在单位时间内剩余的输入输出I/O请求个数、存储资源在所述单位时间内距离达成下限保障所需的I/O请求个数以及在所述单位时间内需要所述目标存储节点处理的I/O请求个数获得所述目标存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数;其中,所述分布式存储系统包含提供所述存储资源的N个存储节点;所述目标存储节点为所述N个存储节点中的一个,其中,N为正整数;分配单元,用于根据所述目标存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数为所述存储资源分配在所述目标存储节点上且在所述单位时间内的下限令牌数;发送单元,用于将所述下限令牌数发送给所述目标存储节点。
- 根据权利要求8所述的控制节点,其特征在于,所述计算单元,还用于根据所述目标存储节点在所述单位时间内剩余的I/O请求个数、所述存储资源在所述单位时间内距离达成上限流控所需的I/O请求个数、所述存储资源在所述单位时间内需要所述目标存储节点处理的I/O请求个数以及所述目标存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数获得所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数;所述分配单元,还用于根据所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数为所述存储资源分配在所述目标存储节点上且在所述单位时间内的上限令牌数;所述发送单元,还用于将所述上限令牌数依次发送给所述目标存储节点。
- 根据权利要求8所述的控制节点,其特征在于,所述计算单元具体用于:根据最大流算法、所述目标存储节点在所述单位时间内剩余的输入输出I/O请求个数、所述存储资源在所述单位时间内距离达成下限保障所需的I/O请求个数以及所述存储资源在所述单位时间内需要所述目标存储节点处理的I/O请求个数计算所述目标存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数。
- 根据权利要求9所述的控制节点,其特征在于,所述计算单元具体用于:计算第一数值,第四数值和第六数值,根据所述第一数值、所述第四数值和所述第六数值获得所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数;其中,所述第一数值为第二数值与第三数值之差,所述第二数值为所述存储资源在所述单位时间内距离达成上限流控所需的I/O请求个数,所述第三数值为所述N个存储节点在所述单位时间内能够为所述存储资源处理的下限I/O请求个数之和;所述第四数值为第 五数值与所述第三数值之差,所述第五数值为所述目标存储节点在所述单位时间内剩余的I/O请求个数;所述第六数值为第七数值与所述第三数值之差,所述第七数值为所述存储资源在所述单位时间内需要所述目标存储节点处理的I/O请求个数。
- 根据权利要求11所述的控制节点,其特征在于,所述分配单元具体用于:根据如下公式为所述存储资源分配在所述目标存储节点上且在所述单位时间内的超额分配值:所述存储资源在所述目标存储节点上且 其中,所述第八数值为所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数,所述第九数值为所述N个存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数之和,所述第十数值为所述存储资源在所述单位时间内需要所述N个存储节点处理的I/O请求个数之和;根据如下公式为所述存储资源分配所述存储资源在所述目标存储节点上且在所述单位时间内的上限令牌数:所述存储资源在所述目标存储节点上且在所述单位时间内的上限令牌数为所述存储资源在所述目标存储节点上且在所述单位时间内的超额分配值与所述存储资源在所述目标存储节点上且在所述单位时间内的下限令牌数之和。
- 根据权利要求11或12所述的控制节点,其特征在于,所述计算单元具体用于:根据最大流算法、所述第一数值、所述第四数值和所述第六数值计算所述目标存储节点在所述单位时间内能够为所述存储资源处理的上限I/O请求个数。
- 根据权利要求8至13任一项所述的控制节点,其特征在于,还包括接收单元;所述接收单元,用于从所述目标存储节点接收所述目标存储节点在所述单位时间内剩余的I/O请求个数以及所述存储资源在所述单位时间内需要所述目标存储节点处理的I/O请求个数。
- 一种控制节点,其特征在于,包括处理器和通信接口,所述处理器和所述通信接口通信,所述处理器用于执行权利要求1至7任一所述的方法。
- 一种分布式存储系统,其特征在于,包括控制节点和N个存储节点,其中,所述控制节点用于执行权利要求1至7任一所述的方法。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP19850031.6A EP3820119A4 (en) | 2018-08-16 | 2019-07-24 | SERVICE QUALITY ASSURANCE PROCEDURE IN DISTRIBUTED STORAGE SYSTEM, CONTROL NODE AND SYSTEM |
US17/175,331 US11320998B2 (en) | 2018-08-16 | 2021-02-12 | Method for assuring quality of service in distributed storage system, control node, and system |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810935171.1 | 2018-08-16 | ||
CN201810935171.1A CN109254726A (zh) | 2018-08-16 | 2018-08-16 | 分布式存储系统中服务质量保障方法、控制节点及系统 |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/175,331 Continuation US11320998B2 (en) | 2018-08-16 | 2021-02-12 | Method for assuring quality of service in distributed storage system, control node, and system |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2020034819A1 true WO2020034819A1 (zh) | 2020-02-20 |
Family
ID=65050209
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2019/097560 WO2020034819A1 (zh) | 2018-08-16 | 2019-07-24 | 分布式存储系统中服务质量保障方法、控制节点及系统 |
Country Status (4)
Country | Link |
---|---|
US (1) | US11320998B2 (zh) |
EP (1) | EP3820119A4 (zh) |
CN (1) | CN109254726A (zh) |
WO (1) | WO2020034819A1 (zh) |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109254726A (zh) * | 2018-08-16 | 2019-01-22 | 华为技术有限公司 | 分布式存储系统中服务质量保障方法、控制节点及系统 |
CN110221775B (zh) * | 2019-05-05 | 2020-10-16 | 华为技术有限公司 | 存储系统中令牌的分配方法和装置 |
CN110795371B (zh) * | 2019-10-25 | 2021-06-29 | 浪潮电子信息产业股份有限公司 | 一种i/o请求处理方法、系统、设备及计算机存储介质 |
US11494229B2 (en) * | 2019-11-25 | 2022-11-08 | Google Llc | Rule violation detection |
CN111338575B (zh) * | 2020-02-20 | 2021-06-29 | 苏州浪潮智能科技有限公司 | 一种存储服务质量控制方法、装置、设备及存储介质 |
CN114115718B (zh) * | 2021-08-31 | 2024-03-29 | 济南浪潮数据技术有限公司 | 分布式块存储系统服务质量控制方法、装置、设备及介质 |
CN113839996B (zh) * | 2021-09-10 | 2024-04-09 | 北京星辰天合科技股份有限公司 | 对象存储分布式质量服务的方法及装置 |
CN114745334B (zh) * | 2022-03-31 | 2024-06-28 | 广州易星信息科技有限公司 | 一种分布式系统限流方法及系统 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8473566B1 (en) * | 2006-06-30 | 2013-06-25 | Emc Corporation | Methods systems, and computer program products for managing quality-of-service associated with storage shared by computing grids and clusters with a plurality of nodes |
CN103442076A (zh) * | 2013-09-04 | 2013-12-11 | 上海海事大学 | 一种云存储系统的可用性保障方法 |
CN108196788A (zh) * | 2017-12-28 | 2018-06-22 | 新华三技术有限公司 | QoS指标监测方法、装置、存储介质 |
CN109254726A (zh) * | 2018-08-16 | 2019-01-22 | 华为技术有限公司 | 分布式存储系统中服务质量保障方法、控制节点及系统 |
Family Cites Families (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130036272A1 (en) * | 2011-08-02 | 2013-02-07 | Microsoft Corporation | Storage engine node for cloud-based storage |
US10078533B2 (en) * | 2014-03-14 | 2018-09-18 | Amazon Technologies, Inc. | Coordinated admission control for network-accessible block storage |
CN103929366B (zh) * | 2014-04-02 | 2017-07-07 | 华为技术有限公司 | 流量控制方法、装置及设备 |
WO2017024965A1 (zh) * | 2015-08-11 | 2017-02-16 | 阿里巴巴集团控股有限公司 | 一种数据流量限制的方法及系统 |
CN110073321B (zh) * | 2016-12-29 | 2020-05-08 | 华为技术有限公司 | 一种存储控制器及io请求处理方法 |
CN109857805B (zh) * | 2018-12-27 | 2020-10-02 | 重庆小雨点小额贷款有限公司 | 基于区块链的数据存储方法、相关设备及存储介质 |
-
2018
- 2018-08-16 CN CN201810935171.1A patent/CN109254726A/zh active Pending
-
2019
- 2019-07-24 EP EP19850031.6A patent/EP3820119A4/en not_active Withdrawn
- 2019-07-24 WO PCT/CN2019/097560 patent/WO2020034819A1/zh unknown
-
2021
- 2021-02-12 US US17/175,331 patent/US11320998B2/en active Active
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8473566B1 (en) * | 2006-06-30 | 2013-06-25 | Emc Corporation | Methods systems, and computer program products for managing quality-of-service associated with storage shared by computing grids and clusters with a plurality of nodes |
CN103442076A (zh) * | 2013-09-04 | 2013-12-11 | 上海海事大学 | 一种云存储系统的可用性保障方法 |
CN108196788A (zh) * | 2017-12-28 | 2018-06-22 | 新华三技术有限公司 | QoS指标监测方法、装置、存储介质 |
CN109254726A (zh) * | 2018-08-16 | 2019-01-22 | 华为技术有限公司 | 分布式存储系统中服务质量保障方法、控制节点及系统 |
Non-Patent Citations (1)
Title |
---|
See also references of EP3820119A4 * |
Also Published As
Publication number | Publication date |
---|---|
US20210173573A1 (en) | 2021-06-10 |
US11320998B2 (en) | 2022-05-03 |
EP3820119A4 (en) | 2021-08-25 |
CN109254726A (zh) | 2019-01-22 |
EP3820119A1 (en) | 2021-05-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2020034819A1 (zh) | 分布式存储系统中服务质量保障方法、控制节点及系统 | |
US11334929B2 (en) | Managing resource requests that exceed reserved resource capacity | |
US9385963B1 (en) | System and method for allocating resources for heterogeneous service requests | |
US9882832B2 (en) | Fine-grained quality of service in datacenters through end-host control of traffic flow | |
US9998531B2 (en) | Computer-based, balanced provisioning and optimization of data transfer resources for products and services | |
US10338970B2 (en) | Multi-platform scheduler for permanent and transient applications | |
US10545871B2 (en) | Coordination of cache and memory reservation | |
US11784942B2 (en) | Dynamic allocation of edge network resources | |
Yin et al. | Online SLA-aware multi-resource allocation for deadline sensitive jobs in edge-clouds | |
US10956228B2 (en) | Task management using a virtual node | |
CN113906720B (zh) | 流量调度方法、设备及存储介质 | |
US10540341B1 (en) | System and method for dedupe aware storage quality of service | |
WO2020166617A1 (ja) | リソース競合調停装置、リソース競合調停方法、及びプログラム | |
CN114884823A (zh) | 流量拥塞控制方法、装置、计算机可读介质及电子设备 | |
US10924416B2 (en) | Method for traffic shaping using a serial packet processing algorithm and a parallel packet processing algorithm | |
Mehta et al. | Utility-based allocation of industrial IoT applications in mobile edge clouds | |
US11533362B1 (en) | Network interface controller aware placement of virtualized workloads | |
Sutagundar et al. | Development of fog based dynamic resource allocation and pricing model in IoT | |
Meyer et al. | Low latency packet processing in software routers | |
US11621919B2 (en) | Dynamic load balancing in reactive systems | |
US11880726B1 (en) | Fair queuing of request tasks spawned by requests to execute generative operations | |
Runge et al. | A modeling approach for resource management in resource-constrained nodes | |
EP3433985B1 (en) | Method for traffic shaping using a serial packet processing algorithm and a parallel packet processing algorithm | |
Truong-Huu et al. | Handling Uncertainty and Diversity in Cloud Bandwidth Demands for Revenue Maximization | |
CN115242727A (zh) | 用户请求处理方法、装置、设备和介质 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 19850031 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |