WO2016106516A1 - 在分布式资源系统中用户请求的调度方法和装置 - Google Patents

在分布式资源系统中用户请求的调度方法和装置 Download PDF

Info

Publication number
WO2016106516A1
WO2016106516A1 PCT/CN2014/095381 CN2014095381W WO2016106516A1 WO 2016106516 A1 WO2016106516 A1 WO 2016106516A1 CN 2014095381 W CN2014095381 W CN 2014095381W WO 2016106516 A1 WO2016106516 A1 WO 2016106516A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
scheduler
resource
request
scheduling
Prior art date
Application number
PCT/CN2014/095381
Other languages
English (en)
French (fr)
Inventor
曾华荣
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to PCT/CN2014/095381 priority Critical patent/WO2016106516A1/zh
Priority to CN201480001882.1A priority patent/CN105934928B/zh
Priority to EP14891590.3A priority patent/EP3057288A4/en
Priority to US14/932,628 priority patent/US9575691B2/en
Publication of WO2016106516A1 publication Critical patent/WO2016106516A1/zh
Priority to US15/391,643 priority patent/US10127079B2/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G06F3/0659—Command handling arrangements, e.g. command buffers, queues, command scheduling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/50—Queue scheduling
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/50—Queue scheduling
    • H04L47/60—Queue scheduling implementing hierarchical scheduling
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/50—Queue scheduling
    • H04L47/62—Queue scheduling characterised by scheduling criteria
    • H04L47/622—Queue service order
    • H04L47/623—Weighted service order
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/70—Admission control; Resource allocation

Definitions

  • the present invention relates to the field of information technology, and in particular, to a scheduling method and apparatus for user request in a distributed resource system.
  • Resources including central processing unit (CPU) computing power, hard disk space, memory, and network bandwidth, etc.
  • resources can be physical resources or virtual resources.
  • the distributed resource system shown in FIG. 1 is a distributed file system, including user agent A, user agent B, scheduler A, scheduler B, and resource providing entity A.
  • the entity B is provided with the resource, and the user agent A and the user agent B do not communicate.
  • the user agent A requests the distributed file system metadata server according to the request of the user A.
  • the user agent A determines, according to the queried information, which requester is sent to the scheduler shown in FIG. 1; similarly, the user agent A according to the user B requests to query the distributed file system metadata server, and the user agent B determines which of the schedulers shown in FIG. 1 to send the user B request based on the queried information.
  • the resource providing entity A and the resource providing entity B are file systems.
  • the resource providing entity A is used to provide the resource A, and the resource A is specifically configured to provide the input/output operations per second (IOPS).
  • the resource providing entity B is used to provide the resource B and the resource B. Specifically, the ability to provide IOPS.
  • User A and User B send requests to the distributed resource system, respectively. For example, the user A sends a user A request to the resource providing entity A and the resource providing entity B through the user agent A, respectively, and the resource providing entity A requests the user A to provide the resource A, and the resource providing entity B requests the user A to provide the resource B.
  • User B sends a User B request to the resource providing entity A and the resource providing entity B through the user agent B, respectively, and the resource providing entity A and The resource providing entity B provides resources for the user B request.
  • the user A sends the user A request to the resource providing entity A and the resource providing entity B respectively.
  • the resources provided by the resource providing entity A and the resource providing entity B to the user A are all IOPS capabilities.
  • the resources provided by the resource providing entity A and the resource providing entity B to the user B are also IOPS capable.
  • user A and user B are respectively assigned resource weights. For example, user A has a resource weight of 2 and user B has a resource weight of 1.
  • the user's resource weight represents the resource share allocated by the distributed resource system to the user.
  • User Agent A counts the number of User A requests sent by User A to Scheduler A and Scheduler B.
  • User Agent B counts the number of User B requests that User B sends to Scheduler A and Scheduler B.
  • the scheduler A requests the user A to allocate resources from the resource providing entity A according to the resource weight of the user A, the number of user A requests sent by the user agent A to the scheduler A and the scheduler B.
  • the scheduler A requests the user B to allocate resources from the resource providing entity A according to the resource weight of the user B, the number of user B requests sent by the user agent B to the scheduler A and the scheduler B.
  • the scheduler B requests the user A to allocate resources from the resource providing entity B according to the resource weight of the user A, the number of user A requests sent by the user agent A to the scheduler A and the scheduler B.
  • the scheduler B requests the user B to allocate resources from the resource providing entity B according to the resource weight of the user B, the number of user B requests sent by the user agent B to the scheduler A and the scheduler B.
  • an embodiment of the present invention provides a scheduling method for a user request in a distributed resource system, where the distributed resource system includes a scheduler S x , a resource providing entity R x , and a coordinator G y , where x is 1 to M consecutive natural numbers, M ⁇ 2; y is 1 to successive natural numbers Y, Y is a natural number; S x R x in communication with, G y S x with any communication; the method comprising:
  • S d uses a scheduling algorithm to schedule P i z, d according to ⁇ z , C z (T n ), C z,d (T n ) and N z,d (T n ), P i z,d is S d
  • the received i-th user z request, C z,d (T n ) is the resource provided by R z received by S d in the T n period , d (T n ) users z requesting consumption of R d .
  • S d is used according to ⁇ z , C z (T n ), C z, d (T n ), and N z,d (T n ).
  • the scheduling algorithm schedules P i z,d , and specifically includes:
  • the S d ⁇ z, C z (T n ), C z, d (T n) and N z, d (T n) is calculated P i z, d virtual start time S (P i z, d), and Ending the virtual time F(P i z,d ), adding P i z,d to the scheduling queue; the scheduling queue sorts the user requests according to the size of the starting virtual time requested by the user.
  • S(P i z,d ) max ⁇ v(P i z,d ), F(P I-1 z,d ) ⁇ , v (P i z, d) represents the received S d P i z, S d of d when the virtual time, c (P i z, d ) represents the resource P i z, d d R & lt consumed provided.
  • v (P i z, d) represents the received S d P i z, S d of d when the virtual time
  • c (P i z, d ) represents the resource P i z, d d R & lt consumed provided.
  • an embodiment of the present invention provides a scheduler S d , where the scheduler S d is applied to a distributed resource system, where the distributed resource system includes a scheduler S x , a resource providing entity R x , and a coordinator G y . , wherein x is from 1 to M consecutive natural numbers, M ⁇ 2, d x is one; Y 1 to Y successive natural numbers; and R & lt S x x communications, C y S x with any communication; the scheduling The device S d includes a central processing unit and a memory, the central processing unit executing executable instructions in the memory to perform any one of the first aspect of the embodiments of the present invention to the third possible implementation of the first aspect .
  • an embodiment of the present invention provides a scheduler S d , where the scheduler S d is applied to a distributed resource system, where the distributed resource system includes a scheduler S x , a resource providing entity R x , and a coordinator G y .
  • x is from 1 to M consecutive natural numbers, M ⁇ 2, d x is one; Y 1 to Y successive natural numbers; and R & lt S x x communications, C y S x with any communication;
  • the scheduling The device S d includes an obtaining unit and a scheduling unit; wherein
  • the obtaining unit is configured to acquire, from the coordinator G k of the user z, the resource C z (T n ) requested by the user z during the T n period, in the T n+1 cycle, Where k is a natural number, 1 ⁇ k ⁇ Y; the resource weight of the user z is ⁇ z ; C z,x (T n ) is N z,x (T n ) users received by S x in the T n period z requesting the resource provided by R x ; z represents the identity of the user;
  • the scheduling unit is configured to schedule P i z,d using a scheduling algorithm according to ⁇ z , C z (T n ), C z, d (T n ) and N z,d (T n ), P i z, d is the i-th user z request received by S d , C z,d (T n ) is the resource provided by R z received by S d in the T n period , d (T n ) R d consumed by the user z request.
  • an embodiment of the present invention provides a non-transitory computer readable storage medium storing computer instructions for performing scheduling of user requests in a distributed resource system
  • the distributed resource system includes a scheduler S x , a resource providing entity R x and a coordinator G y , where x is a continuous natural number from 1 to M, M ⁇ 2; y is a natural number from 1 to Y, and Y is a natural number; S x communicates with R x , C y communicates with any S x ; scheduler S d executes the computer instructions for performing any of the first aspect of the first embodiment of the present invention to the third aspect of the first aspect a method; wherein d is one of x.
  • an embodiment of the present invention provides a distributed resource system, where the distributed resource system includes a scheduler S x , a resource providing entity R x , and a coordinator G y , where x is a continuous natural number of 1 to M. M ⁇ 2; y is 1 to successive natural numbers Y, Y is a natural number; S x R x in communication with, G y S x with any communication;
  • the coordinator G k of the user z is used to provide the scheduler S d with the resource C z (T n ) consumed by the user z request during the T n period, in the T n+1 period,
  • d and k are natural numbers, 1 ⁇ d ⁇ M,1 ⁇ k ⁇ Y;
  • the resource weight of the user z is ⁇ z ;
  • C z,x (T n ) is the N z received by S x in the T n period , x (T n ) users z request to consume the resources provided by R x ;
  • z represents the identity of the user;
  • the scheduler S d is configured to schedule P i z,d using a scheduling algorithm according to ⁇ z , C z (T n ), C z, d (T n ) and N z,d (T n ), P i z, d is the i-th user z request received by S d , C z,d (T n ) is the N z received by S d in the T n period , and d (T n ) is provided by the user d requesting R d Resources.
  • an embodiment of the present invention provides a resource scheduling method in a distributed resource system, where the distributed resource system includes multiple schedulers.
  • a first one of the plurality of schedulers acquires, from the coordinator of the first user, a sum of resources consumed by the user of the first user in the previous cycle to be consumed by the plurality of schedulers;
  • Determining, by the first scheduler, the sum of resources consumed by the plurality of schedulers in the previous period according to the resource weight of the first user, and the user of the first user in the previous period, in the previous period The user of the first user requests scheduling of the user request of the first user in the resource consumed by the first scheduler and the number of user requests by the first scheduler to receive the first user in the previous period.
  • the first scheduler according to the resource weight of the first user, the first user in the previous period
  • the user requests the sum of resources consumed by the plurality of schedulers, the user of the first user in the previous period requests resources consumed in the first scheduler, and the first in the previous period
  • the scheduler receives the user request of the first user, and schedules the user request of the first user, specifically:
  • the first scheduler according to the resource weight of the first user, the sum of resources consumed by the plurality of schedulers in the previous period of the user of the first user, in the previous period, Calculating the user request of the first user by the user of the first user requesting the resource consumed by the first scheduler and the number of user requests of the first scheduler receiving the first user in the previous period
  • the initial virtual time and the ending virtual time, the user request of the first user is added to the scheduling queue; the scheduling queue sorts the user requests of the first user according to the size of the initial virtual time requested by the user.
  • the embodiment of the present invention provides a distributed resource system, where the distributed resource system includes multiple schedulers, where a first one of the multiple schedulers includes an obtaining unit and a scheduling unit.
  • the obtaining unit is configured to acquire, from the coordinator of the first user, a sum of resources consumed by the user of the first user in the previous cycle to be consumed by the multiple schedulers;
  • the scheduling unit is configured to: according to the resource weight of the first user, a sum of resources consumed by the user of the first user in the previous period, and the resources in the previous period, The user of the first user requests scheduling of the user request of the first user in the resource consumed by the first scheduler and the number of user requests by the first scheduler to receive the first user in the previous period.
  • the scheduling unit is specifically configured to: according to the resource weight of the first user, the user request of the first user in the previous period is a sum of resources consumed by the scheduler, a resource requested by the first user in the previous period, and a first scheduler receiving the first resource in the previous period Counting the number of user requests of the user, calculating a starting virtual time and ending virtual time of the user request of the first user, and adding a user request of the first user to the scheduling queue; the scheduling queue is virtualized according to the start of the user request The size of the time requests the user of the first user put in order.
  • S d acquires the resource C consumed by the user z request in the T n period from the coordinator G k of the user z.
  • S d uses a scheduling algorithm to schedule P i z,d according to ⁇ z , C z (T n ), C z,d (T n ) and N z,d (T n ), without Depending on the user agent, the user z request can be scheduled, and S d uses a scheduling algorithm for P i according to ⁇ z , C z (T n ), C z, d (T n ) and N z,d (T n ) z, d is scheduled to implement global scheduling of user z requests to ensure the performance requirements of user z.
  • FIG. 1 is a schematic diagram of a prior art distributed resource scheduling system
  • FIG. 2 is a schematic diagram of a distributed resource system according to an embodiment of the present invention.
  • Figure 3a is a relationship diagram of scheduler A, resource providing entity A, and coordinator A;
  • Figure 3b is a diagram of the relationship between the scheduler A, the resource providing entity A, and the coordinator A;
  • FIG. 4 is a schematic diagram of a method according to an embodiment of the present invention.
  • FIG. 5 is a schematic diagram of a state of a distributed resource system according to an embodiment of the present invention.
  • FIG. 6 is a schematic diagram of a user request status in a scheduling queue
  • FIG. 7 is a schematic diagram of a user request status in a scheduling queue
  • FIG. 8 is a schematic diagram of a state of a distributed resource system according to an embodiment of the present invention.
  • FIG. 9 is a schematic diagram of a user request status in a scheduling queue
  • FIG. 10 is a schematic diagram of a user request status in a scheduling queue
  • FIG. 11 is a structural diagram of a distributed resource system according to an embodiment of the present invention.
  • FIG. 12 is a schematic structural diagram of a scheduler according to an embodiment of the present invention.
  • an embodiment of the present invention provides a distributed resource system.
  • the distributed resource system includes a scheduler A, a scheduler B, a resource providing entity A, a resource providing entity B, a coordinator A, and a coordinator B.
  • the scheduler A communicates with the resource providing entity A for scheduling user requests sent to the resource providing entity A.
  • the scheduler B communicates with the resource providing entity B for scheduling user requests sent to the resource providing entity B.
  • Coordinator A communicates with scheduler A and scheduler B, respectively, and coordinator B communicates with scheduler A and scheduler B, respectively.
  • the resources in the embodiment of the present invention may be physical resources, and may also be virtual resources, such as CPU computing power, hard disk space, memory, and network bandwidth.
  • the user in the embodiment of the present invention may be referred to as a client, a virtual machine, an application process, and the like, which are not specifically limited in the embodiment of the present invention.
  • the resources provided by the resource providing entity for the user A request and the resources allocated for the user B request are the same type of resources.
  • the resources provided by the resource providing entity A and the resource providing entity B for the user A request and the resources provided by the resource providing entity A and the resource providing entity B for the user B are the same type of resources.
  • the resource providing entity requests the resource allocated by the user and the resource providing entity has the same meaning for providing the resource for the user request, and may also express the resource of the resource providing entity requested by the user request or the resource providing entity requested by the user. Resources provided.
  • the resource provided by the resource requesting entity that the user requests to consume refers to the resource that the user requests to consume, including the resource that has been consumed. If the user request has been processed by the resource providing entity, the scheduler also receives the user request and does not provide the resource. The resource that the entity processes the user request to consume. In one implementation manner, the scheduler receives the user request, and can calculate the amount of resources provided by the resource providing entity that the user request needs to consume. For example, in one cycle, scheduler A receives 100 user requests, then 100 uses The resource provided by the resource requesting entity that the user requests to consume can be calculated according to the resource summation provided by the resource providing entity that needs to be consumed by each user request.
  • the resource is the IOPS capability
  • the resource providing entity A is the storage array A
  • the resource providing entity B is the storage array B.
  • the relationship between the scheduler A, the coordinator A, and the resource providing entity A shown in FIG. 2 is only a logical representation.
  • the resource providing entity A, the scheduler A, and the coordinator A are located in the same device; or as shown in FIG. 3b, the resource providing entity A and the scheduler A are located in the same device, and the coordinator A is independent of the resource providing entity A.
  • the scheduler A, or other implementation manners, the specific implementation manner is not limited herein.
  • the relationship between the scheduler B, the coordinator B, and the resource providing entity B shown in FIG. 2 can be referred to FIG. 3a and FIG. 3b.
  • the scheduler can be a standalone device or a hardware module.
  • the functions of the scheduler can also be implemented by the processor executing specific computer instructions. If the scheduler A, the coordinator A, and the resource providing entity A are all implemented by the processor executing specific computer instructions, the scheduler A communicates with the coordinator A, and the scheduler A communicates with the resource providing entity A, which can be expressed. For the instruction call, or the link of different program modules, etc.; similarly, the coordinator can be a stand-alone device or a hardware module.
  • the function of the coordinator can also be implemented by the processor executing specific computer instructions, which is not limited by the embodiment of the present invention.
  • the method steps implemented by the scheduler in the embodiment of the present invention may be implemented by a processor of a computer executing computer instructions in a memory. I will not repeat them here.
  • both storage array A and storage array B need to consume IOPS capability when processing user A request.
  • user B sends user B requests to storage array A and storage array B, and also consumes the IOPS capabilities of storage array A and storage array B.
  • the scheduler A needs to receive the user A request and the user B request according to the adjustment.
  • the degree algorithm performs scheduling; when the IOPS capability in the storage array B cannot satisfy the simultaneous processing of the user A request and the user B request, the scheduler B needs to schedule the received user A request and the user B request according to the scheduling algorithm.
  • User A and user B are users who have a resource competition relationship. To ensure the performance of each user, in a distributed resource system, each user is assigned a resource weight.
  • user A and user B are taken as an example.
  • the resource weight ⁇ A of user A is 2, and the resource weight ⁇ B of user B is 1.
  • the resource weight ratio of user A and user B is 2:1.
  • the resource weights of user A and user B are configured on scheduler A, and the resource weights of user A and user B are configured on scheduler B.
  • the user A can deliver the resource weight of the user A to the scheduler A and the scheduler B respectively, and the user B delivers the resource weight of the user B to the scheduler A and the scheduler B respectively, and the scheduler A and the scheduler B perform the above configuration.
  • the resource weight is not limited to the form shown in the embodiment of the present invention, and the resource weight ⁇ A of the user A may be 2/3, and the resource weight ⁇ B of the user B may be 1/3.
  • Scheduler A receives User A request sent by User A, and Scheduler A receives User B request sent by User B. Both user A request and user B request require storage array A to provide IOPS capability, or both user A request and user B request consume the IOPS capability provided by storage array A. Similarly, scheduler B receives the user A request sent by user A, and scheduler B receives the user B request sent by user B. When the IOPS capability of the storage array A can satisfy the user A request and the user B request, the scheduler A does not need to schedule the received user A request and the user B request according to the scheduling algorithm, and the IOPS capability of the storage array B can be satisfied.
  • Scheduler B does not need to schedule the received User A request and User B request according to the scheduling algorithm.
  • the scheduler A only needs to calculate the initial virtual time and the ending virtual time of the received user request according to the scheduling algorithm.
  • the scheduler B only needs to calculate the initial virtual time and the ending virtual of the received user request according to the scheduling algorithm. time. For the meaning of the start virtual time and the end virtual time, see the Start-time fairness queuing (SFQ) algorithm.
  • SFQ Start-time fairness queuing
  • Step 401 in the T n+1 period, the scheduler A acquires the resource C A (T n ) requested by the user A in the T n period from the home coordinator A of the user A, and from the user B.
  • the home coordinator B acquires the resource C B (T n ) that the user B requests to consume during the T n period.
  • T n represents the nth period, T n+1 represents the n+1th period;
  • C A (T n ) represents the N A, A (T n ) user A requests received by scheduler A during the T n period
  • the consumed resource provides the resources C A, A (T n ) provided by the entity A and the N A received by the scheduler B
  • the B (T n ) user A requests to consume the resources provided by the resource providing entity B C A, B (T n ) sum;
  • C B (T n) represents the T n cycles, the scheduler a received N B, a (T n) user B requests the resource consumption of resources C B entity a provides, a (T n)
  • the scheduler B receives N B, B (T n ) the sum of the resources C B, B (T n ) provided by the resource providing entity B requested by the user B.
  • the user coordinator A of the user A refers to the sum of the resources of the scheduler A processing user A requesting to consume the storage array A and the resource of the scheduler B processing user A requesting to consume the storage array A in the distributed resource system of the T n period.
  • Coordinator User A's home coordinator A is also referred to as user A's coordinator A, and the same user's home coordinator is also referred to as the user's coordinator.
  • the scheduler A obtains the resource provided by the resource A provided by the scheduler A in the T n cycle , and the A ( A n ) user A requests the consumed resource in the T n period according to the received resource provided by the user A.
  • the coordinator A acquires C A, A (T n ) and C A, B (T n ), one implementation is that the scheduler A actively sends C A, A (T n ) to the coordinator A, the scheduler B actively sends C A,B (T n ) to coordinator A.
  • the coordinator A requests the scheduler A to acquire C A,A (T n ), and the coordinator A requests the scheduler B to acquire C A,B (T n ).
  • the coordinator A can obtain C A, A (T n ) and C A, B (T n ) at a time, and the coordinator A can also communicate with the scheduler A and the scheduler B in real time, thereby obtaining the scheduler A reception in real time.
  • Each user A requests the consumed resource to provide the resource provided by the entity A, and acquires the resource provided by the resource providing entity B that each user A requests to receive by the scheduler B in real time.
  • the coordinator B obtains the resource C B (T n ) that is processed by the user B, please refer to the coordinator A, and details are not described herein again.
  • the scheduler A acquires the resource C A (T n ) requested by the user A in the T n period from the home coordinator A of the user A, and acquires the resource C requested by the user B in the T n period from the home coordinator B of the user B.
  • B (T n ) an implementation, in the T n+1 period, the coordinator A sends C A (T n ) to the scheduler A and the scheduler B, respectively, and the coordinator B goes to the scheduler A and the scheduler B, respectively.
  • scheduler B requests C A (T n ) from coordinator A
  • scheduler B requests C B (T n ) from coordinator B.
  • the above two methods are all called: in the T n+1 period, the scheduler A obtains the resource C A (T n ) requested by the user A in the T n period from the home coordinator A of the user A, and the attribution from the user B.
  • the coordinator B acquires the resource C B (T n ) that the user B requests to consume during the T n period.
  • the scheduler A stores N A, A (T n ), N B, A (T n ), C A, A (T n ), and C B, A (T n ), scheduler B stores N A, B (T n ), N B, B (T n ), C A, B (T n ), and C B, B (T n ).
  • one way may configure the coordinator as a home coordinator of a certain user; or when the user identifier is an integer, determine the user's home coordinator according to the total number of coordinators according to the user identifier. You can also use the hash function to calculate the hash value for the user ID, and then use the hash value to modulo the total number of coordinators to determine the user's home coordinator. If there are 100 users, there are 20 coordinators, the user's ID is 1 to 100, and the 20 coordinators are numbered from 1 to 20 respectively. According to the user ID, the total number of coordinators is modulo to determine the attribution of each user.
  • the coordinator is not limited by the present invention. The total number of coordinators can be less than or equal to the number of schedulers, and each coordinator can communicate with either scheduler.
  • Step 402 scheduler A uses a scheduling algorithm to schedule the received user A request according to ⁇ A , C A (T n ), C A, A (T n ), and N A, A (T n ); scheduler A is based on ⁇ B , C B (T n ), C B, A (T n ) and N B, A (T n ), scheduling the received User B request using a scheduling algorithm.
  • the scheduler B uses the scheduling algorithm to schedule the received user A request according to ⁇ A , C A (T n ), C A, B (T n ), and N A, B (T n ); the scheduler B is based on ⁇ B , C B (T n ), C B, B (T n ), and N B, B (T n ), scheduling the received User B request using a scheduling algorithm.
  • the scheduler A requests the amount of resources consumed by the user in the distributed resource system and the resource weight of the user A according to the previous period obtained from the coordinator A.
  • the number of user A requests received by the user scheduler A in the previous period and the resources provided by the resource entity A requested by the user A received by the user scheduler A in the previous period are scheduled, without the user agent in the prior art.
  • the coordinator is used to obtain the amount of resources that the user A requests to consume in the distributed resource system in the previous cycle, so that the scheduler A is not required to perform additional operations, and the resources of the scheduler A can be reduced, and the scheduler according to ⁇ A , C A (T n ), C A, A (T n ) and N A, A (T n ) schedule the user A request, thereby implementing the global scheduling requested by the user A, and ensuring the performance requirements of the user z.
  • Scheduler A requests scheduling for User B, or Scheduler B requests user A for scheduling, or Scheduler B requests scheduling for User B, which also achieves the above effects.
  • the scheduler can only obtain the user information sent by the user agent and the statistical information of the user request, and therefore, it is difficult to achieve global scheduling of the same user request.
  • the scheduler A schedules the user A request as an example.
  • the global schedule requested by the user A is that the scheduler A needs to consider not only the previous periods ⁇ A , C A , A (T n ) and N A, A ( T n ), also consider C A (T n ), so as to achieve the scheduling requested by user A, thereby ensuring the performance of user A.
  • P i A, A indicates that scheduler A receives the i-th user A request sent by user A, and storage array A provides resources for P i A, A ;
  • P i A, B indicates scheduling The device B receives the i-th user A request sent by the user A, and the storage array B provides resources for the P i A, B.
  • P k B, A indicates that scheduler A receives the kth user B request sent by user B, and storage array A provides P k B, A provides resources, P k B, and B indicates that scheduler B receives the kth sent by user B.
  • User B requests that resources are provided by storage array B for P k B,B .
  • the number of user A requests and user B requests received by the scheduler A is smaller than the IOPS capability of the storage array A, there is no need to schedule the user A request received by the scheduler A and the user B request using the scheduling algorithm.
  • the number of user A requests and user B requests received by the scheduler B is less than the IOPS capability of the storage array B, the user A request received by the scheduler B and the user B request are scheduled using the scheduling algorithm, directly by the storage array. B processing.
  • the home coordinator A of the user A needs to acquire the consumed resources requested by the user A received by each scheduler in the period T n .
  • the home coordinator A of the user A acquires the consumed resource requested by the user A received by the T n cycle scheduler A and the scheduler B as C A (T n ).
  • the home coordinator B of the user B acquires the resource consumed by the user B request received by the T n cycle scheduler A and the scheduler B as C B (T n ).
  • Coordinator A sends C A (T n ) to scheduler A and scheduler B, and coordinator B sends C B (T n ) to scheduler A and scheduler B.
  • Number scheduler A receives the user A requests the T n cycle N A, A (T n) , resource consumption memory array A is provided for the C A, A (T n) ; scheduler A receives the user B T n cycles The number of requests is N B, A (T n ), and the resources provided by the storage array A are N B, A (T n ).
  • Number scheduler B receives the user A requests the T n cycle N A, B (T n) , the consumption of resources storage arrays B provides for C A, B (T n) ; scheduler B receives the user B T n cycles The number of requests is N B,B (T n ), and the resources provided by the storage array B are C B,B (T n ).
  • the scheduler in Example A acquires resource A C A (T n) T n periods consumed from the user A requests the home A user A coordinator.
  • the scheduler A requests the received user A and the user B to use the scheduling algorithm to join the scheduling queue of the scheduler A. Scheduling.
  • the scheduler A receives the user A request and the user B request, it calculates the user A request and the user B request start virtual time and end virtual time respectively.
  • the user A request and the user B request are added to the scheduling queue, and the user A request and the user B request are sorted and waited for processing in the scheduling queue according to the initial virtual time in descending order.
  • the user request with the smallest initial virtual time is ranked in the head of the dispatch queue, and the user request with the largest virtual time is arranged at the end of the dispatch queue.
  • the head of the scheduling queue refers to the location where the storage array A first performs user request processing in the scheduling queue.
  • the storage array A takes the user request from the queue head of the scheduling queue for processing, that is, processes the user request with the smallest initial virtual time.
  • the storage array A processes the user request from the scheduling queue in the order of the initial virtual time requested by the user.
  • P i A A receive scheduler, start virtual time A is represented by S (P i A, A) , the end of the virtual time P i A, A with F (P i A, A) Said.
  • S(P i A,A ) max ⁇ v(P i A,A ),F(P i-1 A,A ) ⁇ ; wherein v(P i A,A ) represents that scheduler A receives P i A , A virtual time of scheduler A;
  • F(P i-1 A, A ) represents P i-1 A, the end virtual time of A; max ⁇ v(P i A, A ), F(P i-1 A, A ) ⁇ represents the maximum value in v(P i A, A ) and F(P i-1 A, A ).
  • the fairness queuing (SFQ) algorithm is not described in the embodiment of the present invention.
  • c(P i A,A ) represents P i A
  • A consumes the resources provided by the storage array A, which is 1 IOPS in the embodiment of the present invention
  • d(P A, A (T n+1 )) represents at T n +1 cycle, the delay value requested by each user A received by scheduler A
  • P k B A receive scheduler
  • start virtual time A is represented by S (P k B, A)
  • P k B the end of the virtual time is represented by A F (P k B, A) .
  • S(P k B,A ) max ⁇ v(P k B,A ),F(P k-1 B,A ) ⁇ ;
  • v(P k B,A ) represents that scheduler A receives P k B , A virtual time of scheduler A;
  • F(P k-1 B, A ) represents P k-1 B, the end virtual time of A;
  • max ⁇ v(P k B, A ), F(P k-1 B, A ) ⁇ represents the maximum value in v(P k B, A ) and F(P k-1 B, A ).
  • c(P k B,A ) represents P k B, and A consumes the resources provided by the storage array B, which is 1 IOPS in the embodiment of the present invention
  • d(P B, A (T n+1 )) is represented at T n +1 cycle, the delay value of each user B request received by scheduler A,
  • c(P i A, B ) represents P i A
  • B consumes resources provided by the storage array B, which is 1 IOPS in the embodiment of the present invention
  • d (P A, B (T n+1 )) is represented in T n+1 period, the delay value requested by each user A received by the scheduler B,
  • P k B B scheduler received, the start time of the virtual B is represented by S (P k B, B) , P k B, B end virtual time is represented by F (P k B, B) .
  • S(P k B,B ) max ⁇ v(P k B,B ), F(P k-1 B,B ) ⁇ , where v(P k B,B ) represents that scheduler B receives P i B , B is the virtual time of scheduler B; F(P k-1 B, B ) represents P k-1 B, the end virtual time of B ; max ⁇ v(P k B, B ), F(P k-1 B, B ) ⁇ represents the maximum value in v(P k B, B ) and F(P k-1 B, B ).
  • c(P k B,B ) represents P k B, and B consumes the resources provided by the storage array B, which is 1 IOPS in the embodiment of the present invention
  • d(P B, B (T n+1 )) represents T n+1 period, the delay value of each user B request received by scheduler B
  • P i A A receive scheduler, start virtual time A is represented by S (P i A, A) , the end of the virtual time P i A, A with F (P i A, A) Said.
  • v(P i A,A ) represents the virtual time of scheduler A when scheduler A receives P i A,A
  • d(P A,A (T n+1 )) represents the schedule at T n+1 period.
  • the delay value requested by each user A received by A F(P i-1 A, A ) represents the end virtual time of P i-1 A, A ; Represents v(P i A,A ) and The maximum value in .
  • the fairness queuing (SFQ) algorithm is not described in the embodiment of the present invention.
  • c(P i A, A ) represents P i A, and A consumes the resources provided by the storage array A, which is 1 IOPS in the embodiment of the present invention; similarly, the P k B, A received by the scheduler A start virtual time represented by S (P k B, a) , P k B, the end of the virtual time is represented by a F (P k B, a) .
  • v(P k B,A ) represents the virtual time of scheduler A when scheduler A receives P k B,A ;
  • d(P B,A (T)) represents scheduler A receives at T n+1 cycle
  • the delay value of each user B request, F(P k-1 B,A ) represents P k-1 B, the end virtual time of A;
  • the maximum value in
  • c(P k B,A ) represents P k B, and A consumes resources provided by the storage array B, which is 1 IOPS in the embodiment of the present invention.
  • the start time of the virtual B is represented by S (P i A, B)
  • the end of the virtual time P i A, B is represented by F (P i A, B)
  • v(P i A,B ) represents the virtual time of scheduler B when scheduler B receives P i A,B
  • d(P A,B (T n+1 )) represents the schedule at T n+1 period.
  • the delay value requested by each user A received by device B F(P i-1 A, B ) represents the end virtual time of P i-1 A, B ; Represents v(P i A,B ) and The maximum value in .
  • c(P i A, B ) represents P i A, and B consumes resources provided by the storage array B, which is 1 IOPS in the embodiment of the present invention.
  • P k B B scheduler received the start time of the virtual B is represented by S (P k B, B)
  • P k B, B end virtual time is represented by F (P k B, B) .
  • v(P k B,B ) represents the virtual time of scheduler B when scheduler B receives P i B,B ; d(P B,B (T n+1 )) represents reception at T n+1 period
  • Each user B requests a delay value on scheduler A
  • F(P k-1 B,B ) represents the end virtual time of P k-1 B,B ; Indicates the maximum value in v(P k B,B ) and F(P k-1 B,B ).
  • c(P k B,B ) represents P k B, and B consumes the resources provided by the storage array B, which is 1 IOPS in the embodiment of the present invention.
  • the user A received by the scheduler A is received in the T n+1 cycle.
  • the request P i A, A is P 1 A, A , P 2 A, A , P 3 A, A , P 4 A, A and P 5 A, A , respectively.
  • Scheduler B receives User A request P i A, B is 0.
  • the user B received by the scheduler A requests P k B, A in turn, P 1 B, A , P 2 B, A , P 3 B, A and P 4 B, A , a total of four.
  • the scheduler B receives the user B request P k B, and B is P 1 B, B , P 2 B, B , P 3 B, B , P 4 B, B and P 5 B, B , respectively.
  • the T n+1 period is actually the first period.
  • P 1 A, A is the first user A request received by scheduler A.
  • the received P 1 A, A is added to the scheduling queue of the scheduler A according to the scheduling algorithm for scheduling.
  • the starting virtual time of P 2 B, A is 1, and the ending virtual time is 2;
  • P 3 B the starting virtual time of A is 2, the ending virtual time is 3;
  • P 4 B the starting virtual of A The time is 3 and the end virtual time is 4.
  • P 2 B, B has a starting virtual time of 1, and the ending virtual time is 2;
  • P 3 B, B has a starting virtual time of 2, and the ending virtual time is 3;
  • P 4 B the starting virtual of B The time is 3, the end virtual time is 4;
  • P 5 B, B starts with a virtual time of 4, and ends with a virtual time of 5.
  • Scheduler A calculates the starting virtual time and ending virtual time for P 1 A, A , P 2 A, A , P 3 A, A , P 4 A, A and P 5 A, A respectively, for P 1 B, A , P 2 B, A , P 3 B, A and P 4 B, A calculate the initial virtual time and the ending virtual time respectively, according to the initial virtual time of the user request received by the scheduler A, the user request is started by virtual
  • the order from time to time is sorted and waited in the scheduling queue, that is, in the order of the first to the end of the queue, P 1 A, A , P 2 A, A , P 1 B, A , P 3 A, A , P 4 A, A , P 2 B, A , P 5 A, A , P 3 B, A and P 4 B, A , as shown in FIG.
  • the scheduler B calculates the initial virtual time and the ending virtual time for P 1 B, B , P 2 B, B , P 3 B, B , P 4 B, B and P 5 B, B , respectively, according to the scheduler B.
  • the initial virtual time requested by all users is sorted and waited in the scheduling queue according to the starting virtual time in the order of starting virtual time, that is, the order of the first to the end of the queue is P 1 B, B. , P 2 B, B , P 3 B, B , P 4 B, B and P 5 B, B B B5 , as shown in FIG.
  • the storage array B When the storage array B processes the user request, it processes in the order of P 1 B, B , P 2 B, B , P 3 B, B , P 4 B, B, and P 5 B, B. Since there is only user B request on scheduler B, there is no resource competition, so the user B request is processed in ascending order according to the initial virtual time requested by user B.
  • the number of user requests received by scheduler A and scheduler B may be much larger than the number of user requests provided in the embodiment. Users are not limited to two. The embodiment of the present invention is only for convenience of description.
  • the scheduler A receives the request of the user A as P 1 A, A , P 2 A, A , P 3 A, A , P 4 A, A and P 5 A, A , scheduler A receives user B request in order of P 1 B, A , P 2 B, A , P 3 B, A and P 4 B, A ; scheduler B receives user B request in order of P 1 B, B , P 2 B, B , P 3 B, B , P 4 B, B and P 5 B, B .
  • the scheduler A calculated N A in T n + 1 cycle received, A (T n + 1) th user A requests consume memory array A provides resources C A, A (T n + 1) And the received resource B B, A (T n ) provided by the storage array A requested by the user B requesting N B, A (T n+1 ).
  • Scheduler A is calculated in T n + 1 cycle of the received N A, A (T n + 1) th user A requests consume memory array A provides resources C A, A (T n + 1), particularly in the present
  • the resource C A, A (T n+1 ) provided by array A is 5 IOPS.
  • Coordinator A obtains that the scheduler A requests the user A to consume the resource C A provided by the storage array A in the T n+1 cycle , A (T n+1 ) is 5 IOPS, and the scheduler B processes in the T n+1 cycle.
  • the coordinator A sends a message to the scheduler A and the scheduler B, respectively, carrying C A (T n+1 ), so that the scheduler A and the scheduler B respectively obtain the user A request in the T n+1 period from the coordinator A.
  • the number of IOPS consumed is 5.
  • the coordinator B sends a message to the scheduler A and the scheduler B, respectively, carrying the message C B (T n+1 ), so that the scheduler A and the scheduler B respectively obtain the T n+1 period from the coordinator B.
  • User B requests to consume 9 IOPS.
  • the starting virtual time of P 6 B, A is 6.25, the ending virtual time is 8.5;
  • P 7 B the starting virtual time of A is 8.5, the ending virtual time is 10.75;
  • P 8 B the starting virtual of A The time is 10.75 and the end virtual time is 13.
  • the scheduler A sequentially calculates the initial virtual time and the ending virtual time of P 6 A, A , P 7 A, A , P 8 A, A and P 9 A, A , and the scheduler A sequentially calculates P 5 B, A , P 6 B, A , P 7 B, A and P 8 B, the initial virtual time and the ending virtual time of A, according to the initial virtual time of the user request received by the scheduler A, the user request is initiated by the initial virtual time.
  • the sorting waits in the dispatch queue, that is, the order of the queue from the beginning to the end of the queue is P 6 A, A , P 7 A, A , P 8 A, A , P 9 A, A , P 5 B, A , P 6 B, A , P 7 B, A and P 8 B, A , as shown in FIG.
  • the storage array A processes the user request, press P 6 A, A , P 7 A, A , P 8 A, A , P 9 A, A , P 5 B, A , P 6 B, A , P 7 B,
  • the order of A and P 8 B, A is processed.
  • scheduler B receives the first user A request P 1 A, B and the first user B requests P 6 B, B . It is assumed that scheduler B receives P 1 A, B and P 6 B, B at the same time.
  • the starting virtual time of P 7 B, B is 6.8, the ending virtual time is 8.6; P 8 B, the starting virtual time of B is 8.6, and the ending virtual time is 10.4.
  • P 2 A, B has a starting virtual time of 5.5 and an ending virtual time of 6
  • P 3 A, B has a starting virtual time of 6, and a ending virtual time of 6.5.
  • the scheduler B adds the user request to the dispatch queue, and the start virtual time requested by the user in the dispatch queue is sorted from small to large, that is, the order of the queue from the beginning to the end of the queue is P 1 A, B , P. 6 B, B , P 2 A, B , P 3 A, B , P 7 B, B and P 8 B, B , as shown in FIG.
  • the storage array B processes the user request, it processes in the order of P 1 A, B , P 6 B, B , P 2 A, B , P 3 A, B , P 7 B, B, and P 8 B, B.
  • the scheduling method of the user request of the distributed resource system is provided by the embodiment of the present invention. Further, the scenario shown in FIG. 11 may be applied, including user 1 to user H, scheduler S 1 to scheduler S M , and resource providing entity R. 1 to the resource providing entity R M , the coordinator G 1 to the coordinator G Y .
  • H is a natural number not less than 2
  • M is a natural number not less than 2
  • N is a natural number.
  • the scheduler S x communicates with the resource providing entity R x , x is any natural number from 1 to M, any user z can send a user z request to any scheduler S x , and R x provides resources for user requests received by S x
  • the coordinator G y communicates with any R x , y is any natural number from 1 to Y, and the value of Y may be less than or equal to M or greater than M. Y can also be 1, that is, there is only one coordinator in the distributed resource system.
  • the number of the coordinators may be determined by the number of the schedulers, and may also be determined by the number of users.
  • Step 1201 In the T n+1 period, S d obtains the resource C z (T n ) requested by the user z in the T n period from the coordinator G k of the user z, Where d and k are natural numbers, 1 ⁇ d ⁇ M,1 ⁇ k ⁇ Y; the resource weight of the user z is ⁇ z ; C z,x (T n ) is the N z received by S x in the T n period , x (T n ) users z request to consume the resources provided by R x ; z represents the identity of the user; It is indicated that the user z received at the T n periods S 1 to S M requests the sum of the resources provided by R 1 to R M .
  • C z,x (T n ) specifically includes the sum of resources provided by R x received by S x in the T n period , x (T n ) users z requesting consumption.
  • N z, x (T n) represents the number received in S x S x T n cycles z user request.
  • S x sends C z,x (T n ) to G k , G k according to Obtain C z (T n ).
  • G k obtains C z (T n ) is that S x actively sends C z,x (T n ) to G k , and the other way is that S x receives G k request, according to G k request, C z, x (T n ) is sent to G k .
  • G k according to Obtain C z,x (T n ).
  • S x stores N z,x (T n ) and C z,x (T n ).
  • Step 1202 S d schedules P i z, d according to ⁇ z , C z (T n ), C z,d (T n ) and N z,d (T n ), P i z,d
  • the i-th user z request received for S d , C z,d (T n ) is the resource provided by R z received by S d during the T n period , and d (T n ) users z requesting consumption of R d .
  • S d uses the scheduling algorithm to schedule P i z, d according to ⁇ z , C z (T n ), C z,d (T n ) and N z,d (T n ), thereby implementing the user z request
  • Global scheduling ensures the performance requirements of user z.
  • the step 1202 specifically includes:
  • S d uses the scheduling algorithm to schedule P i z, d according to ⁇ z , C z (T n ), C z, d (T n ) and N z,d (T n ), specifically including: S d according to ⁇ z, C z (T n ), C z, d (T n) and N z, d (T n) is calculated P i z, d virtual start time S (P i z, d) and end virtual time F(P i z,d ), adding P i z,d to the scheduling queue; the scheduling queue sorts the user requests according to the size of the starting virtual time requested by the user.
  • v (P i z, d) represents the received S d P i z, S d of d when the virtual time, c (P i z, d ) represents the P i z, d consumes resources provided by the R & lt d.
  • the fairness queuing (SFQ) algorithm is not described in the embodiment of the present invention.
  • d(P z,d (T n+1 )) represents the delay value of each P i z,d received by S d during the T n+1 period.
  • c(P i z,d ) may specifically be P i z, d consumes IOPS or bandwidth length.
  • Step 1201 specifically includes:
  • the hash function calculates that Hash(z)%Y represents the value of Y calculated using the hash function for z.
  • C K is used to send C z (T n ) to each scheduler S x .
  • the method in which the S d requests the user z to use the scheduling algorithm is applicable to the scheduling requested by any user in the distributed resource system of the embodiment of the present invention. It may be scheduled two or more user's request in the S d.
  • the scheduler S d is based on C z (T n ), C z, d (T n ), and N z,d (T) of the T n period in the entire distributed resource system according to the user z. n ), in combination with the user's weight ⁇ z , the user z requests P i z, d according to the scheduling algorithm for scheduling, thereby realizing the global scheduling of the user z request, to ensure the performance requirements of the user z.
  • the system architecture of the embodiment of the present invention can schedule user z requests without relying on user agents.
  • taking the resource as an IOPS as an example usually one user requests to consume one IOPS.
  • the resource consumed by one user request is the network bandwidth requested by the user.
  • the resource requested by the user is the network bandwidth
  • the size of the resource provided by each user requesting the consumption resource providing entity is determined by the user request.
  • scheduling the user request by using the scheduling algorithm includes calculating a starting virtual time and an ending virtual time of the user request according to the scheduling algorithm in the embodiment of the present invention, and adding the user request to the scheduling queue according to the start of the user request.
  • the virtual time size sorted in the dispatch queue.
  • S d , S x , R x , G y , and G k are only used to identify a specific device.
  • the device herein may refer to a physical device or a logical device. Similarly, the first device may also be used.
  • the second way of expressing is merely a difference in the manner of representation, and does not constitute a limitation on the solution of the present invention.
  • Example 12 of the present invention provides a scheduler S d, the scheduler S d is applied to distributed resource system, said system comprising a distributed resource scheduler S x, R x, and resource providing entity coordinator G y, wherein x is from 1 to M consecutive natural numbers, M ⁇ 2, d is x one; Y 1 to Y successive natural numbers; S x with R & lt x communications, C y with any S x communications; the The scheduler S d includes an obtaining unit 1301 and a scheduling unit 1302;
  • the obtaining unit 1301 is configured to acquire, from the coordinator G k of the user z, the resource C z (T n ) requested by the user z during the T n period, in the T n+1 cycle, Where k is a natural number, 1 ⁇ k ⁇ Y; the resource weight of the user z is ⁇ z ; C z,x (T n ) is N z,x (T n ) users received by S x in the T n period z requesting the resource provided by R x ; z represents the identity of the user;
  • the scheduling unit 1302 is configured to schedule P i z,d according to ⁇ z , C z (T n ), C z,d (T n ) and N z,d (T n ) using a scheduling algorithm, P i z resources, d d received as S i th user z requests, C z, d (T n ) is S d N z in T n cycle received, d (T n) user z requested consumed R d provided .
  • the scheduling unit 1302 is configured to schedule P i z,d according to ⁇ z , C z (T n ), C z, d (T n ), and N z,d (T n ), specifically including :
  • ⁇ z, C z (T n ), C z, d (T n) and N z, d (T n) is calculated P i z, d virtual start time S (P i z, d), and a virtual end
  • the time F(P i z,d ) adds P i z,d to the scheduling queue; the scheduling queue sorts the user requests according to the size of the starting virtual time requested by the user.
  • S(P i z,d ) max ⁇ v(P i z,d ), F(P i-1 z,d ) ⁇ , v (P i z, d) represents the received S d P i z, S d of d when the virtual time, c (P i z, d ) represents the resource P i z, d d R & lt consumed provided.
  • v (P i z, d) represents the received S d P i z, S d of d when the virtual time
  • c (P i z, d ) represents the resource P i z, d d R & lt consumed provided.
  • the fairness queuing (SFQ) algorithm is not described in the embodiment of the present invention.
  • S d uses a scheduling algorithm to schedule P i z, d according to ⁇ z , C z (T n ), C z,d (T n ) and N z,d (T n ), without relying on user agents, Scheduling user z requests, and S d scheduling P i z,d using a scheduling algorithm according to ⁇ z , C z (T n ), C z,d (T n ), and N z,d (T n ), Thereby, the global scheduling of the user z request is realized, and the performance requirement of the user z is guaranteed.
  • the distributed resource system includes a plurality of schedulers, and a first one of the plurality of schedulers is obtained from the coordinator of the first user. The user of the first user requests a sum of resources consumed by the plurality of schedulers in one cycle;
  • Determining, by the first scheduler, the sum of resources consumed by the plurality of schedulers in the previous period according to the resource weight of the first user, and the user of the first user in the previous period, in the previous period The user of the first user requests scheduling of the user request of the first user in the resource consumed by the first scheduler and the number of user requests by the first scheduler to receive the first user in the previous period.
  • the first scheduler requests, according to the resource weight of the first user, the sum of resources consumed by the plurality of schedulers in the previous period of the user of the first user, in the previous one
  • the user request of the first user is periodically
  • the resource consumed by the first scheduler and the number of user requests that the first scheduler receives the first user in the previous period are used to schedule the user request of the first user, and specifically includes:
  • the first scheduler according to the resource weight of the first user, the sum of resources consumed by the plurality of schedulers in the previous period of the user of the first user, in the previous period, Calculating the user request of the first user by the user of the first user requesting the resource consumed by the first scheduler and the number of user requests of the first scheduler receiving the first user in the previous period
  • the initial virtual time and the ending virtual time, the user request of the first user is added to the scheduling queue; the scheduling queue sorts the user requests of the first user according to the size of the initial virtual time requested by the user.
  • the distributed resource system includes a plurality of schedulers, wherein, as shown in FIG. 12, the first one of the plurality of schedulers includes an obtaining unit 1301 and a scheduling unit 1302;
  • the obtaining unit 1301 is configured to acquire, from the coordinator of the first user, a sum of resources consumed by the user of the first user in the previous cycle to be consumed by the multiple schedulers;
  • the scheduling unit 1302 is configured to: according to the resource weight of the first user, the user of the first user in the previous period requests the sum of resources consumed by the multiple schedulers, in the previous one Periodically requesting, by the user of the first user, the resource consumed by the first scheduler and the user requesting the first user by the first scheduler in the previous period to receive the user request of the first user Schedule.
  • the scheduling unit 1302 is specifically configured to: according to the resource weight of the first user, the sum of resources consumed by the user of the first user in the previous period in the previous period, in the Calculating the number of the first user's user requesting the resource consumed by the first scheduler and the first scheduler receiving the first user's user request in the previous period
  • the user request of the first user is added to the scheduling queue by the user's requested virtual time and the ending virtual time; the scheduling queue is used for the user of the first user according to the size of the initial virtual time requested by the user. Request to sort.
  • the distributed resource system is a distributed computing system
  • the resource providing entity provides computing resources for user requests
  • the distributed resource system can also be a distributed network system.
  • the providing entity provides network bandwidth for user requests; the resource providing entity can also provide memory resources for user requests.
  • the disclosed systems and methods can be implemented in other ways.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to implement the embodiment. The purpose of the case.
  • each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the functions, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable non-volatile storage medium.
  • the medium includes instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention.
  • the foregoing non-volatile storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Information Transfer Between Computers (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
  • Multi Processors (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

本发明实施例提供的分布式资源系统中用户请求的调度方法、装置及系统，在Tn+1周期，Sd从用户z的协调器Gk获取在Tn周期用户z请求消耗的资源Cz(Tn)，Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度，不需要依赖用户代理，即可对用户z请求进行调度，并且Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度，从而实现用户z请求的全局调度，保证用户z的性能需求。

Description

在分布式资源系统中用户请求的调度方法和装置 技术领域
本发明涉及信息技术领域，尤其涉及一种在分布式资源系统中用户请求的调度方法和装置。
背景技术
在分布式资源系统中，存在多个资源提供实体。资源，包括中央处理单元(Central Processing Unit,CPU)计算能力、硬盘空间、内存和网络带宽等，资源可以是物理资源也可以是虚拟资源。资源提供实体根据用户请求提供资源时，用户之间会发生资源竞争，因此对用户请求进行调度，保证用户性能变得非常重要。
如图1所示的一种用户请求调度方案，在图1所示的分布式资源系统为分布式文件系统，包括用户代理A、用户代理B、调度器A、调度器B、资源提供实体A和资源提供实体B，并且用户代理A和用户代理B之间不通信。用户代理A根据用户A请求查询分布式文件系统元数据服务器，用户代理A根据查询到的信息，判断将用户A请求发送到图1所示的哪一个调度器；同理，用户代理A根据用户B请求查询分布式文件系统元数据服务器，用户代理B根据查询到的信息，判断将用户B请求发送到图1所示的哪一个调度器。资源提供实体A和资源提供实体B为文件系统。其中，资源提供实体A用于提供资源A，资源A具体为提供每秒钟进行输入输出操作数量(Input/Output Operations Per Second,IOPS)的能力，资源提供实体B用于提供资源B，资源B具体为提供IOPS的能力。用户A和用户B分别向分布式资源系统中发送请求。例如，用户A通过用户代理A分别向资源提供实体A和资源提供实体B发送用户A请求，则资源提供实体A为用户A请求提供资源A，资源提供实体B为用户A请求提供资源B。用户B通过用户代理B分别向资源提供实体A和资源提供实体B发送用户B请求，资源提供实体A和 资源提供实体B为用户B请求提供资源。用户A向资源提供实体A和资源提供实体B分别发送用户A请求，在图1的实施方式中，资源提供实体A和资源提供实体B向用户A提供的资源均为IOPS能力。同理，资源提供实体A和资源提供实体B向用户B提供的资源也均为IOPS能力。
图1所示的分布式资源系统中，为用户A和用户B分别分配资源权重，如用户A的资源权重为2，用户B的资源权重为1。用户的资源权重表示分布式资源系统为用户分配的资源份额。用户代理A统计用户A发送到调度器A和调度器B的用户A请求数量，用户代理B统计用户B发送到调度器A和调度器B的用户B请求数量。调度器A根据用户A的资源权重、用户代理A发送到调度器A和调度器B的用户A请求的数量为用户A请求从资源提供实体A分配资源。调度器A根据用户B的资源权重、用户代理B发送到调度器A和调度器B的用户B请求的数量为用户B请求从资源提供实体A分配资源。同理，调度器B根据用户A的资源权重、用户代理A发送到调度器A和调度器B的用户A请求的数量为用户A请求从资源提供实体B分配资源。调度器B根据用户B的资源权重、用户代理B发送到调度器A和调度器B的用户B请求的数量为用户B请求从资源提供实体B分配资源。
由于用户代理A和用户代理B只适用于分布式文件系统场景，因此，图1所示架构中的方案不能够广泛应用于分布式资源系统中调度用户请求的场景。
发明内容
第一方面，本发明实施例提供了一种分布式资源系统中用户请求的调度方法，所述分布式资源系统包括调度器Sx，资源提供实体Rx和协调器Gy，其中x为1到M连续的自然数，M≥2；y为1到Y连续的自然数，Y为自然数；Sx与Rx通信，Gy与任一Sx通信；所述方法包括：
在Tn+1周期，Sd从用户z的协调器Gk获取在Tn周期用户z请求消耗的资源Cz(Tn)，
Figure PCTCN2014095381-appb-000001
其中，d和k为自然数，1≤d≤M，1≤k≤Y；所述用户z的资源权重为φz；Cz,x(Tn)为Sx在Tn周期接收的Nz,x(Tn)个用户z请求消耗的Rx提供的资源；z表示所述用户的标识；
Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度,Pi z,d为Sd接收的第i个用户z请求，Cz,d(Tn)为Sd在Tn周期接收的Nz,d(Tn)个用户z请求消耗的Rd提供的资源。
结合本发明实施例第一方面，在第一种可能的实施方式中，Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用所述调度算法对Pi z,d进行调度，具体包括：
Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)计算Pi z,d的起始虚拟时间S(Pi z,d)和结束虚拟时间F(Pi z,d)，将Pi z,d加入调度队列；所述调度队列根据用户请求的起始虚拟时间的大小对所述用户请求进行排序。
结合本发明实施例第一方面的第一种可能的实施方式，在第二种可能的实施方式中，S(Pi z,d)＝max{v(Pi z,d),F(Pi-1 z,d)},
Figure PCTCN2014095381-appb-000002
Figure PCTCN2014095381-appb-000003
Figure PCTCN2014095381-appb-000004
v(Pi z,d)表示Sd接收Pi z,d时Sd的虚拟时间，c(Pi z,d)表示Pi z,d消耗的Rd提供的资源。
结合本发明实施例第一方面的第一种可能的实施方式，在第三种可能的实施方式中，
Figure PCTCN2014095381-appb-000005
Figure PCTCN2014095381-appb-000006
Figure PCTCN2014095381-appb-000007
v(Pi z,d)表示Sd接收Pi z,d时Sd的虚拟时间，c(Pi z,d)表示Pi z,d消耗的Rd提供的资源。
第二方面，本发明实施例提供了调度器Sd，所述调度器Sd应用于 分布式资源系统，所述分布式资源系统包括调度器Sx，资源提供实体Rx和协调器Gy，其中x为1到M连续的自然数，M≥2，d为x中的一个；y为1到Y连续的自然数；Sx与Rx通信，Cy与任一Sx通信；所述调度器Sd包括中央处理单元和存储器，所述中央处理单元执行所述存储器中的可执行指令以执行本发明实施例第一方面至第一方面的第三种可能实施方式中的任一种方法。
第三方面，本发明实施例提供了调度器Sd，所述调度器Sd应用于分布式资源系统，所述分布式资源系统包括调度器Sx，资源提供实体Rx和协调器Gy，其中x为1到M连续的自然数，M≥2，d为x中的一个；y为1到Y连续的自然数；Sx与Rx通信，Cy与任一Sx通信；所述调度器Sd包括获取单元和调度单元；其中，
所述获取单元用于在Tn+1周期，从用户z的协调器Gk获取在Tn周期用户z请求消耗的资源Cz(Tn)，
Figure PCTCN2014095381-appb-000008
其中，k为自然数，1≤k≤Y；所述用户z的资源权重为φz；Cz,x(Tn)为Sx在Tn周期接收的Nz,x(Tn)个用户z请求消耗的Rx提供的资源；z表示所述用户的标识；
所述调度单元用于根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度,Pi z,d为Sd接收的第i个用户z请求，Cz,d(Tn)为Sd在Tn周期接收的Nz,d(Tn)个用户z请求消耗的Rd提供的资源。
第四方面，本发明实施例提供了一种非易失性计算机可读存储介质，所述非易失性计算机可读存储介质存储计算机指令用于执行分布式资源系统中的用户请求的调度，所述分布式资源系统包括调度器Sx，资源提供实体Rx和协调器Gy，其中x为1到M连续的自然数，M≥2；y为1到Y连续的自然数，Y为自然数；Sx与Rx通信，Cy与任一Sx通信； 调度器Sd执行所述计算机指令用于执行本发明实施例第一方面至第一方面的第三种可能实施方式中的任一种方法；其中，d为x中的一个。
第五方面，本发明实施例提供了一种分布式资源系统，所述分布式资源系统包括调度器Sx，资源提供实体Rx和协调器Gy，其中x为1到M连续的自然数，M≥2；y为1到Y连续的自然数，Y为自然数；Sx与Rx通信，Gy与任一Sx通信；
用户z的协调器Gk用于在Tn+1周期，向调度器Sd提供在Tn周期用户z请求消耗的资源Cz(Tn)，
Figure PCTCN2014095381-appb-000009
其中，d和k为自然数，1≤d≤M，1≤k≤Y；所述用户z的资源权重为φz；Cz,x(Tn)为Sx在Tn周期接收的Nz,x(Tn)个用户z请求消耗的Rx提供的资源；z表示所述用户的标识；
所述调度器Sd用于根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度,Pi z,d为Sd接收的第i个用户z请求，Cz,d(Tn)为Sd在Tn周期接收的Nz,d(Tn)个用户z请求消耗的Rd提供的资源。
第六方面，本发明实施例提供了一种分布式资源系统中资源调度方法，所述分布式资源系统包括多个调度器，
所述多个调度器中的第一调度器从所述第一用户的协调器获取在上一个周期所述第一用户的用户请求在所述多个调度器消耗的资源总和；
所述第一调度器根据所述第一用户的资源权重、在所述上一周期所述第一用户的用户请求在所述多个调度器消耗的资源总和、在所述上一个周期所述第一用户的用户请求在所述第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量对第一用户的用户请求进行调度。
结合第六方面，在第一种可能的实施方式中，所述第一调度器根据所述第一用户的资源权重、在所述上一周期所述第一用户 的用户请求在所述多个调度器消耗的资源总和、在所述上一个周期所述第一用户的用户请求在所述第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量对第一用户的用户请求进行调度，具体包括：
所述第一调度器根据所述第一用户的资源权重、在所述上一周期所述第一用户的用户请求在所述多个调度器消耗的资源总和、在所述上一周期所述第一用户的用户请求在所述第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量，计算所述第一用户的用户请求的起始虚拟时间和结束虚拟时间，将所述第一用户的用户请求加入调度队列；所述调度队列根据用户请求的起始虚拟时间的大小对所述第一用户的用户请求进行排序。
第七方面，本发明实施例提供了一种分布式资源系统，所述分布式资源系统包括多个调度器，其中，所述多个调度器中的第一调度器包括获取单元和调度单元；
所述获取单元用于从所述第一用户的协调器获取在上一个周期所述第一用户的用户请求在所述多个调度器消耗的资源总和；
所述调度单元用于根据所述第一用户的资源权重、在所述上一周期所述第一用户的用户请求在所述多个调度器消耗的资源总和、在所述上一个周期所述第一用户的用户请求在所述第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量对第一用户的用户请求进行调度。
结合第七方面，在第一种可能的实施方式中，所述调度单元具体用于根据所述第一用户的资源权重、在所述上一周期所述第一用户的用户请求在所述多个调度器消耗的资源总和、在所述上一周期所述第一用户的用户请求在所述第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量，计算所述第一用户的用户请求的起始虚拟时间和结束虚拟时间，将所述第一用户的用户请求加入调度队列；所述调度队列根据用户请求的起始虚拟时间的大小对所述第一用户的用户请求 进行排序。
本发明实施例提供的分布式资源系统中用户请求的调度方法、装置及系统，在Tn+1周期，Sd从用户z的协调器Gk获取在Tn周期用户z请求消耗的资源Cz(Tn)，Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度，不需要依赖用户代理，即可对用户z请求进行调度，并且Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度，从而实现用户z请求的全局调度，保证用户z的性能需求。
附图说明
为了更清楚地说明本发明实施例的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，下面描述中的附图仅仅是本发明的一些实施例，还可以根据这些附图获得其他的附图。
图1为现有技术分布式资源调度系统示意图；
图2为本发明实施例分布式资源系统示意图；
图3a为调度器A、资源提供实体A和协调器A关系图；
图3b为调度器A、资源提供实体A和协调器A关系图；
图4为本发明实施例方法示意图；
图5为本发明实施例分布式资源系统状态示意图；
图6为调度队列中用户请求状态示意图；
图7为调度队列中用户请求状态示意图；
图8为本发明实施例分布式资源系统状态示意图；
图9为调度队列中用户请求状态示意图；
图10为调度队列中用户请求状态示意图；
图11为本发明实施例分布式资源系统架构图；
图12为本发明实施例调度器结构示意图。
具体实施例
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚地描述。显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发明提供的实施例所获得的所有其他实施例，都属于本发明保护的范围。
如图2所示，本发明实施例提供了一种分布式资源系统。分布式资源系统，包括调度器A、调度器B、资源提供实体A、资源提供实体B、协调器A和协调器B。调度器A与资源提供实体A通信，用于调度发送到资源提供实体A的用户请求。调度器B与资源提供实体B通信，用于调度发送到资源提供实体B的用户请求。协调器A分别与调度器A和调度器B通信，协调器B与分别与调度器A和调度器B通信。本发明实施例中的资源可以为物理资源，也可以为虚拟资源，如CPU计算能力、硬盘空间、内存和网络带宽等。本发明实施例中的用户，可以指客户端，虚拟机、应用进程等，本发明实施例不作具体限定。资源提供实体为用户A请求分配的资源和为用户B请求分配的资源为同一类型的资源。资源提供实体A与资源提供实体B为用户A请求提供的资源与资源提供实体A与资源提供实体B为用户B提供的资源为同一类型的资源。本发明实施例中，资源提供实体为用户请求分配的资源和资源提供实体为用户请求提供资源具有相同的含义，还可以表述为用户请求消耗的资源提供实体的资源或者用户请求消耗的资源提供实体提供的资源。用户请求消耗的资源提供实体提供的资源是指用户请求需要消耗的资源，具体包括已经消耗的资源，如果用户请求已经由资源提供实体处理，也包括调度器接收用户请求后，还未经资源提供实体处理的用户请求需要消耗的资源。其中一种实现方式，调度器接收到用户请求，即可计算出用户请求需要消耗的资源提供实体提供的资源量。例如，一个周期，调度器A接收100个用户请求，则这100个用 户请求消耗的资源提供实体提供的资源即可根据每个用户请求需要消耗的资源提供实体提供的资源求和算出。
本发明实施例以资源为IOPS能力，资源提供实体A为存储阵列A，资源提供实体B为存储阵列B为例进行说明。
在对本发明实施例进一步描述之前，需要说明的是，图2中所示的调度器A、协调器A和资源提供实体A的之间关系只是逻辑上的呈现，具体实现中，可如图3a所示，资源提供实体A、调度器A和协调器A位于同一装置内；或者如图3b所示，资源提供实体A和调度器A位于同一装置内，而协调器A独立于资源提供实体A和调度器A，或者其他实现方式，具体实现方式在此不作限定。同理，图2中所示的调度器B、协调器B和资源提供实体B的之间关系可参照图3a和图3b。调度器可以为独立的设备或者硬件模块。调度器的功能还可以由处理器执行特定的计算机指令来实现。如调度器A、协调器A与资源提供实体A均由处理器执行特定的计算机指令来实现，则调度器A与协调器A间通信，调度器A与资源提供实体A之间通信，可以表现为指令的调用，或不同程序模块的链接等；同理，协调器可以为独立的设备或者硬件模块。协调器的功能还可以由处理器执行特定的计算机指令来实现，本发明实施例对此不作限定。本发明实施例中的调度器所实施的方法步骤，具体的一种实现，可以由计算机的处理器执行存储器中的计算机指令来实现。在此不再赘述。
如图2所示的分布式资源系统中，用户A向存储阵列A和存储阵列B发送用户A请求，则存储阵列A和存储阵列B在处理用户A请求时，均需要消耗IOPS能力。同理，用户B向存储阵列A和存储阵列B发送用户B请求，也需要消耗存储阵列A和存储阵列B的IOPS能力。特别地，当存储阵列A中的IOPS能力不能够满足同时处理用户A请求和用户B请求，则需要调度器A对接收到的用户A请求和用户B请求根据调 度算法进行调度；当存储阵列B中的IOPS能力不能够满足同时处理用户A请求和用户B请求，则需要调度器B对接收到的用户A请求和用户B请求根据调度算法进行调度。
用户A与用户B为存在资源竞争关系的用户，为保证各用户的性能，通常分布式资源系统中，会给每个用户分配资源权重。本发明实施例以用户A和用户B为例，用户A的资源权重φA为2，用户B的资源权重φB为1。则用户A和用户B的资源权重比为2:1。在调度器A上配置用户A和用户B的资源权重，在调度器B上配置用户A和用户B的资源权重。具体可由用户A向调度器A和调度器B分别下发用户A的资源权重，用户B向调度器A和调度器B分别下发用户B的资源权重，调度器A和调度器B进行上述配置。资源权重并不限于本发明实施例所示的形式，也可以用户A的资源权重φA为2/3，用户B的资源权重φB为1/3。
调度器A接收用户A发送的用户A请求，调度器A接收用户B发送的用户B请求。用户A请求和用户B请求均需要存储阵列A提供IOPS能力，或者说用户A请求和用户B请求均消耗存储阵列A提供的IOPS能力。同样，调度器B接收用户A发送的用户A请求，调度器B接收用户B发送的用户B请求。当存储阵列A的IOPS能力能够满足用户A请求和用户B请求时，则调度器A不需要将接收到的用户A请求和用户B请求根据调度算法进行调度，当存储阵列B的IOPS能力能够满足用户A请求和用户B请求时，调度器B也不需要将接收到的用户A请求和用户B请求根据调度算法进行调度。调度器A只需要根据调度算法计算接收到的用户请求的起始虚拟时间和结束虚拟时间，同理，调度器B也只需要根据调度算法计算接收到的用户请求的起始虚拟时间和结束虚拟时间。关于起始虚拟时间和结束虚拟时间的含义，请参见起始时间公平序列(Start-time fairness queuing，SFQ)算法。在需要对用户请求进行调度时，调度器A需要根据调度算法将接收到的用户A请求和用户B 请求加入调度器A的调度队列进行调度，调度器B也需要根据调度算法将接收到的用户A请求和用户B请求加入调度器B的调度队列进行调度。
如图4所示:步骤401，在Tn+1周期，调度器A从用户A的归属协调器A获取在Tn周期用户A请求消耗的资源CA(Tn)，以及从用户B的归属协调器B获取在Tn周期用户B请求消耗的资源CB(Tn)。其中Tn表示第n个周期，Tn+1表示第n+1个周期；CA(Tn)表示在Tn周期，调度器A接收的NA,A(Tn)个用户A请求消耗的资源提供实体A提供的资源CA,A(Tn)和调度器B接收的NA,B(Tn)个用户A请求消耗资源提供实体B提供的资源CA,B(Tn)之和；CB(Tn)表示在Tn周期，调度器A接收的NB,A(Tn)个用户B请求消耗的资源提供实体A提供的资源CB,A(Tn)和调度器B接收NB,B(Tn)个用户B请求消耗的资源提供实体B提供的资源CB,B(Tn)之和。其中，用户A的归属协调器A是指获取在Tn周期分布式资源系统中调度器A处理用户A请求消耗存储阵列A的资源和调度器B处理用户A请求消耗存储阵列A的资源之和的协调器。用户A的归属协调器A也称为用户A的协调器A，同理用户的归属协调器也称为用户的协调器。
调度器A在Tn周期根据接收的每个用户A请求消耗的资源提供实体A提供的资源获得调度器A在Tn周期处理的NA,A(Tn)个用户A请求消耗的资源提供实体A提供的资源CA,A(Tn)；调度器A在Tn周期根据接收的每个用户B请求消耗的资源提供实体A提供的资源，获得调度器A在Tn周期处理的NB,A(Tn)个用户B请求消耗的资源提供实体A提供的资源CB,A(Tn)；同理，调度器B在Tn周期根据接收的每个用户A请求消耗的资源提供实体B提供的资源，获得调度器B在Tn周期处理的NA,B(Tn)个用户A请求消耗的资源提供实体B提供的资源CA,B(Tn)；调度器B在Tn 周期根据接收的每个用户B请求消耗的资源提供实体B提供的资源获得调度器B在Tn周期处理的NB,B(Tn)个用户B请求消耗的资源提供实体B提供的资源CB,B(Tn)。在Tn周期处理用户A请求消耗的资源CA(Tn)＝CA,A(Tn)+CA,B(Tn)，处理用户B请求消耗的资源CB(Tn)＝CB,A(Tn)+CB,B(Tn)。
具体地，协调器A获取CA,A(Tn)和CA,B(Tn)，一种实现方式为调度器A主动向协调器A发送CA,A(Tn)，调度器B主动向协调器A发送CA,B(Tn)。另一种实现方式，协调器A向调度器A请求获取CA,A(Tn)，协调器A向调度器B请求获取CA,B(Tn)。具体地，协调器A可以一次获得CA,A(Tn)和CA,B(Tn)，协调器A也可以实时与调度器A和调度器B通信，从而实时获取调度器A接收的每个用户A请求消耗的资源提供实体A提供的资源，实时获取调度器B接收的每个用户A请求消耗的资源提供实体B提供的资源。同时，协调器B获取处理用户B请求消耗的资源CB(Tn)的方式请参照协调器A，在此不再赘述。
调度器A从用户A的归属协调器A获取在Tn周期用户A请求消耗的资源CA(Tn)，以及从用户B的归属协调器B获取在Tn周期用户B请求消耗的资源CB(Tn)，一种实现方式，在Tn+1周期，协调器A分别向调度器A和调度器B发送CA(Tn)，协调器B分别向调度器A和调度器B发送CB(Tn)；另一种实现方式，在Tn+1周期，调度器A向协调器A请求CA(Tn)，调度器A向协调器B请求CB(Tn)；调度器B向协调器A请求CA(Tn)，调度器B向协调器B请求CB(Tn)。以上两种方式皆称为：在Tn+1周期，调度器A从用户A的归属协调器A获取在Tn周期用户A请求消耗的资源CA(Tn)，以及从用户B的归属协调器B获取在Tn周期用户B请求消耗的资源CB(Tn)。无论上述任一种实现方式，通过上述描述可知，调度器A存储NA,A(Tn)、NB,A(Tn)、CA,A(Tn)和CB,A(Tn)，调度器B存储NA,B(Tn)、 NB,B(Tn)、CA,B(Tn)和CB,B(Tn)。
关于用户的归属协调器的确定，一种方式可以配置协调器为某一用户的归属协调器；或者在用户标识为整数时，根据用户标识对协调器总个数取模确定用户的归属协调器，还可以使用哈希函数对用户标识计算得到哈希值，再用哈希值对协调器总个数取模确定用户的归属协调器。如有100个用户，有20个协调器，用户的标识分别为1到100，对20个协调器从1到20分别编号，根据用户标识对协调器总个数取模确定每个用户的归属协调器，本发明对此不作限定。协调器的总个数可以小于或者等于调度器的数量，并且每一个协调器与任一调度器均可通信。
步骤402，调度器A根据φA、CA(Tn)、CA,A(Tn)和NA,A(Tn)使用调度算法对接收的用户A请求进行调度；调度器A根据φB、CB(Tn)、CB,A(Tn)和NB,A(Tn)，使用调度算法对接收的用户B请求进行调度。
同理，调度器B根据φA、CA(Tn)、CA,B(Tn)和NA,B(Tn)使用调度算法对接收的用户A请求进行调度；调度器B根据φB、CB(Tn)、CB,B(Tn)和NB,B(Tn)，使用调度算法对接收的用户B请求进行调度。
根据图4所示的方法对调度器上的用户请求进行调度时，调度器A根据从协调器A获得的上一周期用户A请求在分布式资源系统中消耗的资源量、用户A的资源权重、上一周期用户调度器A接收的用户A请求的个数以及上一周期用户调度器A接收的用户A请求消耗的资源实体A提供的资源进行调度，在没有现有技术中的用户代理的架构下，使用协调器获得上一周期用户A请求在分布式资源系统中消耗的资源量，从而不需要调度器A进行额外的操作，能够减少调度器A的资源，并且调度器根据φA、CA(Tn)、CA,A(Tn)和NA,A(Tn)对用户A请求进行调度，从而实现用户A请求的全局调度，保证用户z的性能需求。
调度器A对用户B请求调度，或调度器B对用户A请求进行调度，或者调度器B对用户B请求调度，同样达到上述效果。现有技术中基于用户代理架构，调度器只能够获得一个用户代理发送的用户请求及用户请求的统计信息，因此，难以实现同一用户请求的全局调度。
本发明实施例中，以调度器A调度用户A请求为例，用户A请求的全局调度是调度器A不仅需要考虑上一周期φA、CA,A(Tn)和NA,A(Tn)，还要考虑CA(Tn)，从而实现用户A请求的调度，进而保证用户A的性能。
具体地，本发明实施例中，Pi A,A表示调度器A接收用户A发送的第i个用户A请求，由存储阵列A为Pi A,A提供资源；Pi A,B表示调度器B接收用户A发送的第i个用户A请求，由存储阵列B为Pi A,B提供资源。Pk B,A表示调度器A接收用户B发送的第k个用户B请求，由存储阵列A为Pk B,A提供资源，Pk B,B表示调度器B接收用户B发送的第k个用户B请求，由存储阵列B为Pk B,B提供资源。本发明实施例中，φA＝2，φB＝1。
在Tn周期，当调度器A接收的用户A请求和用户B请求的数量小于存储阵列A的IOPS能力时，则不需要对调度器A接收的用户A请求和用户B请求使用调度算法进行调度，直接由存储阵列A处理。调度器B接收到的用户A请求和用户B请求的数量小于存储阵列B的IOPS能力时，也不需要对调度器B接收的用户A请求和用户B请求使用调度算法进行调度，直接由存储阵列B处理。但用户A的归属协调器A需要获取周期Tn各调度器接收的用户A请求的消耗的资源。本发明实施例中，用户A的归属协调器A获取Tn周期调度器A和调度器B接收的用户A请求的消耗的资源为CA(Tn)。同时，用户B的归属协调器B获取Tn周期调度器A和调度器B接收的用户B请求消耗的资源为CB(Tn)。协调器A将CA(Tn)发送给调度器A和调度器B，协调器B将CB(Tn)发送给调度器A和调度器B。调度器A在Tn周期接收用户A请求的数量为NA,A(Tn)，消 耗存储阵列A提供的资源为CA,A(Tn)；调度器A在Tn周期接收用户B请求的数量为NB,A(Tn)，消耗存储阵列A提供的资源为NB,A(Tn)。调度器B在Tn周期接收用户A请求的数量为NA,B(Tn)，消耗存储阵列B提供的资源为CA,B(Tn)；调度器B在Tn周期接收用户B请求的数量为NB,B(Tn)，消耗存储阵列B提供的资源为CB,B(Tn)。
在Tn+1周期，以调度器A为例，调度器A从用户A的归属协调器A获取Tn周期用户A请求消耗的资源CA(Tn)。在Tn+1周期，当存储阵列A的IOPS能力不能够满足用户A请求和用户B请求时，调度器A对接收的用户A请求和用户B请求使用调度算法加入调度器A的调度队列进行调度。调度器A接收到用户A请求和用户B请求时，分别计算用户A请求和用户B请求起始虚拟时间和结束虚拟时间。将用户A请求和用户B请求加入调度队列，将用户A请求和用户B请求按照起始虚拟时间由小到大的顺序在调度队列中排序等待处理。起始虚拟时间最小的用户请求排在调度队列的队首，起始虚拟时间最大的用户请求排列在调度队列的队尾。其中，调度队列的队首是指调度队列中存储阵列A最先进行用户请求处理的位置。存储阵列A在处理用户请求时，从调度队列的队首取用户请求进行处理，即先处理起始虚拟时间最小的用户请求。存储阵列A从调度队列中按用户请求的起始虚拟时间由小到大的顺序处理用户请求。
第一种实现方式，调度器A接收的Pi A,A的起始虚拟时间用S(Pi A,A)表示，Pi A,A的结束虚拟时间用F(Pi A,A)表示。S(Pi A,A)＝max{v(Pi A,A),F(Pi-1 A,A)}；其中，v(Pi A,A)表示调度器A接收Pi A,A时调度器A的虚拟时间；F(Pi-1 A,A)表示Pi-1 A,A的结束虚拟时间；max{v(Pi A,A),F(Pi-1 A,A)}表示v(Pi A,A)和F(Pi-1 A,A)中的最大值。
Figure PCTCN2014095381-appb-000010
Figure PCTCN2014095381-appb-000011
关于起始虚拟时间S(Pi A,A)、结束虚拟时间F(Pi A,A)和虚拟时间v(Pi A,A)的含义，请参见起始时间公平序列(Start-time fairness queuing，SFQ)算法，本发明实施例不再赘述。其中，c(Pi A,A)表示Pi A,A消耗存储阵列A提供的资源，本发明实施例为1个IOPS；d(PA,A(Tn+1))表示在Tn+1周期，调度器A接收的每个用户A请求的延迟值，
Figure PCTCN2014095381-appb-000012
同理，调度器A接收的Pk B,A的起始虚拟时间用S(Pk B,A)表示，Pk B,A的结束虚拟时间用F(Pk B,A)表示。S(Pk B,A)＝max{v(Pk B,A),F(Pk-1 B,A)}；其中，v(Pk B,A)表示调度器A接收Pk B,A时调度器A的虚拟时间；F(Pk-1 B,A)表示Pk-1 B,A的结束虚拟时间；max{v(Pk B,A),F(Pk-1 B,A)}表示v(Pk B,A)和F(Pk-1 B,A)中的最大值。
Figure PCTCN2014095381-appb-000013
其中，c(Pk B,A)表示Pk B,A消耗存储阵列B提供的资源，本发明实施例为1个IOPS；d(PB,A(Tn+1))表示在Tn+1周期，调度器A接收的每个用户B请求的延迟值，
Figure PCTCN2014095381-appb-000014
Figure PCTCN2014095381-appb-000015
调度器B接收的Pi A,B的起始虚拟时间用S(Pi A,B)表示，Pi A,B的结束虚拟时间用F(Pi A,B)表示。S(Pi A,B)＝max{v(Pi A,B),F(Pi-1 A,B)}，其中，v(Pi A,B)表示调度器B接收Pi A,B时调度器B的虚拟时间；F(Pi-1 A,B)表示Pi-1 A,B的结束虚拟时间；max{v(Pi A,B),F(Pi-1 A,B)}表示v(Pi A,B)和F(Pi-1 A,B)中的最大值。
Figure PCTCN2014095381-appb-000016
其中，c(Pi A,B)表示Pi A,B消耗存储阵列B提供的资源，在本发明实施例为1个IOPS；d(PA,B(Tn+1))表示在Tn+1周期，调度器B接收的每个用户A请求的延迟值，
Figure PCTCN2014095381-appb-000017
同理，调度器B接收的Pk B,B的起始虚拟时间用S(Pk B,B)表示，Pk B,B的结束虚拟时间用F(Pk B,B)表示。S(Pk B,B)＝max{v(Pk B,B),F(Pk-1 B,B)},其中，v(Pk B,B)表示调度器B接收Pi B,B时调度器B的虚拟时间；F(Pk-1 B,B)表示Pk-1 B,B的结束虚拟时间；max{v(Pk B,B),F(Pk-1 B,B)}表示v(Pk B,B)和F(Pk-1 B,B)中的最大值。
Figure PCTCN2014095381-appb-000018
其中，c(Pk B,B)表示Pk B,B消耗存储阵列B提供的资源，在本发明实施例中为1个IOPS；d(PB,B(Tn+1))表示在Tn+1周期，调度器B接收的每个用户B请求的延迟值，
Figure PCTCN2014095381-appb-000019
第二种实现方法，调度器A接收的Pi A,A的起始虚拟时间用S(Pi A,A)表示，Pi A,A的结束虚拟时间用F(Pi A,A)表示。
Figure PCTCN2014095381-appb-000020
Figure PCTCN2014095381-appb-000021
其中，v(Pi A,A)表示调度器A接收Pi A,A时调度器A的虚拟时间，d(PA,A(Tn+1))表示在Tn+1周期，调度器A接收的每个用户A请求的延迟值，
Figure PCTCN2014095381-appb-000022
F(Pi-1 A,A)表示Pi-1 A,A的结束虚拟时间；
Figure PCTCN2014095381-appb-000023
表示v(Pi A，A)和
Figure PCTCN2014095381-appb-000024
中的最大值。
Figure PCTCN2014095381-appb-000025
关于起始虚拟时间S(Pi A,A)、结束虚拟时间F(Pi A,A)和虚拟时间v(Pi A,A)的含义，请参见起始时间公平序列(Start-time fairness queuing，SFQ)算法，本发明实施例不再赘述。其中，c(Pi A,A)表示Pi A,A消耗存储阵列A提供的资源，在本发明实施例中为1个IOPS；同理，调度器A接收的Pk B,A的起始虚拟时间用S(Pk B,A)表示，Pk B,A的结束虚拟时间用F(Pk B,A) 表示。
Figure PCTCN2014095381-appb-000026
其中，v(Pk B,A)表示调度器A接收Pk B,A时调度器A的虚拟时间；d(PB,A(T))表示在Tn+1周期，调度器A接收的每个用户B请求的延迟值，
Figure PCTCN2014095381-appb-000027
Figure PCTCN2014095381-appb-000028
F(Pk-1 B,A)表示Pk-1 B,A的结束虚拟时间；
Figure PCTCN2014095381-appb-000029
Figure PCTCN2014095381-appb-000030
表示v(Pk B,A)和
Figure PCTCN2014095381-appb-000031
Figure PCTCN2014095381-appb-000032
中的最大值。
Figure PCTCN2014095381-appb-000033
其中，其中，c(Pk B,A)表示Pk B,A消耗存储阵列B提供的资源，在本发明实施例中为1个IOPS。
调度器B接收的Pi A,B的起始虚拟时间用S(Pi A,B)表示，Pi A,B的结束虚拟时间用F(Pi A,B)表示。
Figure PCTCN2014095381-appb-000034
Figure PCTCN2014095381-appb-000035
其中，v(Pi A,B)表示调度器B接收Pi A,B时调度器B的虚拟时间，d(PA,B(Tn+1))表示在Tn+1周期，调度器B接收的每个用户A请求的延迟值，
Figure PCTCN2014095381-appb-000036
F(Pi-1 A,B)表示Pi-1 A,B的结束虚拟时间；
Figure PCTCN2014095381-appb-000037
表示v(Pi A,B)和
Figure PCTCN2014095381-appb-000038
中的最大值。
Figure PCTCN2014095381-appb-000039
其中，c(Pi A,B)表示Pi A,B消耗存储阵列B提供的资源，在本发明实施例为1个IOPS。同理，调度器B接收的Pk B,B的起始虚拟时间用S(Pk B,B)表示，Pk B,B的结束虚拟时间用F(Pk B,B)表示。
Figure PCTCN2014095381-appb-000040
Figure PCTCN2014095381-appb-000041
其中，v(Pk B,B)表示调度器B接收Pi B,B时调度 器B的虚拟时间；d(PB,B(Tn+1))表示在Tn+1周期，接收的每个用户B请求在调度器A上的延迟值，
Figure PCTCN2014095381-appb-000042
F(Pk-1 B,B)表示Pk-1 B,B的结束虚拟时间；
Figure PCTCN2014095381-appb-000043
Figure PCTCN2014095381-appb-000044
表示v(Pk B,B)和F(Pk-1 B,B)中的最大值。
Figure PCTCN2014095381-appb-000045
Figure PCTCN2014095381-appb-000046
其中，c(Pk B,B)表示Pk B,B消耗存储阵列B提供的资源，在本发明实施例中为1个IOPS。
以第一种计算用户请求起始虚拟时间和结束虚拟时间方式为例，如图5所示，本实施例以分布式资源系统初始化后，在Tn+1周期，调度器A接收的用户A请求Pi A,A依次分别为P1 A,A、P2 A,A、P3 A,A、P4 A,A和P5 A,A，共5个。调度器B接收用户A请求Pi A,B为0个。调度器A接收的用户B请求Pk B,A依次分别为P1 B,A、P2 B,A、P3 B,A和P4 B,A，共4个。调度器B接收用户B请求Pk B,B依次分别为P1 B,B、P2 B,B、P3 B,B、P4 B,B和P5 B,B，共5个。在本发明实施例中，Tn+1周期实际为第一个周期。P1 A,A，为调度器A接收的第1个用户A请求。此时需要对接收到的P1 A,A根据调度算法加入调度器A的调度队列进行调度。首先，计算P1 A,A的起始虚拟时间S(P1 A,A)＝max{v(P1 A,A),F(Pi-1 A,A)}，由于Tn+1周期实际为第一个周期，即调度器A之前没有接收到任何Pi A,A，因此F(P0 A,A)为空，调度器A初始化完成即接收P1 A,A，则S(P1 A,A)＝v(P1 A,A)＝0。计算P1 A,A的结束虚拟时间
Figure PCTCN2014095381-appb-000047
S(P1 A,A)＝0，c(Pi A,A)＝1,φA＝2，
Figure PCTCN2014095381-appb-000048
CA,B(Tn)＝0，NA,A(Tn)＝0，则d(PA,A(Tn+1))＝0，
Figure PCTCN2014095381-appb-000049
同理，P2 A,A的起始虚拟时间为0.5，结束虚拟时间为1；P3 A,A的起始虚拟时间为1，结束虚拟时间 为1.5；P4 A,A的起始虚拟时间为1.5，结束虚拟时间为2；P5 A,A的起始虚拟时间为2，结束虚拟时间为2.5。其中，图5中
Figure PCTCN2014095381-appb-000050
表示P1 A,A的起始虚拟时间为0，结束虚拟时间为0.5，图5及后面其他图中的相类及类似标识请参考
Figure PCTCN2014095381-appb-000051
的含义，不再赘述。
对调度器A接收到的用户B请求，P1 B,A，计算P1 B,A的起始虚拟时间S(P1 B,A)＝max{v(P1 B,A),F(P0 B,A)}。由于Tn+1周期实际为第一个周期，即调度器A之前没有接收到任何Pk B,A，因此F(P0 B,A)为空，调度器A初始化完成即接收P1 B,A，v(P1 B,A)＝0。P1 B,A的结束虚拟时间
Figure PCTCN2014095381-appb-000052
c(P1 B,A)＝1，
Figure PCTCN2014095381-appb-000053
Figure PCTCN2014095381-appb-000054
CB,B(Tn)＝0，NB,A(Tn)＝0，则d(PB,A(Tn+1))＝0，φB＝1，F(P1 B,A)＝1。同理，P2 B,A的起始虚拟时间为1，结束虚拟时间为2；P3 B,A的起始虚拟时间为2，结束虚拟时间为3；P4 B,A的起始虚拟时间为3，结束虚拟时间为4。
对调度器B接收到的用户B请求，P1 B,B，此时需要对接收到的P1 B,B进行调度，计算P1 B,B的起始虚拟时间S(P1 B,B)＝max{v(P1 B,B),F(P0 B,B)}，由于Tn+1周期实际为第一个周期，即调度器B之前没有接收到任何Pk B,B，因此F(P0 B,B)为空，调度器B初始化完成即接收P1 B,B，v(P1 B,B)＝0。BB1的结束虚拟时间
Figure PCTCN2014095381-appb-000055
c(P1 B,B)＝1，
Figure PCTCN2014095381-appb-000056
CB,A(Tn)＝0，NB,B(Tn)＝0，d(PB,B(Tn+1))＝0，φB＝1，F(P1 B,B)＝1。同理，P2 B,B的起始虚拟时间为1，结束虚拟时间为2；P3 B,B的起始虚拟时间为2，结束虚拟时间为3；P4 B,B的起始虚拟时间为3，结束虚拟时间为4；P5 B,B的起始虚拟时间为4， 结束虚拟时间为5。
调度器A对P1 A,A、P2 A,A、P3 A,A、P4 A,A和P5 A,A分别计算起始虚拟时间和结束虚拟时间，对P1 B,A、P2 B,A、P3 B,A和P4 B,A分别计算起始虚拟时间和结束虚拟时间，根据调度器A接收的用户请求的起始虚拟时间，将用户请求按起始虚拟时间由小到大的顺序在调度队列排序等待处理，即调度队列中按队首到队尾的顺序依次为P1 A,A、P2 A,A、P1 B,A、P3 A,A、P4 A,A、P2 B,A、P5 A,A、P3 B,A和P4 B,A，如图6所示。存储阵列A处理用户请求时，按P1 A,A、P2 A,A、P1 B,A、P3 A,A、P4 A,A、P2 B,A、P5 A,A、P3 B,A和P4 B,A的顺序进行处理。如图6所，在用户A请求与用户B请求存在资源竞争时，调度队列中一组用户请求P1 A,A、P2 A,A和P1 B,A按照2：1的关系消耗存储阵列A提供的IOPS。调度队列中另一组用户请求P3 A,A、P4 A,A和P2 B,A也满足该要求。
调度器B对P1 B,B、P2 B,B、P3 B,B、P4 B,B和P5 B,B分别计算起始虚拟时间和结束虚拟时间，根据调度器B接收的所有用户请求的起始虚拟时间，将用户请求按起始虚拟时间由小到大的顺序在调度队列中排序等待处理，即调度队列中按队首到队尾的顺序依次为P1 B,B、P2 B,B、P3 B,B、P4 B,B和P5 B,B BB5，如图7所示。存储阵列B处理用户请求时，按P1 B,B、P2 B,B、P3 B,B、P4 B,B和P5 B,B的顺序进行处理。由于调度器B上只有用户B请求，不存在资源竞争，因此按照用户B请求的起始虚拟时间由小到大顺序处理用户B请求。
在Tn+1周期内，本领域技术人员可知，在具体实现中，调度器A和调度器B接收的用户请求数量可能会远远大于实施例中提供的用户请求数量。用户也不仅仅限于两个。本发明实施例只是为描述方便，在Tn+1周期内，调度器A接收用户A请求依次为P1 A,A、P2 A,A、P3 A,A、P4 A,A和P5 A,A，调度器A接收用户B请求依次为P1 B,A、P2 B,A、P3 B,A和P4 B,A； 调度器B接收用户B请求依次为P1 B,B、P2 B,B、P3 B,B、P4 B,B和P5 B,B。
根据前面描述的方式，调度器A计算在Tn+1周期接收的NA,A(Tn+1)个用户A请求消耗的存储阵列A提供的资源CA,A(Tn+1)，以及接收的NB,A(Tn+1)个用户B请求消耗的存储阵列A提供的的资源CB,A(Tn)。调度器A在计算在Tn+1周期接收的NA,A(Tn+1)个用户A请求消耗的存储阵列A提供的资源CA,A(Tn+1)，具体为在本发明实施例中，1个用户A请求消耗1个IOPS，则调度器A根据在Tn+1周期接收的用户A请求个数NA,A(Tn+1)＝5，则消耗的存储阵列A提供的资源CA,A(Tn+1)为5个IOPS。同理，调度器A在Tn+1周期接收的用户B请求个数NB,A(Tn+1)＝4，则消耗的存储阵列A提供的资源CB,A(Tn+1)为4个IOPS。调度器B在Tn+1周期接收的用户A请求个数NA,B(Tn+1)＝0，则消耗的存储阵列B提供的资源CA,B(Tn+1)为0个IOPS。同理，调度器B在Tn+1周期接收的用户B请求个数NB,B(Tn+1)＝5，则消耗的存储阵列B提供的资源CB,B(Tn+1)为5个IOPS。协调器A分别获得调度器A在Tn+1周期处理用户A请求消耗存储阵列A提供的资源CA,A(Tn+1)为5个IOPS，调度器B在Tn+1周期处理用户A请求消耗存储阵列B提供的资源CA,B(Tn+1)为0个IOPS；则在Tn+1周期，用户A请求消耗的资源总量CA(Tn+1)＝CA,A(Tn+1)+CA,B(Tn+1)，CA(Tn+1)＝5个IOPS。同理，协调器B获得在Tn+1周期，用户B请求消耗的资源总量CB(Tn+1)＝CB,A(Tn+1)+CB,B(Tn+1)，CB(Tn+1)＝9个IOPS。协调器A分别向调度器A和调度器B发送消息，消息中携带CA(Tn+1)，从而调度器A和调度器B分别从协调器A获得在Tn+1周期用户A请求消耗IOPS数量为5。同理，协调器B分别向调度器A和调度器B发送消息，消息中携带CB(Tn+1)，从而调度器A和调度器B分别从协调器B获得在Tn+1周期用户B请求消耗IOPS数量为9。
如图8所示，在Tn+2周期内，调度器A接受第一个用户A请求P6 A,A，P6 A,A的起始虚拟时间为S(P6 A,A)＝max{v(P6 A,A),F(P5 A,A)}，v(P6 A,A)＝F(P5 A,A)＝2.5，因此，S(P6 A,A)＝2.5。
Figure PCTCN2014095381-appb-000057
Figure PCTCN2014095381-appb-000058
Figure PCTCN2014095381-appb-000059
在Tn+1周期，CA,B(Tn+1)＝0，NA,A(Tn+1)＝5，则d(PA,A(Tn+2))＝0，c(P6 A,A)＝1，φA＝2，因此，F(P6 A,A)＝3。同理，P7 A,A的起始虚拟时间为3，结束虚拟时间为3.5；P8 A,A的起始虚拟时间为3.5，结束虚拟时间为4；P9 A,A的起始虚拟时间为4，结束虚拟时间为4.5。
在Tn+2周期内，调度器A接收第一个用户B请求P5 B,A，S(P5 B,A)＝max{v(P5 B,A),F(P4 B,A)}，v(P5 B,A),＝F(P4 B,A)＝4，因此，S(P5 B,A)＝4。P5 B,A的结束虚拟时间
Figure PCTCN2014095381-appb-000060
c(P5 B,A)＝1，
Figure PCTCN2014095381-appb-000061
CB,B(Tn+1)＝5，NB,A(Tn+1)＝4，d(PB,A(Tn+2))＝1.25，φB＝1，F(P5 B,A)＝6.25。同理，P6 B,A的起始虚拟时间为6.25，结束虚拟时间为8.5；P7 B,A的起始虚拟时间为8.5，结束虚拟时间为10.75；P8 B,A的起始虚拟时间为10.75，结束虚拟时间为13。
调度器A依次计算P6 A,A、P7 A,A、P8 A,A和P9 A,A的起始虚拟时间和结束虚拟时间，调度器A依次计算P5 B,A、P6 B,A、P7 B,A和P8 B,A的起始虚拟时间和结束虚拟时间，根据调度器A接收的用户请求的起始虚拟时间，将用户请求按起始虚拟时间由小到大的顺序在调度队列中排序等待处理，即调度队列中按队首到队尾的顺序依次为P6 A,A、P7 A,A、P8 A,A、P9 A,A、P5 B,A、P6 B,A、P7 B,A和P8 B,A，如图9所示。存储阵列A处理用户请求时，从按P6 A,A、P7 A,A、P8 A,A、P9 A,A、P5 B,A、P6 B,A、P7 B,A和P8 B,A的顺序进行处理。
在Tn+2周期内，调度器B接收第一个用户A请求P1 A,B和第一个用户B请求P6 B,B。假设调度器B在相同时刻接收到P1 A,B和P6 B,B。P6 B,B的起始虚拟时间S(P6 B,B)＝max{v(P6 B,B),F(P5 B,B)}，v(P6 B,B)＝F(P5 B,B)＝5，因此，S(P6 B,B)＝5。P6 B,B的结束虚拟时间
Figure PCTCN2014095381-appb-000062
Figure PCTCN2014095381-appb-000063
c(P6 B,B)＝1，
Figure PCTCN2014095381-appb-000064
Figure PCTCN2014095381-appb-000065
φB＝1，F(P6 B,B)＝6.8。同理，P7 B,B的起始虚拟时间为6.8，结束虚拟时间为8.6；P8 B,B的起始虚拟时间为8.6，结束虚拟时间为10.4。P1 A,B的起始虚拟时间S(P1 A,B)＝max{v(P1 A,B),F(P0 A,B)}，F(P0 A,B)＝0，v(P1 A,B)＝v(P6 B,B)＝5，因此，S(P1 A,B)＝5。P1 A,B的结束虚拟时间
Figure PCTCN2014095381-appb-000066
c(P1 A,B)＝1，
Figure PCTCN2014095381-appb-000067
Figure PCTCN2014095381-appb-000068
CA,A(Tn+1)＝5，NA,A(Tn+1)＝0，则d(PA,B(Tn+2))＝0，φA＝2，F(P1 A,B)＝5.5。同理，P2 A,B的起始虚拟时间为5.5，结束虚拟时间为6；P3 A,B的起始虚拟时间为6，结束虚拟时间为6.5。
调度器B将用户请求加入调度队列，在调度队列中用户请求的按起始虚拟时间由小到大进行排序，即调度队列中按队首到队尾的顺序依次为P1 A,B、P6 B,B、P2 A,B、P3 A,B、P7 B,B和P8 B,B，如图10所示。存储阵列B处理用户请求时，按P1 A,B、P6 B,B、P2 A,B、P3 A,B、P7 B,B和P8 B,B的顺序进行处理。
本发明实施例提供的分布式资源系统用户请求的调度方法，进一步地，可以应用如图11所示的场景，包括用户1到用户H，调度器S1到调度器SM，资源提供实体R1到资源提供实体RM，协调器G1到协调器GY。其中，H为不小于2的自然数，M为不小于2的自然数，N为自然数。调度器Sx与资源提供实体Rx通信，x为1到M的任一自然数，任一 用户z可以向任一调度器Sx发送用户z请求，Rx为Sx接收的用户请求提供资源，协调器Gy与任一Rx通信，y为1到Y的任一自然数，Y的取值可以小于等于M，也可大于M。Y也可以为1，即分布式资源系统中只有一个协调器。协调器的个数可以由调度器的数量决定，也可以由用户数量决定，本发明实施例对于不限定。对任一用户z，在Tn+1周期，向调度器Sd发送第i个用户z请求Pi z,d，i为自然数，d为自然数，1≤d≤M,Sd表示S1到SM中的一个。当Sd需要对接收的用户请求进行调度时，执行如下步骤：
步骤1201：在Tn+1周期，Sd从用户z的协调器Gk获取在Tn周期用户z请求消耗的资源Cz(Tn)，
Figure PCTCN2014095381-appb-000069
其中，d和k为自然数，1≤d≤M，1≤k≤Y；所述用户z的资源权重为φz；Cz,x(Tn)为Sx在Tn周期接收的Nz,x(Tn)个用户z请求消耗的Rx提供的资源；z表示所述用户的标识；
Figure PCTCN2014095381-appb-000070
表示在Tn周期S1至SM接收的用户z请求消耗R1至RM提供的资源之和。
步骤1201中，Cz,x(Tn)具体包括Sx在Tn周期接收的Nz,x(Tn)个用户z请求消耗的Rx提供的资源之和。Nz,x(Tn)表示Sx在Tn周期Sx接收的用户z请求的个数。Sx将Cz,x(Tn)发送给Gk，Gk根据
Figure PCTCN2014095381-appb-000071
获得Cz(Tn)。Gk获得Cz(Tn)的一种方式为Sx将Cz,x(Tn)主动发送给Gk，另一种方式为Sx接收Gk请求，根据Gk请求，将Cz,x(Tn)发送给Gk。Gk根据
Figure PCTCN2014095381-appb-000072
得到Cz,x(Tn)。Sx存储Nz,x(Tn)和Cz,x(Tn)。
步骤1202：Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度,Pi z,d为Sd接收的第i个用户z请求，Cz,d(Tn)为Sd在Tn周 期接收的Nz,d(Tn)个用户z请求消耗的Rd提供的资源。
Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用所述调度算法对Pi z,d进行调度，从而实现用户z请求的全局调度，保证用户z的性能需求。
其中，步骤1202具体包括：
Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用所述调度算法对Pi z,d进行调度，具体包括：Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)计算Pi z,d的起始虚拟时间S(Pi z,d)和结束虚拟时间F(Pi z,d)，将Pi z,d加入调度队列；所述调度队列根据用户请求的起始虚拟时间的大小对所述用户请求进行排序。
S(Pi z,d)＝max{v(Pi z,d),F(Pi-1 z,d)},
Figure PCTCN2014095381-appb-000073
Figure PCTCN2014095381-appb-000074
Figure PCTCN2014095381-appb-000075
v(Pi z,d)表示Sd接收Pi z,d时Sd的虚拟时间，c(Pi z,d)表示Pi z,d消耗Rd提供的资源。
或者，
Figure PCTCN2014095381-appb-000076
Figure PCTCN2014095381-appb-000077
Figure PCTCN2014095381-appb-000078
Figure PCTCN2014095381-appb-000079
v(Pi z,d)表示Sd接收Pi z,d时Sd的虚拟时间，c(Pi z,d)表示Pi z,d消耗Rd提供的资源。
关于起始虚拟时间S(Pi A,A)、结束虚拟时间F(Pi A,A)和虚拟时间v(Pi A,A)的含义，请参见起始时间公平序列(Start-time fairness queuing，SFQ)算法，本发明实施例不再赘述。
其中，d(Pz,d(Tn+1))表示在Tn+1周期，Sd接收的每个Pi z,d的延迟值。
当Cz(Tn)-Cz,d(Tn)和Nz,d(Tn)中任一个为0，则d(Pz,d(Tn+1))＝0。
其中，c(Pi z,d)具体可为Pi z,d消耗的IOPS或带宽长度。
步骤1201具体包括：
Gk是根据z％Y＝k确定的，其中z％Y表示用z对Y取模，或者，Gk是根据Hash(z)％Y＝k确定的，其中Hash(z)表示对z使用哈希函数计算，Hash(z)％Y表示对z使用哈希函数计算得到的值对Y取模。
图11所示的应用场景中，CK用于向每一个调度器Sx发送Cz(Tn)。
图11所示的应用场景中，Sd使用调度算法对用户z请求调度的方法适用于本发明实施例的分布式资源系统中任一用户请求的调度。在Sd中可对两个或两个以上用户的请求进行调度。
图11所示的应用场景中，调度器Sd根据用户z在整个分布式资源系统中在Tn周期的Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)，结合用户的权重φz，将用户z请求Pi z,d根据调度算法进行调度，从而实现用户z请求的全局调度，保证用户z的性能需求。
本发明实施例的系统架构不需要依赖用户代理，即可对用户z请求进行调度。
本发明实施例中，以资源为IOPS为例，通常1个用户请求消耗1个IOPS。以资源为网络带宽为例，1个用户请求消耗的资源为用户请求的网络带宽。当用户请求消耗的资源为网络带宽时，每个用户请求消耗资源提供实体提供的资源的大小由用户请求决定。
本发明实施例中，使用调度算法对用户请求进行调度包括根据本发明实施例中的调度算法计算用户请求的起始虚拟时间和结束虚拟时间，将用户请求加入调度队列，根据用户请求的起始虚拟时间大小，在调度队列中排序。在用户请求不需要调度时，则需要根据本发明实施例的调度算法计算用户请求的起始虚拟时间和结束虚拟时间，不需要加入调度队列排序。
本发明实施例中，Sd、Sx、Rx、Gy和Gk只是用来标识某一具体 设备，这里的设备可以指实体设备也可以指逻辑设备，同理，也可以用第一、第二这种方式来表述，具体只是表示方式的差异，不构成对本发明方案的限定。
如图12所示，本发明实施例提供了一种调度器Sd，调度器Sd应用于分布式资源系统，所述分布式资源系统包括调度器Sx，资源提供实体Rx和协调器Gy，其中x为1到M连续的自然数，M≥2，d为x中的一个；y为1到Y连续的自然数；Sx与Rx通信，Cy与任一Sx通信；所述调度器Sd包括获取单元1301和调度单元1302；其中，
所述获取单元1301用于在Tn+1周期，从用户z的协调器Gk获取在Tn周期用户z请求消耗的资源Cz(Tn)，
Figure PCTCN2014095381-appb-000080
其中，k为自然数，1≤k≤Y；所述用户z的资源权重为φz；Cz,x(Tn)为Sx在Tn周期接收的Nz,x(Tn)个用户z请求消耗的Rx提供的资源；z表示所述用户的标识；
所述调度单元1302用于根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度,Pi z,d为Sd接收的第i个用户z请求，Cz,d(Tn)为Sd在Tn周期接收的Nz,d(Tn)个用户z请求消耗的Rd提供的资源。
具体地，调度单元1302用于根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度，具体包括：
根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)计算Pi z,d的起始虚拟时间S(Pi z,d)和结束虚拟时间F(Pi z,d)，将Pi z,d加入调度队列；所述调度队列根据用户请求的起始虚拟时间的大小对所述用户请求进行排序。
具体地，S(Pi z,d)＝max{v(Pi z,d),F(Pi-1 z,d)},
Figure PCTCN2014095381-appb-000081
Figure PCTCN2014095381-appb-000082
Figure PCTCN2014095381-appb-000083
v(Pi z,d)表示Sd接收Pi z,d时Sd的虚拟时间，c(Pi z,d)表示Pi z,d消耗的Rd提供的资源。
具体地，
Figure PCTCN2014095381-appb-000084
Figure PCTCN2014095381-appb-000085
Figure PCTCN2014095381-appb-000086
v(Pi z,d)表示Sd接收Pi z,d时Sd的虚拟时间，c(Pi z,d)表示Pi z,d消耗的Rd提供的资源。
关于起始虚拟时间S(Pi A,A)、结束虚拟时间F(Pi A,A)和虚拟时间v(Pi A,A)的含义，请参见起始时间公平序列(Start-time fairness queuing，SFQ)算法，本发明实施例不再赘述。
具体地，当Cz(Tn)-Cz,d(Tn)和Nz,d(Tn)中任一个为0，则d(Pz,d(Tn+1))＝0。
Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度，不需要依赖用户代理，即可对用户z请求进行调度，并且Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度，从而实现用户z请求的全局调度，保证用户z的性能需求。
在另一种分布式资源系统中资源调度方法实施方式中，分布式资源系统包括多个调度器，所述多个调度器中的第一调度器从所述第一用户的协调器获取在上一个周期所述第一用户的用户请求在所述多个调度器消耗的资源总和；
所述第一调度器根据所述第一用户的资源权重、在所述上一周期所述第一用户的用户请求在所述多个调度器消耗的资源总和、在所述上一个周期所述第一用户的用户请求在所述第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量对第一用户的用户请求进行调度。
进一步地，所述第一调度器根据所述第一用户的资源权重、在所述上一周期所述第一用户的用户请求在所述多个调度器消耗的资源总和、在所述上一个周期所述第一用户的用户请求在所述 第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量对第一用户的用户请求进行调度，具体包括：
所述第一调度器根据所述第一用户的资源权重、在所述上一周期所述第一用户的用户请求在所述多个调度器消耗的资源总和、在所述上一周期所述第一用户的用户请求在所述第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量，计算所述第一用户的用户请求的起始虚拟时间和结束虚拟时间，将所述第一用户的用户请求加入调度队列；所述调度队列根据用户请求的起始虚拟时间的大小对所述第一用户的用户请求进行排序。
在另一种分布式资源系统，分布式资源系统包括多个调度器，其中，如图12所示，所述多个调度器中的第一调度器包括获取单元1301和调度单元1302；
所述获取单元1301用于从所述第一用户的协调器获取在上一个周期所述第一用户的用户请求在所述多个调度器消耗的资源总和；
所述调度单元1302用于用于根据所述第一用户的资源权重、在所述上一周期所述第一用户的用户请求在所述多个调度器消耗的资源总和、在所述上一个周期所述第一用户的用户请求在所述第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量对第一用户的用户请求进行调度。
进一步地，所述调度单元1302具体用于根据所述第一用户的资源权重、在所述上一周期所述第一用户的用户请求在所述多个调度器消耗的资源总和、在所述上一周期所述第一用户的用户请求在所述第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量，计算所述第一用户的用户请求的起始虚拟时间和结束虚拟时间，将所述第一用户的用户请求加入调度队列；所述调度队列根据用户请求的起始虚拟时间的大小对所述第一用户的用户请求进行排序。
本发明实施例提供的技术方案也可以应用到其他场景，例如，分布式资源系统为分布式计算系统，资源提供实体为用户请求提供计算资源；分布式资源系统也可以为分布式网络系统，资源提供实体为用户请求提供网络带宽；资源提供实体还可为用户请求提供内存资源。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本发明的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所公开的系统、方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方 案的目的。
另外，在本发明各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取非易失性存储介质中。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个非易失性存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本发明各个实施例所述方法的全部或部分步骤。而前述的非易失性存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、磁碟或者光盘等各种可以存储程序代码的介质。

Claims (29)

  1. 一种分布式资源系统中用户请求的调度方法，其特征在于，所述分布式资源系统包括调度器Sx，资源提供实体Rx和协调器Gy，其中x为1到M连续的自然数，M≥2；y为1到Y连续的自然数，Y为自然数；Sx与Rx通信，Gy与任一Sx通信；所述方法包括：
    在Tn+1周期，Sd从用户z的协调器Gk获取在Tn周期用户z请求消耗的资源Cz(Tn)，
    Figure PCTCN2014095381-appb-100001
    其中，d和k为自然数，1≤d≤M，1≤k≤Y；所述用户z的资源权重为φz；Cz,x(Tn)为Sx在Tn周期接收的Nz,x(Tn)个用户z请求消耗的Rx提供的资源；z表示所述用户的标识；
    Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度,Pi z,d为Sd接收的第i个用户z请求，Cz,d(Tn)为Sd在Tn周期接收的Nz,d(Tn)个用户z请求消耗的Rd提供的资源。
  2. 根据权利要求1所述的方法，其特征在于，Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用所述调度算法对Pi z,d进行调度，具体包括：
    Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)计算Pi z,d的起始虚拟时间S(Pi z,d)和结束虚拟时间F(Pi z,d)，将Pi z,d加入调度队列；所述调度队列根据用户请求的起始虚拟时间的大小对所述用户请求进行排序。
  3. 根据权利要求2所述的方法，其特征在于，S(Pi z,d)＝max{v(Pi z,d),F(Pi-1 z,d)},
    Figure PCTCN2014095381-appb-100002
    Figure PCTCN2014095381-appb-100003
    Figure PCTCN2014095381-appb-100004
    v(Pi z,d)表示Sd接收Pi z,d时Sd的虚拟时间，c(Pi z,d)表示Pi z,d消耗的Rd提供的资源。
  4. 根据权利要求3所述的方法，其特征在于，当Cz(Tn)-Cz,d(Tn)和 Nz,d(Tn)中任一个为0，则d(Pz,d(Tn+1))＝0。
  5. 根据权利要求2所述的方法，其特征在于，
    Figure PCTCN2014095381-appb-100005
    Figure PCTCN2014095381-appb-100006
    Figure PCTCN2014095381-appb-100007
    Figure PCTCN2014095381-appb-100008
    Figure PCTCN2014095381-appb-100009
    v(Pi z,d)表示Sd接收Pi z,d时Sd的虚拟时间，c(Pi z,d)表示Pi z,d消耗的Rd提供的资源。
  6. 根据权利要求5所述的方法，其特征在于，当Cz(Tn)-Cz,d(Tn)和Nz,d(Tn)中任一个为0，则d(Pz,d(Tn+1))＝0。
  7. 根据权利要求1所述的方法，其特征在于，Gk是根据z％Y＝k确定的，其中z％Y表示用z对Y取模。
  8. 根据权利要求1所述的方法，其特征在于，Gk是根据Hash(z)％Y＝k确定的，其中Hash(z)表示对z使用哈希函数计算，Hash(z)％Y表示对z使用哈希函数计算得到的值对Y取模。
  9. 一种调度器Sd，其特征在于，所述调度器Sd应用于分布式资源系统，所述分布式资源系统包括调度器Sx，资源提供实体Rx和协调器Gy，其中x为1到M连续的自然数，M≥2，d为x中的一个；y为1到Y连续的自然数；Sx与Rx通信，Cy与任一Sx通信；所述调度器Sd包括中央处理单元和存储器，所述中央处理单元执行所述存储器中的可执行指令以执行：
    在Tn+1周期，从用户z的协调器Gk获取在Tn周期用户z请求消耗的资源Cz(Tn)，
    Figure PCTCN2014095381-appb-100010
    其中，k为自然数，1≤k≤Y；所述用户z的资源权重为φz；Cz,x(Tn)为Sx在Tn周期接收的Nz,x(Tn)个用户z请求消耗的Rx提供的资源；z表示所述用户的标识；
    根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度,Pi z,d为Sd接收的第i个用户z请求，Cz,d(Tn)为Sd在Tn周期接收的Nz,d(Tn)个用户z请求消耗的Rd提供的资源。
  10. 根据权利要求9所述的调度器Sd，其特征在于，根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用所述调度算法对Pi z,d进行调度，具体包括：
    Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)计算Pi z,d的起始虚拟时间S(Pi z,d)和结束虚拟时间F(Pi z,d)，将Pi z,d加入调度队列；所述调度队列根据用户请求的起始虚拟时间的大小对所述用户请求进行排序。
  11. 根据权利要求10所述的的调度器Sd，其特征在于，S(Pi z,d)＝max{v(Pi z,d),F(Pi-1 z,d)},
    Figure PCTCN2014095381-appb-100011
    Figure PCTCN2014095381-appb-100012
    v(Pi z,d)表示Sd接收Pi z,d时Sd的虚拟时间，c(Pi z,d)表示Pi z,d消耗的Rd提供的资源。
  12. 根据权利要求11所述的的调度器Sd，其特征在于，当Cz(Tn)-Cz,d(Tn)和Nz,d(Tn)中任一个为0，则d(Pz,d(Tn+1))＝0。
  13. 根据权利要求10所述的的调度器Sd，其特征在于，
    Figure PCTCN2014095381-appb-100013
    Figure PCTCN2014095381-appb-100014
    Figure PCTCN2014095381-appb-100015
    v(Pi z,d)表示Sd接收Pi z,d时Sd的虚拟时间，c(Pi z,d)表示Pi z,d消耗的Rd提供的资源。
  14. 根据权利要求11所述的的调度器Sd，其特征在于，当Cz(Tn)-Cz,d(Tn)和Nz,d(Tn)中任一个为0，则d(Pz,d(Tn+1))＝0。
  15. 一种调度器Sd，其特征在于，所述调度器Sd应用于分布式资 源系统，所述分布式资源系统包括调度器Sx，资源提供实体Rx和协调器Gy，其中x为1到M连续的自然数，M≥2，d为x中的一个；y为1到Y连续的自然数；Sx与Rx通信，Cy与任一Sx通信；所述调度器Sd包括获取单元和调度单元；其中，
    所述获取单元用于在Tn+1周期，从用户z的协调器Gk获取在Tn周期用户z请求消耗的资源Cz(Tn)，
    Figure PCTCN2014095381-appb-100016
    其中，k为自然数，1≤k≤Y；所述用户z的资源权重为φz；Cz,x(Tn)为Sx在Tn周期接收的Nz,x(Tn)个用户z请求消耗的Rx提供的资源；z表示所述用户的标识；
    所述调度单元用于根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度,Pi z,d为Sd接收的第i个用户z请求，Cz,d(Tn)为Sd在Tn周期接收的Nz,d(Tn)个用户z请求消耗的Rd提供的资源。
  16. 根据权利要求15所述的调度器Sd，其特征在于，所述调度单元用于根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度，具体包括：
    根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)计算Pi z,d的起始虚拟时间S(Pi z,d)和结束虚拟时间F(Pi z,d)，将Pi z,d加入调度队列；所述调度队列根据用户请求的起始虚拟时间的大小对所述用户请求进行排序。
  17. 根据权利要求16所述的调度器Sd，其特征在于，S(Pi z,d)＝max{v(Pi z,d),F(Pi-1 z,d)},
    Figure PCTCN2014095381-appb-100017
    Figure PCTCN2014095381-appb-100018
    v(Pi z,d)表示Sd接收Pi z,d时Sd的虚拟时间，c(Pi z,d)表示Pi z,d消耗的Rd提供的资源。
  18. 根据权利要求17所述的调度器Sd，其特征在于，当Cz(Tn)- Cz,d(Tn)和Nz,d(Tn)中任一个为0，则d(Pz,d(Tn+1))＝0。
  19. 根据权利要求16所述的调度器Sd，其特征在于，
    Figure PCTCN2014095381-appb-100019
    Figure PCTCN2014095381-appb-100020
    Figure PCTCN2014095381-appb-100021
    Figure PCTCN2014095381-appb-100022
    v(Pi z,d)表示Sd接收Pi z,d时Sd的虚拟时间，c(Pi z,d)表示Pi z,d消耗的Rd提供的资源。
  20. 根据权利要求18所述的调度器Sd，其特征在于，当Cz(Tn)-Cz,d(Tn)和Nz,d(Tn)中任一个为0，则d(Pz,d(Tn+1))＝0。
  21. 一种非易失性计算机可读存储介质，其特征在于，所述非易失性计算机可读存储介质存储计算机指令用于执行分布式资源系统中的用户请求的调度，所述分布式资源系统包括调度器Sx，资源提供实体Rx和协调器Gy，其中x为1到M连续的自然数，M≥2；y为1到Y连续的自然数，Y为自然数；Sx与Rx通信，Cy与任一Sx通信；调度器Sd执行所述计算机指令用于执行权利要求1至8任一所述的方法，其中，d为x中的一个。
  22. 一种分布式资源系统，其特征在于，所述分布式资源系统包括调度器Sx，资源提供实体Rx和协调器Gy，其中x为1到M连续的自然数，M≥2；y为1到Y连续的自然数，Y为自然数；Sx与Rx通信，Gy与任一Sx通信；
    用户z的协调器Gk用于在Tn+1周期，向调度器Sd提供在Tn周期用户z请求消耗的资源Cz(Tn)，
    Figure PCTCN2014095381-appb-100023
    其中，d和k为自然数，1≤d≤M，1≤k≤Y；所述用户z的资源权重为φz；Cz,x(Tn)为Sx在Tn周期接收的Nz,x(Tn)个用户z请求消耗的Rx提供的资源；z表示所述用户 的标识；
    所述调度器Sd用于根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)使用调度算法对Pi z,d进行调度,Pi z,d为Sd接收的第i个用户z请求，Cz,d(Tn)为Sd在Tn周期接收的Nz,d(Tn)个用户z请求消耗的Rd提供的资源。
  23. 根据权利要求22所述的分布式资源系统，其特征在于，所述调度器Sd具体用于Sd根据φz、Cz(Tn)、Cz,d(Tn)和Nz,d(Tn)计算Pi z,d的起始虚拟时间S(Pi z,d)和结束虚拟时间F(Pi z,d)，将Pi z,d加入调度队列；所述调度队列根据用户请求的起始虚拟时间的大小对所述用户请求进行排序。
  24. 根据权利要求23所述的方法，其特征在于，S(Pi z,d)＝max{v(Pi z,d),F(Pi-1 z,d)},
    Figure PCTCN2014095381-appb-100024
    Figure PCTCN2014095381-appb-100025
    v(Pi z,d)表示Sd接收Pi z,d时Sd的虚拟时间，c(Pi z,d)表示Pi z,d消耗的Rd提供的资源。
  25. 根据权利要求23所述的方法，其特征在于，
    Figure PCTCN2014095381-appb-100026
    Figure PCTCN2014095381-appb-100027
    Figure PCTCN2014095381-appb-100028
    Figure PCTCN2014095381-appb-100029
    v(Pi z,d)表示Sd接收Pi z,d时Sd的虚拟时间，c(Pi z,d)表示Pi z,d消耗的Rd提供的资源。
  26. 一种分布式资源系统中资源调度方法，其特征在于，所述分布式资源系统包括多个调度器，
    所述多个调度器中的第一调度器从所述第一用户的协调器获取在上一个周期所述第一用户的用户请求在所述多个调度器消耗的资源总和；
    所述第一调度器根据所述第一用户的资源权重、在所述上一 周期所述第一用户的用户请求在所述多个调度器消耗的资源总和、在所述上一个周期所述第一用户的用户请求在所述第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量对第一用户的用户请求进行调度。
  27. 根据权利要求26所述的方法，其特征在于，所述第一调度器根据所述第一用户的资源权重、在所述上一周期所述第一用户的用户请求在所述多个调度器消耗的资源总和、在所述上一个周期所述第一用户的用户请求在所述第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量对第一用户的用户请求进行调度，具体包括：
    所述第一调度器根据所述第一用户的资源权重、在所述上一周期所述第一用户的用户请求在所述多个调度器消耗的资源总和、在所述上一周期所述第一用户的用户请求在所述第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量，计算所述第一用户的用户请求的起始虚拟时间和结束虚拟时间，将所述第一用户的用户请求加入调度队列；所述调度队列根据用户请求的起始虚拟时间的大小对所述第一用户的用户请求进行排序。
  28. 一种分布式资源系统，其特征在于，所述分布式资源系统包括多个调度器，其中，所述多个调度器中的第一调度器包括获取单元和调度单元；
    所述获取单元用于从所述第一用户的协调器获取在上一个周期所述第一用户的用户请求在所述多个调度器消耗的资源总和；
    所述调度单元用于根据所述第一用户的资源权重、在所述上一周期所述第一用户的用户请求在所述多个调度器消耗的资源总和、在所述上一个周期所述第一用户的用户请求在所述第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量对第一用户的用户请求进行调度。
  29. 根据权利要求28所述的分布式资源系统，其特征在于，所述调度单元具体用于根据所述第一用户的资源权重、在所述上 一周期所述第一用户的用户请求在所述多个调度器消耗的资源总和、在所述上一周期所述第一用户的用户请求在所述第一调度器消耗的资源和在所述上一周期所述第一调度器接收所述第一用户的用户请求的数量，计算所述第一用户的用户请求的起始虚拟时间和结束虚拟时间，将所述第一用户的用户请求加入调度队列；所述调度队列根据用户请求的起始虚拟时间的大小对所述第一用户的用户请求进行排序。
PCT/CN2014/095381 2014-12-29 2014-12-29 在分布式资源系统中用户请求的调度方法和装置 WO2016106516A1 (zh)

Priority Applications (5)

Application Number Priority Date Filing Date Title
PCT/CN2014/095381 WO2016106516A1 (zh) 2014-12-29 2014-12-29 在分布式资源系统中用户请求的调度方法和装置
CN201480001882.1A CN105934928B (zh) 2014-12-29 2014-12-29 在分布式资源系统中用户请求的调度方法、装置和系统
EP14891590.3A EP3057288A4 (en) 2014-12-29 2014-12-29 METHOD AND APPARATUS FOR PLANNING USER REQUEST IN DISTRIBUTED RESOURCE SYSTEM
US14/932,628 US9575691B2 (en) 2014-12-29 2015-11-04 Scheduling user requests in a distributed resource system having a plurality of schedulers and coordinators
US15/391,643 US10127079B2 (en) 2014-12-29 2016-12-27 Scheduling user requests in a distributed resource system having plurality of schedulers and coordinators

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2014/095381 WO2016106516A1 (zh) 2014-12-29 2014-12-29 在分布式资源系统中用户请求的调度方法和装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US14/932,628 Continuation US9575691B2 (en) 2014-12-29 2015-11-04 Scheduling user requests in a distributed resource system having a plurality of schedulers and coordinators

Publications (1)

Publication Number Publication Date
WO2016106516A1 true WO2016106516A1 (zh) 2016-07-07

Family

ID=56164271

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2014/095381 WO2016106516A1 (zh) 2014-12-29 2014-12-29 在分布式资源系统中用户请求的调度方法和装置

Country Status (4)

Country Link
US (2) US9575691B2 (zh)
EP (1) EP3057288A4 (zh)
CN (1) CN105934928B (zh)
WO (1) WO2016106516A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114170335A (zh) * 2021-10-18 2022-03-11 深圳追一科技有限公司 数字人视频的生成方法、装置、计算机设备及存储介质

Families Citing this family (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9600312B2 (en) 2014-09-30 2017-03-21 Amazon Technologies, Inc. Threading as a service
US9678773B1 (en) 2014-09-30 2017-06-13 Amazon Technologies, Inc. Low latency computational capacity provisioning
US9146764B1 (en) 2014-09-30 2015-09-29 Amazon Technologies, Inc. Processing event messages for user requests to execute program code
US9537788B2 (en) 2014-12-05 2017-01-03 Amazon Technologies, Inc. Automatic determination of resource sizing
US9588790B1 (en) 2015-02-04 2017-03-07 Amazon Technologies, Inc. Stateful virtual compute system
US9733967B2 (en) 2015-02-04 2017-08-15 Amazon Technologies, Inc. Security protocols for low latency execution of program code
US11132213B1 (en) 2016-03-30 2021-09-28 Amazon Technologies, Inc. Dependency-based process of pre-existing data sets at an on demand code execution environment
US10102040B2 (en) 2016-06-29 2018-10-16 Amazon Technologies, Inc Adjusting variable limit on concurrent code executions
US11948009B2 (en) * 2017-01-13 2024-04-02 Boe Technology Group Co., Ltd. Method and device for operating instance resources based on instance arranging property
US10853115B2 (en) 2018-06-25 2020-12-01 Amazon Technologies, Inc. Execution of auxiliary functions in an on-demand network code execution system
US11099870B1 (en) 2018-07-25 2021-08-24 Amazon Technologies, Inc. Reducing execution times in an on-demand network code execution system using saved machine states
US11243953B2 (en) 2018-09-27 2022-02-08 Amazon Technologies, Inc. Mapreduce implementation in an on-demand network code execution system and stream data processing system
US11943093B1 (en) 2018-11-20 2024-03-26 Amazon Technologies, Inc. Network connection recovery after virtual machine transition in an on-demand network code execution system
US11861386B1 (en) 2019-03-22 2024-01-02 Amazon Technologies, Inc. Application gateways in an on-demand network code execution system
US11119809B1 (en) 2019-06-20 2021-09-14 Amazon Technologies, Inc. Virtualization-based transaction handling in an on-demand network code execution system
US11159528B2 (en) 2019-06-28 2021-10-26 Amazon Technologies, Inc. Authentication to network-services using hosted authentication information
US11190609B2 (en) 2019-06-28 2021-11-30 Amazon Technologies, Inc. Connection pooling for scalable network services
US11119826B2 (en) 2019-11-27 2021-09-14 Amazon Technologies, Inc. Serverless call distribution to implement spillover while avoiding cold starts
US11714682B1 (en) 2020-03-03 2023-08-01 Amazon Technologies, Inc. Reclaiming computing resources in an on-demand code execution system
US11550713B1 (en) 2020-11-25 2023-01-10 Amazon Technologies, Inc. Garbage collection in distributed systems using life cycled storage roots
US11593270B1 (en) 2020-11-25 2023-02-28 Amazon Technologies, Inc. Fast distributed caching using erasure coded object parts
US11388210B1 (en) 2021-06-30 2022-07-12 Amazon Technologies, Inc. Streaming analytics using a serverless compute system

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102495758A (zh) * 2011-12-05 2012-06-13 中南大学 分布式高性能计算环境中实时任务的调度方法

Family Cites Families (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6480861B1 (en) * 1999-02-26 2002-11-12 Merrill Lynch, Co., Inc Distributed adaptive computing
US6519595B1 (en) * 1999-03-02 2003-02-11 Nms Communications, Inc. Admission control, queue management, and shaping/scheduling for flows
US6888842B1 (en) * 1999-10-05 2005-05-03 Nortel Networks Limited Scheduling and reservation for dynamic resource control systems
US6434631B1 (en) * 1999-10-15 2002-08-13 Lucent Technologies Inc. Method and system for providing computer storage access with quality of service guarantees
CA2393740A1 (en) * 1999-12-08 2001-06-14 The University Of British Columbia Weighted fair queuing scheduler
US6747976B1 (en) * 2000-05-23 2004-06-08 Centre for Wireless Communications of The National University of Singapore Distributed scheduling architecture with efficient reservation protocol and dynamic priority scheme for wireless ATM networks
US6909691B1 (en) * 2000-08-07 2005-06-21 Ensim Corporation Fairly partitioning resources while limiting the maximum fair share
KR20040052012A (ko) * 2002-12-13 2004-06-19 한국전자통신연구원 고속 패킷 망을 위한 패킷 스케줄링 시스템 및 방법
US20050147103A1 (en) * 2003-04-11 2005-07-07 Samsung Electronics Co., Ltd. Packet scheduling method and apparatus
US7665090B1 (en) * 2004-03-08 2010-02-16 Swsoft Holdings, Ltd. System, method, and computer program product for group scheduling of computer resources
US7761875B2 (en) * 2005-06-10 2010-07-20 Hewlett-Packard Development Company, L.P. Weighted proportional-share scheduler that maintains fairness in allocating shares of a resource to competing consumers when weights assigned to the consumers change
US7823154B2 (en) * 2005-09-16 2010-10-26 Hewlett-Packard Development Company, L.P. System and method for providing, by a plurality of schedulers, differentiated service to consumers of distributed resources
CN101035389A (zh) * 2006-03-08 2007-09-12 上海交通大学 在无源光网络的远端设备内进行带宽分配的系统和方法
US8165033B1 (en) * 2007-08-30 2012-04-24 Altera Corporation Method and apparatus for performing generalized processor sharing scheduling
US8392633B2 (en) * 2008-06-25 2013-03-05 Hewlett-Packard Development Company, L.P. Scheduling requesters of a shared storage resource
US9244732B2 (en) * 2009-08-28 2016-01-26 Vmware, Inc. Compensating threads for microarchitectural resource contentions by prioritizing scheduling and execution
US8667493B2 (en) * 2010-05-07 2014-03-04 Advanced Micro Devices, Inc. Memory-controller-parallelism-aware scheduling for multiple memory controllers
US8819687B2 (en) * 2010-05-07 2014-08-26 Advanced Micro Devices, Inc. Scheduling for multiple memory controllers
US20130044755A1 (en) * 2011-08-16 2013-02-21 Futurewei Technologies, Inc. Scalable Packet Scheduling Policy for Vast Number of Sessions
CN103973594B (zh) * 2014-05-09 2017-07-28 清华大学 基于多资源的任务调度方法

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102495758A (zh) * 2011-12-05 2012-06-13 中南大学 分布式高性能计算环境中实时任务的调度方法

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
DONG, RONGSHENG ET AL.: "Research on Fault-tolerance Model of Object-based distributed Real-Time System", COMPUTER ENGINEERING AND APPLICATIONS, 31 March 2004 (2004-03-31), XP008182361 *
See also references of EP3057288A4 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114170335A (zh) * 2021-10-18 2022-03-11 深圳追一科技有限公司 数字人视频的生成方法、装置、计算机设备及存储介质
CN114170335B (zh) * 2021-10-18 2022-10-04 深圳追一科技有限公司 数字人视频的生成方法、装置、计算机设备及存储介质

Also Published As

Publication number Publication date
EP3057288A4 (en) 2016-10-26
US20160188367A1 (en) 2016-06-30
US9575691B2 (en) 2017-02-21
CN105934928A (zh) 2016-09-07
CN105934928B (zh) 2017-07-07
US10127079B2 (en) 2018-11-13
US20170109200A1 (en) 2017-04-20
EP3057288A1 (en) 2016-08-17

Similar Documents

Publication Publication Date Title
WO2016106516A1 (zh) 在分布式资源系统中用户请求的调度方法和装置
Kliazovich et al. CA-DAG: Modeling communication-aware applications for scheduling in cloud computing
Tan et al. Coupling task progress for mapreduce resource-aware scheduling
US8434085B2 (en) Scalable scheduling of tasks in heterogeneous systems
EP3129880A1 (en) Method and device for augmenting and releasing capacity of computing resources in real-time stream computing system
WO2022247105A1 (zh) 一种任务调度方法、装置、计算机设备和存储介质
US20110161965A1 (en) Job allocation method and apparatus for a multi-core processor
CN109947551A (zh) 一种多轮次任务分配方法、边缘计算系统及其存储介质
US10733022B2 (en) Method of managing dedicated processing resources, server system and computer program product
CN114675964A (zh) 基于联邦决策树模型训练的分布式调度方法、系统及介质
CN108259568B (zh) 任务分配方法、装置、计算机可读存储介质及电子设备
CN113138802A (zh) 命令分发装置、方法、芯片、计算机设备及存储介质
CN109491775B (zh) 一种用于边缘计算环境下的任务处理与调度方法
WO2022110796A1 (zh) 云服务请求响应方法及装置、电子设备和存储介质
CN105049372A (zh) 一种扩展消息中间件吞吐量的方法及系统
Ghouma et al. Context aware resource allocation and scheduling for mobile cloud
CN104572276A (zh) 一种基于云计算的并发数据处理方法
Mana A feature based comparison study of big data scheduling algorithms
KR101491689B1 (ko) 다중 사용자를 위한 자원 할당 방법 및 장치
CN115391053B (zh) 基于cpu和gpu混合计算的在线服务方法及装置
CN107689979B (zh) 一种下载请求处理方法和处理设备
CN111858019B (zh) 任务调度方法、装置及计算机可读存储介质
CN109688177B (zh) 一种数据同步方法及装置、设备、存储介质
Zhang et al. An Optimized Algorithm for Reduce Task Scheduling.
US20140139533A1 (en) Graphic processing unit virtual apparatus, graphic processing unit host apparatus, and graphic processing unit program processing methods thereof

Legal Events

Date Code Title Description
REEP Request for entry into the european phase

Ref document number: 2014891590

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2014891590

Country of ref document: EP

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14891590

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE