CN107045456A

CN107045456A - A kind of resource allocation methods and explorer

Info

Publication number: CN107045456A
Application number: CN201610080980.XA
Authority: CN
Inventors: 辛现银
Original assignee: Huawei Technologies Co Ltd
Current assignee: Honor Device Co Ltd
Priority date: 2016-02-05
Filing date: 2016-02-05
Publication date: 2017-08-15
Anticipated expiration: 2036-02-05
Also published as: WO2017133351A1; CN107045456B

Abstract

The embodiment of the present invention provides a kind of resource allocation methods and explorer, for improving resource utilization, and/or, the execution efficiency for lifting user job.This method includes：The operation that client device is submitted is received, and is multiple tasks by the breakdown of operation, wherein, each task in the plurality of task is each equipped with corresponding resources requirement；Estimate the run time of each task；According to the corresponding resources requirement of each task and run time, with reference to default scheduling strategy, determine first point of coordination type of the plurality of task, first point of coordination type is used to indicate the distribution situation run calculate node in of the plurality of task in multiple calculate nodes, and the scheduling strategy includes at least one of resource utilization preference strategy and efficiency comes first strategy；The plurality of task is assigned to the running in calculate node of the plurality of task according to first point of coordination type.The present invention is applied to High-Performance Computing Cluster field.

Description

A kind of resource allocation methods and explorer

Technical field

The present invention relates to High-Performance Computing Cluster field, more particularly to a kind of resource allocation methods and explorer.

Background technology

The high speed development of internet generates substantial amounts of user data, and distributed treatment is then processing large-scale dataset Standard approach.Its typical module is by a user job (English：Job) be decomposed into it is a series of can distributed operation task (English：Task), and pass through scheduler (English：Scheduler) by these task schedulings to suitable node (English：node) Upper carry out computing.After task run is completed, the operation result of task is collected, arranged, form the final result of operation defeated Go out.

Scheduler is the Coupling point of cluster resource and user job.The quality of scheduling strategy directly affects whole cluster The execution efficiency of resource utilization and user job.The scheduling strategy for the Hadoop system being widely used at present is as shown in Figure 1.Its In, Hadoop is by the Task for having resource requirement according to certain strategy, fair (the English full name of such as primary resource：dominant Resource fairness, english abbreviation：DRF it is) tactful) queue up, and each node reports the resource on this node by heartbeat Amount, and trigger distribution mechanism.If the stock number on the node meets first Task demand, scheduler just lays the Task On this node.However, the scheduling strategy has taken into consideration only the fairness of resource, and it is relatively simple, can not be according to different scenes Need neatly to select resource utilization preference strategy and efficiency comes first strategy to carry out resource allocation, so that cluster can not be caused The utilization rate of resource is higher, and/or, the execution efficiency of user job is higher.

The content of the invention

The embodiment of the present invention provides a kind of resource allocation methods and explorer, for flexibly selecting resource utilization excellent First strategy carries out resource allocation with efficiency comes first strategy, so that resource utilization is improved, and/or, lift holding for user job Line efficiency.

To reach above-mentioned purpose, the embodiment of the present invention provides following technical scheme：

First aspect includes there is provided the resource allocation methods in a kind of distributed computing system, the distributed computing system Multiple calculate nodes, this method includes：The operation that client device is submitted is received, and is multiple tasks by the breakdown of operation, its In, each task in the plurality of task is each equipped with corresponding resources requirement；Estimate the run time of each task；Root According to the corresponding resources requirement of each task and run time, with reference to default scheduling strategy, the first of the plurality of task is determined Divide coordination type, first point of coordination type is used to indicate the plurality of task running in calculate node in multiple calculate nodes Distribution situation, the scheduling strategy includes at least one of resource utilization preference strategy and efficiency comes first strategy；Will be the plurality of Task is assigned to the running in calculate node of the plurality of task according to first point of coordination type.

Based on resource allocation methods provided in an embodiment of the present invention, in the resource allocation methods, client device is being received The operation of submission, and by the breakdown of operation for multiple the having a configuration of corresponding resources requirement of the tasks after, also estimate each appoint The run time of business, and according to the resources requirement and run time of each task, with reference to default scheduling strategy, determine that this is more First point of coordination type of individual task, is then assigned to transporting for the plurality of task by the plurality of task according to first point of coordination type In row calculate node.Wherein, first point of coordination type is used to indicate that the plurality of task runs calculating section in the plurality of task Distribution situation on point, scheduling strategy includes at least one of resource utilization preference strategy and efficiency comes first strategy.Also It is to say, the program considers the factor of the run time of each task, and by space requirement (i.e. the resources requirement of task) , can be according to corresponding scheduling strategy when fixed Task is dispatched on corresponding node with time demand (i.e. the time of task) Resource utilization preference strategy and efficiency comes first strategy is flexibly selected to carry out resource allocation so that finally using resource utilization The higher point coordination type of higher and/or efficiency.On the one hand, due to the higher point coordination type of resource utilization can be used, you can It is scheduled for the Task combinations for causing those node resource utilization rates higher by scheduling strategy on node, therefore the distribution side The problem of case can effectively mitigate resource fragmentation in the prior art, so as to lift the resource utilization of cluster.On the other hand, due to can With using the higher point coordination type of efficiency, you can to cause those Job execution times most short Task to combine by scheduling strategy It is scheduled on node, therefore compared with prior art, the allocative decision can significantly shorten the Job execution time, lifting operation is held Line efficiency.To sum up, resource allocation methods provided in an embodiment of the present invention can flexibly select resource according to corresponding scheduling strategy Utilization rate preference strategy and efficiency comes first strategy carry out resource allocation, so as to improve resource utilization, and/or, can be with Lift the execution efficiency of user job.

With reference in a first aspect, in first aspect in the first possible implementation, if scheduling strategy is resource utilization Preference strategy, then first point of coordination type is specially each calculate node run in calculate node so that the plurality of task The maximum point coordination type of single node resource utilization.

With reference in a first aspect, in second of possible implementation of first aspect, if scheduling strategy is efficiency comes first plan Slightly, then first point of coordination type is specially the most fast point coordination type of overall execution speed for causing operation.

With reference to first aspect or first aspect, second of the first possible implementation or first aspect are possible realizes Mode, in first aspect in the third possible implementation, the run time of each task of estimation can specifically be wrapped Include：For each task, handled according to the operation below for first task：By the hard information and sample of first task The hard information of historic task in storehouse is matched；If the match is successful, the history matched according to the hard information with first task is appointed The run time of the history run time Estimate first task of business.

Specifically, the hard information in the embodiment of the present invention can specifically include homework type, perform the information such as user.

It should be noted that the embodiment of the present invention is only exemplary to provide a kind of specific reality for estimating Runtime It is existing, it is, of course, also possible to estimate the run time of task otherwise, such as, by way of pre-run task.That is, lead to Cross and be previously run the estimation that a bit of job instances obtain the accurate entire run time.In addition, the follow-up work of same operation Run time can also obtain more accurate estimation with reference to the run time of operation task.The embodiment of the present invention is appointed to estimation The specific implementation of business run time is not limited.

With reference to any one possible implementation in first aspect to first aspect the third possible implementation, In the 4th kind of possible implementation of first aspect, according to the corresponding resources requirement of each task and run time, knot Default scheduling strategy is closed, before first point of coordination type for determining the plurality of task, in addition to：By the plurality of task according to resource Species classified, obtain an at least generic task；

It is described according to the corresponding resources requirement of each task and run time, with reference to default scheduling strategy, really First point of coordination type of fixed the multiple task, is specifically included：For every generic task in this at least a generic task, under Face is handled for the operation of the first generic task：According to the corresponding resources requirement of each task and operation in the first generic task Time, with reference to default scheduling strategy, determine the son point coordination type of the first generic task, the son point coordination type for indicate this first The distribution situation run calculate node in of the generic task in multiple calculate nodes；Every class in this at least a generic task is appointed The combination of the son of business point coordination type is defined as first point of coordination type of the plurality of task.

Because resource allocation methods provided in an embodiment of the present invention first can enter multiple tasks according to the species of resource Row classification, and then carries out resource allocation respectively for each generic task, that is to say, that can consider simultaneously for isomeric group and The resource allocation of special resource demand operation, thus with wider universality and more preferable general performance.

Optionally, it is contemplated that the estimation meeting of run time and actual conditions would generally produce some deviations.If to these Deviation is not controlled, then with the extension of time, the predistribution result of operation resource may differ by with desired result it is more and more big. Therefore, in resource allocation methods provided in an embodiment of the present invention, it may be incorporated into Variation mechanism (redistributing).I.e.：

With reference to any one possible implementation in the 4th kind of possible implementation of first aspect to first aspect, In the 5th kind of possible implementation of first aspect, that the plurality of task is assigned into this according to first point of coordination type is more described After the running in calculate node of individual task, in addition to：According to first point of coordination type, all timess being waited for are determined Business operates in the first integral dispensing target function value when on distributed node；According to all tasks pair being waited for The resources requirement and run time answered, with reference to default scheduling strategy, determine the second of all being waited for for tasks Point coordination type, second point of coordination type is used to indicating that all being waited for for tasks all to be waited at this The distribution situation run in calculate node of task；According to second point of coordination type, all timess being waited for are determined Business operates in the second integral dispensing target function value when on distributed node；If the second integral dispensing target function value is more than All being waited for for tasks are assigned to all places by the first integral dispensing target function value according to second point of coordination type Run in the task of wait state in calculate node.

By above-mentioned Variation mechanism, the predistribution result of operation resource can be caused to be evolved to more preferable direction.

Second aspect includes there is provided a kind of explorer, the explorer：Receiving unit, resolving cell, estimation are single Member, determining unit and allocation unit：Receiving unit, the operation for receiving client device submission；Resolving cell, for should Breakdown of operation is multiple tasks, wherein, each task in the plurality of task is each equipped with corresponding resources requirement；Estimation Unit, the run time for estimating each task；Determining unit, for according to the corresponding resources requirement of each task and fortune The row time, with reference to default scheduling strategy, first point of coordination type of the plurality of task is determined, first point of coordination type is used to indicate The distribution situation run calculate node in of the plurality of task in multiple calculate nodes, the scheduling strategy includes the utilization of resources At least one of rate preference strategy and efficiency comes first strategy；Allocation unit, for distributing position according to first by the plurality of task Shape is assigned to the running in calculate node of the plurality of task.

Based on explorer provided in an embodiment of the present invention, the explorer is receiving the work that client device is submitted Industry, and by the breakdown of operation for multiple the having a configuration of corresponding resources requirement of the tasks after, also estimate the operation of each task Time, and according to the resources requirement and run time of each task, with reference to default scheduling strategy, determine the plurality of task First point of coordination type, the calculating that runs that the plurality of task then is assigned into the plurality of task according to first point of coordination type is saved Point on.Wherein, first point of coordination type is used to indicate point that runs calculate node in of the plurality of task in the plurality of task Cloth situation, scheduling strategy includes at least one of resource utilization preference strategy and efficiency comes first strategy.That is, the money Source manager considers the factor of the run time of each task when carrying out resource allocation, and by space requirement (i.e. task Resources requirement) and time demand (i.e. the time of task) fixed Task when being dispatched on corresponding node, being capable of basis Corresponding scheduling strategy flexibly selects resource utilization preference strategy and efficiency comes first strategy to carry out resource allocation so that final Using point coordination type that resource utilization is higher and/or efficiency is higher.On the one hand, due to that resource utilization can be used higher Divide coordination type, you can node is scheduled for the Task combinations for causing those node resource utilization rates higher by scheduling strategy On, therefore the explorer is the problem of can effectively mitigate resource fragmentation in the prior art, so as to lift the utilization of resources of cluster Rate.On the other hand, due to the higher point coordination type of efficiency can be used, you can to cause those Job executions by scheduling strategy Time, most short Task combinations were scheduled on node, therefore compared with prior art, the explorer can significantly shorten work Industry performs the time, lifts Job execution efficiency.To sum up, explorer provided in an embodiment of the present invention can be according to corresponding tune Degree policy-flexible selects resource utilization preference strategy and efficiency comes first strategy to carry out resource allocation, so as to improve resource Utilization rate, and/or, the execution efficiency of user job can be lifted.

With reference to second aspect, in second aspect in the first possible implementation, if scheduling strategy is resource utilization Preference strategy, then first point of coordination type is specially each calculate node run in calculate node so that the plurality of task The maximum point coordination type of single node resource utilization.

With reference to second aspect, in second of possible implementation of second aspect, if scheduling strategy is efficiency comes first plan Slightly, then first point of coordination type is specially the most fast point coordination type of overall execution speed for causing operation.

With reference to second aspect or second aspect, second of the first possible implementation or second aspect are possible realizes Mode, in second aspect in the third possible implementation, estimation unit specifically for：For each task, under Face is handled for the operation of first task：The hard information of historic task in the hard information and Sample Storehouse of first task is entered Row matching；If the match is successful, the history run time Estimate first of the historic task matched according to the hard information with first task The run time of task.

It should be noted that the embodiment of the present invention, which is only a kind of exemplary estimation unit that provides, estimates Runtime Implement, certainly, estimation unit can also estimate the run time of task otherwise, such as, pass through prerun The mode of task.That is, the estimation of accurate entire run time is obtained by being previously run a bit of job instances.In addition, same The run time of the follow-up work of one operation can also obtain more accurate estimation with reference to the run time of operation task.This hair Bright embodiment estimates that the specific implementation of Runtime is not limited to estimation unit.

With reference to any one possible implementation in second aspect to second aspect the third possible implementation, In the 4th kind of possible implementation of second aspect, explorer also includes taxon；In determining unit according to each The corresponding resources requirement of task and run time, with reference to default scheduling strategy, determine the first distribution position of the plurality of task Before shape, taxon, for the plurality of task to be classified according to the species of resource, obtains an at least generic task；

Determining unit specifically for：For every generic task in this at least a generic task, according to below for the first kind The operation of task is handled：According to the corresponding resources requirement of each task and run time in the first generic task, with reference to pre- If scheduling strategy, determine the son point coordination type of the first generic task, the son point coordination type is for indicating first generic task many The distribution situation run in calculate node in individual calculate node；By the son distribution of every generic task in this at least a generic task The combination of position shape is defined as first point of coordination type of the plurality of task.

Because explorer provided in an embodiment of the present invention first can be carried out multiple tasks according to the species of resource Classification, and then resource allocation is carried out respectively for each generic task, that is to say, that it can consider simultaneously for isomeric group and spy The resource allocation of different resource demanding tasks, thus with wider universality and more preferable general performance.

Optionally, it is contemplated that the estimation meeting of run time and actual conditions would generally produce some deviations.If to these Deviation is not controlled, then with the extension of time, the predistribution result of operation resource may differ by with desired result it is more and more big. Therefore, explorer provided in an embodiment of the present invention may be incorporated into Variation mechanism and (divide again when carrying out resource allocation With).I.e.：

With reference to any one possible implementation in the 4th kind of possible implementation of second aspect to second aspect, In the 5th kind of possible implementation of second aspect, the plurality of task is assigned to according to first point of coordination type in allocation unit After the running in calculate node of the plurality of task, determining unit is additionally operable to：According to first point of coordination type, all places are determined The first integral dispensing target function value when the task run of wait state is on the node distributed；According to it is all in etc. The corresponding resources requirement of task and run time of state are treated, with reference to default scheduling strategy, is determined all in wait shape Second point of coordination type of the task of state, second point of coordination type is used to indicate that all being waited for for tasks are all at this The distribution situation run in calculate node of being waited for for task；According to second point of coordination type, all places are determined The second integral dispensing target function value when the task run of wait state is on the node distributed；Allocation unit, is also used If being more than the first integral dispensing target function value in the second integral dispensing target function value, by all timess being waited for It is engaged in being assigned to the running in calculate node of all being waited for for tasks according to second point of coordination type.

With reference to the 5th kind of possible implementation of first aspect, in the 6th kind of possible implementation of first aspect；Or Person, with reference to the 5th kind of possible implementation of second aspect, in the 6th kind of possible implementation of second aspect, described first When integral dispensing target function value is equal to first point of coordination type, all task runs being waited for are being distributed The single node of each node distributes the sum of target function value when on node；

When the second integral dispensing target function value is equal to second point of coordination type, all timess being waited for Business operates in the sum of the single node distribution target function value of each node when on distributed node.

Optionally, in a kind of possible implementation, above-mentioned single node distribution object function is specifically included：Wherein, S_nRepresent single node n distribution target function value；P tables It is shown as the industry Time priority factor, p ＞ 0；F represents operation fairness factor, f ＞ 0；p+f≤1；M represents the quantity of operation；S_{E, n} The resource utilization score on node n is represented,r_nRepresent node n resource, r_tExpression task t resource Demand；S_{P, j}Operation j implementation progress is represented,T_jRepresent operation j also needs to how long be finished, should Value can draw by historical statistical data, T₀Represent operation j total runtime；S_{F, j}Operation j fairness score is represented,r_jRepresent operation j resource requirement, r_fRepresent deserved resources of the operation i in the case of complete justice.

Wherein, above-mentioned single node distribution object function in, it is contemplated that the resource utilization of node, the fairness of operation with And the implementation progress of operation.Wherein, work as f=0, during p=0, resource utilization is considered completely, i.e., explorer 110 is according to whole Body resource utilization highest principle distributes resource；Work as f=1, during p=0, it is fair to consider completely, i.e. the meeting of explorer 110 exists Different operating rooms liberally distribute resource；Work as f=1, during p=1, then consider Time priority, i.e. explorer 110 completely Preferentially allocate resources to those operations faster completed.Certainly, f and p can also be other numerical value, and user can be according to operation Operation demand is configured so that distribution obtains balance in optimal resource utilization and optimal Job execution time, and the present invention is real Example is applied to be not especially limited this.

The third aspect includes there is provided a kind of explorer, the explorer：Processor, memory, bus and communication Interface；Memory is used to store computer executed instructions, and processor is connected with memory by bus, when explorer operation When, the computer executed instructions of computing device memory storage so that explorer perform it is above-mentioned such as first aspect or the On the one hand resource allocation methods shown in any one possible implementation.

Because explorer provided in an embodiment of the present invention can be used for performing above-mentioned such as first aspect or first aspect Resource allocation methods shown in any one possible implementation, therefore, it can be obtained technique effect and may be referred to The technique effect of the above-mentioned resource allocation methods as shown in first aspect or first aspect any one possible implementation, Here is omitted.

Fourth aspect includes multiple calculate nodes there is provided a kind of Distributed Computer System, the Distributed Computer System With the explorer described in first aspect or first aspect any one possible implementation；Or, distribution meter Calculation machine system includes the explorer described in multiple calculate nodes and the third aspect.

Due to Distributed Computer System provided in an embodiment of the present invention include first aspect or first aspect any one Explorer described in possible implementation；Or, including the explorer described in the third aspect, therefore, its institute Obtainable technique effect refers to the technique effect of above-mentioned explorer, and the embodiment of the present invention will not be repeated here.

5th aspect is there is provided a kind of computer-readable recording medium, including computer executed instructions, when the computing device of explorer During the computer executed instructions, the explorer performs any one optional side such as above-mentioned first aspect or first aspect Resource allocation methods described in formula.

Wherein, the aspects of the invention or other aspects can more straightforwards in the following description.

Brief description of the drawings

Fig. 1 is the scheduling strategy schematic diagram of existing Hadoop system；

Fig. 2 is a kind of logical architecture figure of distributed computing system provided in an embodiment of the present invention；

Fig. 3 is a kind of physical structure schematic diagram of distributed computing system provided in an embodiment of the present invention；

Fig. 4 is the principle schematic of resource allocation methods provided in an embodiment of the present invention；

Fig. 5 is resource allocation methods schematic flow sheet one provided in an embodiment of the present invention；

Fig. 6 is resource allocation methods schematic flow sheet two provided in an embodiment of the present invention；

Fig. 7 is resource allocation methods schematic flow sheet three provided in an embodiment of the present invention；

Fig. 8 is the Variation mechanism schematic diagram of resource allocation result provided in an embodiment of the present invention；

Fig. 9 is the result signal that use resource utilization preferential principle provided in an embodiment of the present invention carries out resource allocation Figure；

Figure 10 carries out the result schematic diagram of resource allocation for the principle of use fair priority provided in an embodiment of the present invention；

Figure 11 is the structural representation one of explorer provided in an embodiment of the present invention；

Figure 12 is the structural representation two of explorer provided in an embodiment of the present invention；

Figure 13 is the structural representation three of explorer provided in an embodiment of the present invention.

Embodiment

It should be noted that the technical scheme for the ease of clearly describing the embodiment of the present invention, in embodiments of the invention In, employ the printed words such as " first ", " second " and function and the essentially identical identical entry of effect or similar item are made a distinction, ability Field technique personnel are appreciated that the printed words such as " first ", " second " are not defined to quantity and execution order.

It should be noted that "/" herein represent or the meaning, for example, A/B can represent A or B；Herein "and/or" is only a kind of incidence relation for describing affiliated partner, and expression may have three kinds of relations, for example, A and/or B, can To represent：Individualism A, while there is A and B, these three situations of individualism B." multiple " refer to two or more.

As used in this application, term " component ", " module ", " system " etc. are intended to refer to computer related entity, The computer related entity can be hardware, firmware, the combination of hardware and software, software or operating software.For example, group Part, which may be, but not limited to, is：The processing that runs on a processor, processor, object, executable file, executory thread, Program and/or computer.As an example, the application run on the computing device and the computing device can be components.One Or multiple components may reside in executory process and/or thread, and component can be located in computer and/ Or be distributed between two or more computers.In addition, these components can be from having various data structures thereon Performed in various computer-readable mediums.These components can by such as according to have one or more packets (for example, Data from a component, the component is interacted with another component in local system, distributed system and/or to believe Number mode interacted by the network of such as internet etc and other systems) signal, with locally and/or remotely process Mode communicated.

The application will be around may include the system of multiple equipment, component, module etc. that various aspects, embodiment or spy is presented Levy.It is to be understood and appreciated that, each system can include other equipment, component, module etc., and/or can be simultaneously Do not include with reference to all devices, component, module of accompanying drawing discussion etc..Further, it is also possible to use the combination of these schemes.

In addition, in embodiments of the present invention, " example " one word makees example, illustration or explanation for expression.In the application Any embodiment or design for being described as " example " are not necessarily to be construed as than other embodiments or design more preferably Or more advantage.Specifically, using a word of example it is intended to that concept is presented in a concrete fashion.

The scene of description of the embodiment of the present invention is in order to more clearly illustrate the technical scheme of the embodiment of the present invention, not The restriction for technical scheme provided in an embodiment of the present invention is constituted, those of ordinary skill in the art understand, with new scene Occur, technical scheme provided in an embodiment of the present invention is equally applicable for similar technical problem.

In order to which the description of following each embodiments is clearly succinct, the brief introduction of related notion is provided first：

Cluster：Cluster, refers to many isomorphisms or isomery computer node by combination of network, with unification Fixed cluster management system, formation can externally provide unified calculation or storage service facility.

Resource：Resource, that is, refer to internal memory, central processing unit (the English full name for being available for utilizing on distributed type assemblies： Central processing unit, english abbreviation：CPU), network, disk etc. are used to run hardware necessary to operation.

Job：Operation, refers to be run complete task that user is submitted by client device to cluster.

Task：Task, when an operation is submitted to and performed on cluster, is generally decomposed into many tasks, each task fortune Row takes a certain amount of resource on a specific clustered node.

Scheduler：Scheduler, is the engine modules for distributing the resource for being available for task run to operation, is also collection The most important part of cluster management system.

The scheme of the embodiment of the present invention can be typically used in distributed computing system, for realize task scheduling and The efficient distribution of resource.Fig. 2 shows a kind of logical architecture figure of distributed computing system, according to Fig. 2, the Distributed Calculation system System includes the resource pool being made up of cluster resource, explorer and Computational frame, and cluster resource is each calculating in cluster The hardware resources such as computing, the storage of node, explorer is deployed in one or more of cluster calculate node, Huo Zheye As an independent physical equipment resource tune can be provided for being managed collectively cluster resource, and for the Computational frame on upper strata Degree ability.One distributed computing system can support a variety of different Computational frames simultaneously, system as shown in Figure 2, and this is System can support MR (English full name：Map reduce), Storm, S4 (English full name：simple scalable streaming ) and MPI (English full name system：Message passing interface) etc. one or more in Computational frame.Money Source manager carries out unified scheduling by the application program of the different Computational frame types sent to client device, to carry High resource utilization.Fig. 3 further illustrates the physical structure schematic diagram of distributed computing system, including cluster, resource management Device and client device, wherein, cluster includes multiple nodes (three nodes are illustrate only in Fig. 3), explorer portion On administration's some node in the cluster, each node can communicate with explorer, and client device is to explorer The resource request of application program is submitted, explorer is according to specific resource dispatching strategy by the resource allocation of node to application Program, to cause application program to be run on this node according to the node resource of distribution.

Mainly the explorer in distributed computing system is optimized for the embodiment of the present invention, makes it more reasonably Resource is distributed for task, to improve resource utilization.Wherein, Fig. 4 is the distribution method of resource provided in an embodiment of the present invention Principle schematic.

As shown in figure 4, in the embodiment of the present invention, after client device submits operation, operation be broken down into it is a series of can The task (Task) of distribution operation, each Task is equipped with corresponding resources requirement (with horizontal width table in Fig. 2 Levy resources requirement).Experience module (the English full name passed through as Task in above-mentioned explorer：Experienced Expert, english abbreviation：When E-Expert), E-Expert is performed by the job history in the Sample Storehouse in explorer Situation estimates each Task run time, and then can obtain space requirement (i.e. Task resources requirement) and time demand (i.e. characterizing Task run times with the length of longitudinal direction in Task run time, Fig. 2) fixed Task.Packetization module (English： Packer certain scheduling strategy) is considered, by space requirement (i.e. Task resources requirement) and time demand (i.e. Task fortune The row time) fixed Task is dispatched on corresponding node.Wherein, the scheduling strategy include resource utilization preference strategy or Person's efficiency comes first strategy.That is, packaging method flexibly can select resource utilization preferential according to corresponding scheduling strategy Strategy and efficiency comes first strategy carry out resource allocation so that final using point that resource utilization is higher and/or efficiency is higher Coordination type.

Wherein, Task is in each node one waiting list of formation, and the length of each waiting list is roughly equal. During Task waits in line, it is possible to which the morph operation resource that causes a new round is redistributed.If new distribution The overall resource utilization of position shape is higher or more efficient, then updates to new point coordination type.

In addition, each node runs a nodes keep track device (English：Node tracker) example, it is responsible for periodically The running situation that Task is reported to E-Expert, and update the statistical information in experience module Sample Storehouse.

It should be noted that in fig. 2, identical is filled needs identical resource type, such as CPU or interior for sign Deposit, the resource type that the embodiment of the present invention is characterized to the filling of each in Fig. 2 is not especially limited.

Because the program considers the factor of the run time of each task, and packetization module by space requirement (i.e. Task resources requirement) and time demand (i.e. Task run time) fixed Task when being dispatched on corresponding node, Resource utilization preference strategy and efficiency comes first strategy can be flexibly selected to carry out resource allocation according to corresponding scheduling strategy, So that finally using point coordination type that resource utilization is higher and/or efficiency is higher, therefore, it is possible to mitigate resource in the prior art The problem of fragment, the utilization rate of cluster resource is significantly improved, and/or, the Job execution time can be shortened, the execution of operation is lifted Efficiency.

Below by the principle schematic of the distribution method based on the resource shown in Fig. 4, to the technology in the embodiment of the present invention Scheme is clearly and completely described.

As shown in figure 5, the embodiment of the present invention provides a kind of resource allocation methods, including step S501-S504：

S501, explorer receive the operation that client device is submitted, and are multiple tasks by the breakdown of operation, its In, each task in the plurality of task is each equipped with corresponding resources requirement.

S502, explorer estimate the run time of each task.

S503, explorer are according to the corresponding resources requirement of each task and run time, with reference to default scheduling Strategy, determines first point of coordination type of the plurality of task, and first point of coordination type is used to indicate that the plurality of task is calculated multiple The distribution situation run in calculate node in node, the scheduling strategy includes resource utilization preference strategy and efficiency comes first At least one of strategy.

The plurality of task is run meter by S504, explorer according to what first point of coordination type was assigned to the plurality of task On operator node.

Specifically, in step of embodiment of the present invention S501：

Resources requirement can be specifically the demand of cpu resource, and/or, the demand of memory source, and/or, network Demand of bandwidth, etc., the embodiment of the present invention is not especially limited to this.

Specifically, in step of embodiment of the present invention S502：

In general, the external world of specific execution time of a task can not be learnt.But, in most situations Under, a generic task can be repeated.Such as in corporate client scene, it may be necessary to perform the data statistics work repeated daily. Therefore, the statistics based on historical information tends to provide the estimation of a generic task run time.Therefore, can in explorer Module can be needed to safeguard a statistic information base, for record clustering Historical Jobs information, such as the E-Expert in Fig. 4 Module.When there is new task to arrive, the task is matched by a certain class according to historical statistical information, then according to the generic task History run estimate the run time of the task.

Information is generally divided into hard information and the class of Soft Inform ation two by E-Expert modules.

Hard information includes homework type, performs user etc..The task of different work type is obviously not belonging to same class.And it is same There is a strong possibility is same class for the operation of one user operation, the operation even repeated.Hard information is by following Sample Storehouses Safeguard.

Soft Inform ation includes size, input data size, output data size of data volume that the task is handled etc..This kind of letter Breath is frequently not fixed, but close associate is there is between run time and this category information.Soft Inform ation needs to do extra Statistics, statistical information safeguarded by following statistics storehouse.

And then, optionally, explorer can estimate the run time of each task in the following way, specific bag Include：

For each task, handled according to the operation below for first task：

The hard information of first task is matched with the hard information of the historic task in Sample Storehouse.

If the match is successful, the history run time Estimate of the historic task matched according to the hard information with the first task should The run time of first task.

It should be noted that when there is new task to arrive, if the task can not be matched according to historical statistical information A certain class, then can estimate the run time of the task, the global draw by way of assigning global draw value to the task Value can be the average value of the run time of all historic tasks, and the embodiment of the present invention is not especially limited to the situation.

Specifically, in step of embodiment of the present invention S503：

Scheduling strategy can specifically include at least one of resource utilization preference strategy and efficiency comes first strategy.Its In,

If scheduling strategy is resource utilization preference strategy, first point of coordination type, which is specifically as follows, causes multiple tasks The maximum point coordination type of the single node resource utilization of each calculate node in calculate node can be run.

Or, if scheduling strategy is efficiency comes first strategy, first point of coordination type is specifically as follows the entirety for causing operation Perform fastest point coordination type.

The embodiment of the present invention is not especially limited to the concrete form of first point of coordination type.

Optionally, in resource allocation methods provided in an embodiment of the present invention, can also consider simultaneously for isomeric group and The resource allocation of special resource demand operation.

That is, as shown in fig. 6, in explorer according to the corresponding resources requirement of each task and run time, with reference to Before default scheduling strategy, first point of coordination type (step S503) for determining the plurality of task, step S505 can also be included：

S505, explorer are classified the plurality of task according to the species of resource, obtain an at least generic task.

Wherein, resource category can specifically include heterogeneous resource species and non-heterogeneous resource species etc., heterogeneous resource species Can also be according to being which kind of heterogeneous resource is divided again, the embodiment of the present invention is not especially limited to this.

And then, explorer is according to the corresponding resources requirement of each task and run time, with reference to default scheduling Strategy, determines first point of coordination type (step S503) of the plurality of task, can specifically include step S503a and S503b：

S503a, for every generic task in this at least a generic task, explorer according to below for the first kind appoint The operation of business is handled：

The corresponding resources requirement of each task and run time in first generic task, with reference to default scheduling Strategy, determines the son point coordination type of first generic task, and the son point coordination type is used to indicate that first generic task is calculated multiple The distribution situation run in calculate node in node.

The combination of the son of every generic task in this at least a generic task point coordination type is defined as by S503b, explorer First point of coordination type of the plurality of task.

Because resource allocation methods provided in an embodiment of the present invention are first divided multiple tasks according to the species of resource Class, and then resource allocation is carried out respectively for each generic task, that is to say, that can consider simultaneously for isomeric group and special The resource allocation of resource demanding tasks, thus with wider universality and more preferable general performance.

Optionally, it is contemplated that the estimation meeting of run time and actual conditions would generally produce some deviations.If to these Deviation is not controlled, then with the extension of time, the predistribution result of operation resource may differ by with desired result it is more and more big. Therefore, in resource allocation methods provided in an embodiment of the present invention, it may be incorporated into Variation mechanism (redistributing).

That is, as shown in fig. 7, the plurality of task is assigned into the plurality of task according to first point of coordination type in explorer Run in calculate node (step S504) after, step S506-S509 can also be included：

S506, explorer determine all task runs being waited in institute according to first point of coordination type The first integral dispensing target function value when on the node of distribution.

When S507, explorer are according to the corresponding resources requirement of all being waited for for tasks and operation Between, with reference to default scheduling strategy, second point of coordination type of all being waited for for tasks is determined, second point of coordination type For indicating all being waited for for task running in calculate node in all being waited for for tasks Distribution situation.

S508, explorer determine that all task runs being waited for exist according to second point of coordination type The second integral dispensing target function value when on the node distributed.

If S509, the second integral dispensing target function value are more than the first integral dispensing target function value, explorer will All being waited for for tasks are assigned to transporting for all being waited for for tasks according to second point of coordination type In row calculate node.

Optionally, in the embodiment of the present invention, integral dispensing target function value can be obtained by equation below (1)：

S=∑s_nS_nFormula (1)

Wherein, S_nRepresent single node n distribution target function value；S represents integral dispensing target function value.

That is, it is all in wait shape when the first integral dispensing target function value is equal to first point of coordination type in step S506 The single node of each node distributes the sum of target function value when the task run of state is on the node distributed.

It is all to be waited for when the second integral dispensing target function value is equal to second point of coordination type in step S508 The single node of each node distributes the sum of target function value when task run is on the node distributed.

Optionally, in the embodiment of the present invention, single node distribution object function specifically can be as shown in formula (2)：

Formula (2)

Wherein, S_nRepresent single node n distribution target function value；P represents activity duration priority factors, p ＞ 0；F is represented Operation fairness factor, f ＞ 0；p+f≤1；M represents the quantity of operation.

S_{E, n}The resource utilization score on node n is represented,r_nRepresent node n resource, r_tRepresent Task t resources requirement.

S_{P, j}Operation j implementation progress is represented,T_jRepresent operation j also needs to how long be finished, should Value can draw by historical statistical data, T₀Represent operation j total runtime.As can be seen that S_{P, j}Span be [1/ E, 1], 1/e represents that operation just brings into operation, and 1 represents to complete already.

S_{F, j}Operation j fairness score is represented,r_jRepresent operation j resource requirement, r_fRepresent to make Deserved resources of the industry j in the case of complete justice.

It should be noted that in the case of multi dimensional resource, the r in above-mentioned formula_n、r_t、r_fIt is vector etc. parameter.

In above-mentioned formula (2), it is contemplated that the execution of the resource utilization of node, the fairness of operation and operation is entered Degree.Wherein, work as f=0, during p=0, consider resource utilization, i.e. explorer according to overall resource utilization highest completely Principle distributes resource；Work as f=1, during p=0, it is fair to consider completely, i.e., explorer can liberally divide in different operating rooms With resource；Work as f=1, during p=1, then consider that Time priority, i.e. explorer preferentially allocate resources to those faster completely The operation of completion.Certainly, f and p can also be other numerical value, and user can be configured according to the operation demand of operation so that Distribution obtains balance in optimal resource utilization and optimal Job execution time, and the embodiment of the present invention is not especially limited to this.

The embodiment of the present invention is not especially limited to this.

It should be noted that formula (2), which is only a kind of exemplary single node that provides, distributes implementing for object function, Certainly, according to different distribution Considerations, single node distribution object function can also be other, and the embodiment of the present invention is to this It is not especially limited.

By above-mentioned Variation mechanism, the predistribution result of operation resource can be caused to be evolved to more preferable direction.Wherein, scheme 8 show schematically a kind of Variation mechanism schematic diagram of the allocation result of operation resource, it can be seen that prolonging over time Long, predistribution result and the desired result of operation resource may differ by more and more big., can be by node by above-mentioned Variation mechanism Task 1 on 3 is adjusted on node 1, and the task 2 on node 2 is adjusted on node 3, and the task 3 on node 1 is adjusted to On node 2, so that the predistribution result of operation resource is evolved to more preferable direction.

It should be noted that in fig. 8, identical is filled needs identical resource type, such as CPU or interior for sign Deposit, the resource type that the embodiment of the present invention is characterized to the filling of each in Fig. 8 is not especially limited.

The resource allocation methods in the various embodiments described above will be illustrated with combining a specific example below.

It is exemplary, it is assumed that to have four nodes node1, node2, node3 and node4, wherein, node1, node2 and Node3 is isomorphism node, respectively there is 6 cores, 12G internal memories, the 2Gbps network bandwidth；Node4 is the graphics calculations section of an isomery Point, graphics processor (the English full name for having 2 cores, 2G internal memories and 128 cores：Graphics processing unit, English Text abbreviation：GPU) video card.

In addition, it is assumed that there is four Job, the resources requirement for the Task that each Job is submitted is as follows.Wherein, in bracket 3-dimensional digital represent the quantity of the core needed for Task, the size of internal memory and the size of the network bandwidth respectively：

JobA：18 Map Task (1,2,0), 3 Reduce Task (0,0,2)；

JobB：6 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobC：6 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobD：2 Map Task (1,1,0) are, it is necessary to GPU；

Simultaneously, it is assumed that the estimation of all JobA, JobB and JobC Task run time is t, JobD Task Run time is estimated as 4t.

Situation one：

Now, according to resource utilization preferential principle, such as overall resource utilization highest, that is, above-mentioned single-unit F=0, p=0 in point distribution object function (formula (2)), then JobA, JobB, JobC and JobD Task will be according to such as dirty Journey is dispatched：

Step 1, all Task are classified according to whether with heterogeneous resource demand.

Now, JobA, JobB, JobC and JobD all Task can be divided into two classes：Need GPU and need not GPU's.

Step 2,2 Map Task for needing GPU, schedule it to node node4.

By this wheel distribution, the situation for the Task being waited for is：

JobA：18 Map Task (1,2,0), 3 Reduce Task (0,0,2)；

JobB：6 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobC：6 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobD：0 Map Task (1,1,0) is, it is necessary to GPU.

Step 3, for remaining Task, present cluster there remains tri- nodes of node1, node2 and node3, altogether (18,36,6) resource.

Step 4, for node1 nodes, be computed, 6 in JobA Map Task formation Tasks bags (altogether (6, 12,0) resource) cause the single node of node1 nodes to distribute target function value S_nMaximum, therefore, the Tasks bags are placed on On node1.

For node2 and node3 nodes, be computed, 6 in JobA Map Task formation Tasks bags (altogether (6, 12,0) resource) cause the single node of node2 nodes to distribute target function value S_n6 Map Task shapes in maximum, and JobA Into Tasks bags ((6,12,0) resource altogether) cause the single node distribution target function value S of node3 nodes_nIt is maximum.Therefore, 12 Map Task of residue in JobA are evenly distributed on node2 and node3 nodes.

It should be noted that the run time estimation of the Task due to assuming all JobA, JobB and JobC in the example It is t, that is, run time is equal, and f=0, p=0, therefore single node distribution object functionDeteriorate toAnd then the various combinations of traversal, It may finally determine that the Tasks bags ((6,12,0) resource altogether) of 6 Map Task formation in JobA cause node1 nodes Single node distribution target function value S_nMaximum, now,

By contrast, if taking two Map Task formation Tasks bags of JobB, single node distribution target function value S_nIt is less than Above-mentioned value, is just no longer verified one by one herein.

Similarly, according to above-mentioned degeneration formula, it may be determined that the Tasks bags of 6 Map Task formation in JobA are (altogether (6,12,0) resource) cause the single node of node2 nodes to distribute target function value S_n6 Map Task shapes in maximum, JobA Into Tasks bags ((6,12,0) resource altogether) cause the single node distribution target function value S of node3 nodes_nMaximum, the present invention Embodiment is just no longer verified one by one herein.

By this wheel distribution, the situation for the Task being waited for is：

JobA：0 Map Task (1,2,0), 3 Reduce Task (0,0,2)；

JobB：6 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobC：6 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobD：0 Map Task (1,1,0) is, it is necessary to GPU.

Step 5, continuation are distributed：

It is computed, JobA 1 Reduce Task and JobB 2 Map Task formation Tasks bags are (altogether (6,2,2) Resource) cause the single node of node1 nodes to distribute target function value S_nMaximum, therefore, the Tasks bags are placed on node1.

It is computed, JobA 1 Reduce Task and JobB 2 Map Task formation Tasks bags are (altogether (6,2,2) Resource) cause the single node of node2 nodes to distribute target function value S_nMaximum, therefore, the Tasks bags are placed on node2.

It is computed, JobA 1 Reduce Task and JobB 2 Map Task formation Tasks bags are (altogether (6,2,2) Resource) cause the single node of node3 nodes to distribute target function value S_nMaximum, therefore, the Tasks bags are placed on node3.

So after this wheel distribution, the situation for the Task being waited for is：

JobA：0 Map Task (1,2,0), 0 Reduce Task (0,0,2)；

JobB：0 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobC：6 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobD：0 Map Task (1,1,0) needs GPU.

Step 6, continuation are distributed：

It is computed, JobB 1 Reduce Task and JobC 2 Map Task formation Tasks bags are (altogether (6,2,2) Resource) cause the single node of node1 nodes to distribute target function value S_nMaximum, therefore, the Tasks bags are placed on node1.

It is computed, JobB 1 Reduce Task and JobC 2 Map Task formation Tasks bags are (altogether (6,2,2) Resource) cause the single node of node2 nodes to distribute target function value S_nMaximum, therefore, the Tasks bags are placed on node2.

It is computed, JobB 1 Reduce Task and JobC 2 Map Task formation Tasks bags are (altogether (6,2,2) Resource) cause the single node of node3 nodes to distribute target function value S_nMaximum, therefore, the Tasks bags are placed on node3.

JobA：0 Map Task (1,2,0), 0 Reduce Task (0,0,2)；

JobB：0 Map Task (3,1,0), 0 Reduce Task (0,0,2)；

JobC：0 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobD：0 Map Task (1,1,0) is, it is necessary to GPU.

Step 7, continuation are distributed：

By the remaining 3 Task distribution of JobC on three nodes, so, all distribution are finished, final target Divide coordination type as shown in Figure 9.

Situation two：

Now, according to the principle of fair priority, than the f=in such as above-mentioned single node distribution object function (formula (2)) 1, p=0, then JobA, JobB, JobC and JobD Task will be according to following flow schedulings：

Step 2,2 Map Task for needing GPU, schedule it to node node4.

By this wheel distribution, the situation for the Task being waited for is：

JobA：18 Map Task (1,2,0), 3 Reduce Task (0,0,2)；

JobB：6 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobC：6 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobD：0 Map Task (1,1,0) is, it is necessary to GPU.

For node2 nodes, it is computed, ((6,2,0) are provided the Tasks bags of 2 in JobB Map Task formation altogether Source) cause the single node of node2 nodes to distribute target function value S_nMaximum, therefore, the Tasks bags are placed on node2.

For node3 nodes, it is computed, ((6,2,0) are provided the Tasks bags of 2 in JobC Map Task formation altogether Source) cause the single node of node3 nodes to distribute target function value S_nMaximum, therefore, the Tasks bags are placed on node3.

It should be noted that the run time estimation of the Task due to assuming all JobA, JobB and JobC in the example It is t, that is, run time is equal, and f=1, p=0, therefore single node distribution object functionDeteriorate toAnd then various combinations are traveled through, finally It can determine that the Tasks bags ((6,12,0) resource altogether) of 6 Map Task formation in JobA cause the list of node1 nodes Node distribution target function value S_nIt is maximum.

Similarly, according to above-mentioned degeneration formula, it may be determined that the Tasks bags of 2 Map Task formation in JobB are (altogether (6,2,0) resource) cause the single node of node2 nodes to distribute target function value S_n2 Map Task in maximum, JobC are formed Tasks bags ((6,2,0) resource altogether) cause the single node distribution target function value S of node3 nodes_nMaximum, the present invention is real Example is applied just no longer to verify one by one herein.

By this wheel distribution, the situation for the Task being waited for is：

JobA：12 Map Task (1,2,0), 3 Reduce Task (0,0,2)；

JobB：4 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobC：4 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobD：0 Map Task (1,1,0) is, it is necessary to GPU.

Step 5, continuation are distributed：

It is computed, the Tasks bags ((6,12,0) resource altogether) of 6 in JobA Map Task formation cause node1 sections The single node distribution target function value S of point_nMaximum, therefore, the Tasks bags are placed on node1.

It is computed, the Tasks bags ((6,2,0) resource altogether) of 2 in JobB Map Task formation cause node2 nodes Single node distribution target function value S_nMaximum, therefore, the Tasks bags are placed on node2.

It is computed, the Tasks bags ((6,2,0) resource altogether) of 2 in JobC Map Task formation cause node3 nodes Single node distribution target function value S_nMaximum, therefore, the Tasks bags are placed on node3.

JobA：6 Map Task (1,2,0), 3 Reduce Task (0,0,2)；

JobB：2 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobC：2 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobD：0 Map Task (1,1,0) is, it is necessary to GPU.

Step 6, continuation are distributed：

JobA：0 Map Task (1,2,0), 3 Reduce Task (0,0,2)；

JobB：0 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobC：0 Map Task (3,1,0), 3 Reduce Task (0,0,2)；

JobD：0 Map Task (1,1,0) is, it is necessary to GPU.

Step 7, continuation are distributed：

It is computed, the Tasks bags ((0,0,2) resource altogether) of 1 in JobA Reduce Task formation cause node1 The single node distribution target function value S of node_nMaximum, therefore, the Tasks bags are placed on node1.

It is computed, the Tasks bags ((0,0,2) resource altogether) of 1 in JobB Reduce Task formation cause node2 The single node distribution target function value S of node_nMaximum, therefore, the Tasks bags are placed on node2.

It is computed, the Tasks bags ((0,0,2) resource altogether) of 1 in JobC Reduce Task formation cause node3 The single node distribution target function value S of node_nMaximum, therefore, the Tasks bags are placed on node3.

JobA：0 Map Task (1,2,0), 2 Reduce Task (0,0,2)；

JobB：0 Map Task (3,1,0), 2 Reduce Task (0,0,2)；

JobC：0 Map Task (3,1,0), 2 Reduce Task (0,0,2)；

JobD：0 Map Task (1,1,0) is, it is necessary to GPU.

Step 8, continuation are distributed：

JobA：0 Map Task (1,2,0), 1 Reduce Task (0,0,2)；

JobB：0 Map Task (3,1,0), 1 Reduce Task (0,0,2)；

JobC：0 Map Task (3,1,0), 1 Reduce Task (0,0,2)；

JobD：0 Map Task (1,1,0) is, it is necessary to GPU；

Step 9, continuation are distributed：

So after this wheel distribution, all distribution are finished, and final Target Assignment position shape is as shown in Figure 10.

Situation three：

Now, according to the principle of time priority, such as overall execution efficiency highest, that is, above-mentioned single node distribution mesh F=0, p=1 in scalar functions (formula (2)), then JobA, JobB, JobC and JobD Task will be examined completely when being allocated Consider Time priority, i.e. explorer and preferentially allocate resources to those operations faster completed.Now, single node distribution target FunctionDeteriorate toAnd then according to each node The maximum principle of value of single node object function be allocated, the embodiment of the present invention is to the situation just no longer citing in detail Explanation.

If ignoring fairness it can be seen from the corresponding examples of Fig. 9 and Figure 10, the preferential method of salary distribution of resource utilization The operation overall execution time can effectively be shortened to 4t, and if consideration fairness, the overall operation of the method for salary distribution of fair priority The execution time is 6t.

As shown in figure 11, the embodiment of the present invention provides a kind of explorer 110, for performing shown in figure 5 above-Fig. 7 Resource allocation methods.The explorer 110 can include the unit corresponding to corresponding steps, example, can include：Connect Receive unit 1101, resolving cell 1106, estimation unit 1102, determining unit 1103 and allocation unit 1104.

Receiving unit 1101, the operation for receiving client device submission.

Resolving cell 1106, for being multiple tasks by the breakdown of operation, wherein, each task in the plurality of task is equal It is configured with corresponding resources requirement.

Estimation unit 1102, the run time for estimating each task.

Determining unit 1103, for according to the corresponding resources requirement of each task and run time, being adjusted with reference to default Degree strategy, determines first point of coordination type of the plurality of task, and first point of coordination type is used to indicate the plurality of task the plurality of The distribution situation run in calculate node in calculate node, the scheduling strategy includes resource utilization preference strategy and efficiency At least one of preference strategy.

Allocation unit 1104, for the plurality of task to be assigned into running for the plurality of task according to first point of coordination type In calculate node.

Optionally, if scheduling strategy is resource utilization preference strategy, first point of coordination type is specifically as follows so that should Point coordination type of the single node resource utilization for running each calculate node in the calculate node maximum of multiple tasks.

Optionally, if scheduling strategy is efficiency comes first strategy, first point of coordination type can be for so that the entirety of operation be held The fastest point coordination type of row.

Optionally, estimation unit 1102 specifically can be used for：

For each task, handled according to the operation below for first task：

The hard information of first task is matched with the hard information of the historic task in Sample Storehouse；

If the match is successful, the history run time Estimate first of the historic task matched according to the hard information with first task The run time of task.

Specifically, the hard information in the embodiment of the present invention includes homework type, performs the information such as user.

It should be noted that the embodiment of the present invention, which is only the exemplary estimation unit 1102 that provides, estimates Runtime Implement, certainly, estimation unit 1102 can also estimate the run time of task otherwise, such as, by pre- The mode of operation task.That is, the estimation of accurate entire run time is obtained by being previously run a bit of job instances.Separately Outside, the run time of the follow-up work of same operation can also be obtained with reference to the run time of operation task and more accurately estimated Meter.The embodiment of the present invention estimates that the specific implementation of Runtime is not limited to estimation unit 1102.

Based on explorer 110 provided in an embodiment of the present invention, the explorer 110 is carried in reception client device The operation of friendship, and by the breakdown of operation for multiple the having a configuration of corresponding resources requirement of the tasks after, also estimate each task Run time, and according to the resources requirement and run time of each task, with reference to default scheduling strategy, determine the plurality of First point of coordination type of task, is then assigned to running for the plurality of task by the plurality of task according to first point of coordination type In calculate node.Wherein, first point of coordination type is used to indicate that the plurality of task runs calculate node in the plurality of task On distribution situation, scheduling strategy include at least one of resource utilization preference strategy and efficiency comes first strategy.Namely Say, the explorer 110 considers the factor of the run time of each task when carrying out resource allocation, and space is needed When the Task for asking (i.e. the resources requirement of task) and time demand (i.e. the time of task) to fix is dispatched on corresponding node, Resource utilization preference strategy and efficiency comes first strategy can be flexibly selected to carry out resource allocation according to corresponding scheduling strategy, So that finally using point coordination type that resource utilization is higher and/or efficiency is higher.On the one hand, due to the utilization of resources can be used The higher point coordination type of rate, you can to cause the higher Task of those node resource utilization rates to combine scheduled by scheduling strategy Onto node, therefore the explorer 110 is the problem of can effectively mitigate resource fragmentation in the prior art, so as to lift cluster Resource utilization.On the other hand, due to the higher point coordination type of efficiency can be used, you can to cause those by scheduling strategy The Job execution time, most short Task combinations were scheduled on node, therefore compared with prior art, the energy of explorer 110 Significantly shorten the Job execution time, lift Job execution efficiency.To sum up, the energy of explorer 110 provided in an embodiment of the present invention It is enough flexibly to be selected resource utilization preference strategy and efficiency comes first strategy to carry out resource allocation according to corresponding scheduling strategy, from And resource utilization can be improved, and/or, the execution efficiency of user job can be lifted.

Optionally, explorer 110 provided in an embodiment of the present invention can also consider simultaneously when carrying out resource allocation For isomeric group and the resource allocation of special resource demand operation.

Specifically, as shown in figure 12, explorer 110 can also include taxon 1105.

In determining unit 1103 according to the corresponding resources requirement of each task and run time, with reference to default scheduling plan Slightly, before first point of coordination type for determining the plurality of task, taxon 1105, for by kind of the plurality of task according to resource Class is classified, and obtains an at least generic task.

Determining unit 1103 specifically for：

For every generic task in this at least a generic task, according to the operation below for the first generic task at Reason：

According to the corresponding resources requirement of each task and run time in the first generic task, with reference to scheduling strategy, it is determined that The son of first generic task point coordination type, the son point coordination type is used to indicate the first generic task transporting in multiple calculate nodes Distribution situation in row calculate node.

The combination of the son of every generic task in this at least a generic task point coordination type is defined as the first of the plurality of task Divide coordination type.

Because explorer 110 provided in an embodiment of the present invention first can enter multiple tasks according to the species of resource Row classification, and then carries out resource allocation respectively for each generic task, that is to say, that can consider simultaneously for isomeric group and The resource allocation of special resource demand operation, thus with wider universality and more preferable general performance.

Optionally, it is contemplated that the estimation meeting of run time and actual conditions would generally produce some deviations.If to these Deviation is not controlled, then with the extension of time, the predistribution result of operation resource may differ by with desired result it is more and more big. Therefore, in explorer 110 provided in an embodiment of the present invention, in allocation unit 1104 by multiple tasks according to the first distribution position Shape is assigned to after the running in calculate node of multiple tasks, and determining unit 1103 is additionally operable to：

According to first point of coordination type, when all task runs being waited for are on the node distributed is determined One integral dispensing target function value.

According to the corresponding resources requirement of all being waited for for tasks and run time, with reference to scheduling strategy, really Second point of coordination type of fixed all being waited for for tasks, second point of coordination type is used to indicate all be waited for Task all being waited for for tasks the distribution situation run in calculate node.

According to second point of coordination type, determine when all task runs being waited for are on the node distributed Second integral dispensing target function value；

Allocation unit 1104, if being additionally operable to the second integral dispensing target function value more than the first integral dispensing object function All being waited for for tasks are assigned to transporting for all being waited for for tasks by value according to second point of coordination type In row calculate node.

Optionally, in the embodiment of the present invention, integral dispensing target function value can be obtained by above-mentioned formula (1), the present invention Embodiment will not be repeated here.

That is, it is all to be waited for when above-mentioned the first integral dispensing target function value is equal to first point of coordination type The single node of each node distributes the sum of target function value when task run is on the node distributed.

When the second above-mentioned integral dispensing target function value is equal to second point of coordination type, all being waited for for tasks The single node for operating in each node when on distributed node distributes the sum of target function value.

Optionally, in the embodiment of the present invention, single node distribution object function can specifically include：

Wherein, S_nRepresent single node n distribution target function value；P represents activity duration priority factors, p ＞ 0；F is represented Operation fairness factor, f ＞ 0；p+f≤1；M represents the quantity of operation；S_{E, n}The resource utilization score on node n is represented,r_nRepresent node n resource, r_tExpression task t resources requirement；S_{P, j}Represent that operation j execution is entered Degree,T_jRepresent operation j also needs to how long be finished, the value can be drawn by historical statistical data, T₀Represent Operation j total runtime；S_{F, j}Operation j fairness score is represented,r_jRepresenting operation i resource needs Ask, r_fRepresent deserved resources of the operation i in the case of complete justice.

It should be noted that the receiving unit 1101 in the embodiment of the present invention can connect to possess on explorer 110 Receive the interface circuit of function, such as receiver or receiver；Can also for possess on explorer 110 network interface card of receive capabilities or Input/output (English full name：Input/output, english abbreviation：I/O) interface, the embodiment of the present invention does not make specific limit to this It is fixed.

Estimation unit 1102, determining unit 1103, allocation unit 1104 and taxon 1105 can individually be set up Processor, can also be integrated in some processor of explorer 110 and realize, in addition it is also possible to program code Form is stored in the memory of explorer 110, is called by some processor of explorer 110 and is performed the above and estimated Count the function of unit 1102, determining unit 1103, allocation unit 1104 and taxon 1105.Processor described here can be with It is a central processing unit (English full name：Central processing unit, english abbreviation：CPU), it can also be other General processor, digital signal processor (English full name：Digital signal processing, english abbreviation：DSP), specially With integrated circuit (English full name：Application specific integrated circuit, english abbreviation：ASIC it is), existing Field programmable gate array (English full name：Field-programmable gate array, english abbreviation：FPGA) or other PLD, discrete gate or transistor logic, discrete hardware components etc..General processor can be microprocessor Device or the processor can also be any conventional processors etc..In addition, the processor can also be application specific processor, this is special It can include at least one in baseband processing chip, radio frequency processing chip etc. with processor.Further, the application specific processor The chip with other dedicated processes functions of explorer 110 can also be included.

It is appreciated that the explorer 110 in the embodiment of the present invention may correspond to the resource point shown in above-mentioned Fig. 5-Fig. 7 The division of unit in explorer in method of completing the square, and explorer 110 in the embodiment of the present invention and/or Function etc. is for realizing the resource allocation methods flow shown in above-mentioned Fig. 5-Fig. 7, for sake of simplicity, will not be repeated here.

As shown in figure 13, the embodiment of the present invention provides a kind of explorer 130, including：Processor 1301, memory 1302nd, bus 1303 and communication interface 1304.

Memory 1302 is used to store computer executed instructions, and processor 1301 is connected with memory 1302 by bus, When explorer 130 is run, processor 1301 performs the computer executed instructions that memory 1303 is stored, so that resource pipe Reason device 130 performs resource allocation methods as shown in Figure 5-Figure 7.Specific address distribution method can be found in above-mentioned such as Fig. 5-Fig. 7 Associated description in shown embodiment, here is omitted.

Wherein, the processor 1301 in the embodiment of the present invention can be a central processing unit (English full name：central Processing unit, english abbreviation：CPU), it can also be other general processors, digital signal processor (English full name： Digital signal processing, english abbreviation：DSP), application specific integrated circuit (English full name：application Specific integrated circuit, english abbreviation：ASIC), field programmable gate array (English full name：field- Programmable gate array, english abbreviation：) or other PLDs, discrete gate or transistor FPGA Logical device, discrete hardware components etc..General processor can be microprocessor or the processor can also be any routine Processor etc..

In addition, the processor 1301 can also be application specific processor, the application specific processor can include baseband processing chip, At least one in radio frequency processing chip etc..Further, the application specific processor can also include have explorer 130 its The chip of his dedicated processes function.

Memory 1302 can include volatile memory (English：Volatile memory), such as random access memory Device (English full name：Random-access memory, english abbreviation：RAM)；Memory 1302 can also include non-volatile deposit Reservoir (English：Non-volatile memory), such as read-only storage (English full name：Read-only memory, English Abbreviation：ROM), flash memory (English：Flash memory), hard disk (English full name：Hard disk drive, English contracting Write：) or solid state hard disc (English full name HDD：Solid-state drive, english abbreviation：SSD)；In addition, memory 1302 is also The combination of the memory of mentioned kind can be included.

Bus 1303 can include data/address bus, power bus, controlling bus and signal condition bus etc..In the present embodiment For clear explanation, various buses are all illustrated as bus 1303 in fig. 13.

During implementing, each step in above-mentioned resource allocation methods flow as shown in Figure 5-Figure 7 can be with The computer executed instructions for performing the software form stored in memory 1302 by the processor 1301 of example, in hardware are realized.For Avoid repeating, here is omitted.

Optionally, the embodiment of the present invention also provides a kind of computer-readable recording medium, and the computer-readable recording medium is used to store computer and perform to refer to Order, when the computing device of the explorer computer executed instructions, the explorer is performed as shown in Figure 5-Figure 7 Resource allocation methods.Specific resource allocation methods can be found in the associated description in above-mentioned embodiment as shown in Figure 5-Figure 7, this Place is repeated no more.

It is apparent to those skilled in the art that, for convenience and simplicity of description, the device of foregoing description, Only with the division progress of above-mentioned each functional module for example, in practical application, as needed can distribute above-mentioned functions Completed by different functional modules, i.e., the internal structure of device is divided into different functional modules, it is described above to complete All or part of function.The specific work process of the system of foregoing description, device and unit, may be referred to preceding method implementation Corresponding process in example, will not be repeated here.

In several embodiments provided herein, it should be understood that disclosed system, apparatus and method can be with Realize by another way.For example, device embodiment described above is only schematical, for example, the module or The division of unit, only a kind of division of logic function, can there is other dividing mode when actually realizing, such as multiple units Or component can combine or be desirably integrated into another system, or some features can be ignored, or not perform.It is another, institute Display or the coupling each other discussed or direct-coupling or communication connection can be by some interfaces, device or unit INDIRECT COUPLING or communication connection, can be electrical, machinery or other forms.

The unit illustrated as separating component can be or may not be it is physically separate, it is aobvious as unit The part shown can be or may not be physical location, you can with positioned at a place, or can also be distributed to multiple On NE.Some or all of unit therein can be selected to realize the mesh of this embodiment scheme according to the actual needs 's.

In addition, each functional unit in each embodiment of the invention can be integrated in a processing unit, can also That unit is individually physically present, can also two or more units it is integrated in a unit.Above-mentioned integrated list Member can both be realized in the form of hardware, it would however also be possible to employ the form of SFU software functional unit is realized.

If the integrated unit is realized using in the form of SFU software functional unit and as independent production marketing or used When, it can be stored in a computer read/write memory medium.Understood based on such, technical scheme is substantially The part contributed in other words to prior art or all or part of the technical scheme can be in the form of software products Embody, the computer software product is stored in a storage medium, including some instructions are to cause a computer Equipment (can be personal computer, server, or network equipment etc.) or processor (processor) perform the present invention each The all or part of step of embodiment methods described.And foregoing storage medium includes：USB flash disk, mobile hard disk, ROM, RAM), magnetic Dish or CD etc. are various can be with the medium of store program codes.

The foregoing is only a specific embodiment of the invention, but protection scope of the present invention is not limited thereto, any Those familiar with the art the invention discloses technical scope in, change or replacement can be readily occurred in, should all be contained Cover within protection scope of the present invention.Therefore, protection scope of the present invention should be based on the protection scope of the described claims.

Claims

1. a kind of resource allocation methods in distributed computing system, the distributed computing system includes multiple calculate nodes, Characterized in that, methods described includes：

The operation that client device is submitted is received, and is multiple tasks by the breakdown of operation, wherein, in the multiple task Each task is each equipped with corresponding resources requirement；

Estimate the run time of each task；

According to the corresponding resources requirement of each task and run time, with reference to default scheduling strategy, determine described many First point of coordination type of individual task, first point of coordination type is used to indicate the multiple task in the multiple calculate node The distribution situation run in calculate node, the scheduling strategy include resource utilization preference strategy and efficiency comes first strategy At least one of；

The multiple task is assigned to the running in calculate node of the multiple task according to first point of coordination type.

2. according to the method described in claim 1, it is characterised in that if the scheduling strategy is resource utilization preference strategy, Then first point of coordination type is so that the single node for running each calculate node in calculate node of the multiple task The maximum point coordination type of resource utilization.

3. according to the method described in claim 1, it is characterised in that described if the scheduling strategy is efficiency comes first strategy First point of coordination type is causes the overall execution speed of the operation most fast point coordination type.

4. the method according to claim any one of 1-3, it is characterised in that during the operation of estimation each task Between, including：

For each task, handled according to the operation below for first task：

The hard information of the first task is matched with the hard information of the historic task in Sample Storehouse；

If the match is successful, described in the history run time Estimate of the historic task matched according to the hard information with the first task The run time of first task.

5. the method according to claim any one of 1-4, it is characterised in that it is described by the multiple task according to described First point of coordination type is assigned to after the running in calculate node of the multiple task, in addition to：

According to first point of coordination type, when all task runs being waited for are on the node distributed is determined One integral dispensing target function value；

According to the corresponding resources requirement of all being waited for for tasks and run time, with reference to the scheduling plan Slightly, second point of coordination type of all being waited for for tasks is determined, second point of coordination type is described for indicating The distribution feelings run calculate node in of all being waited for for tasks in all being waited for for tasks Condition；

According to second point of coordination type, when determining that all task runs being waited for are on the node distributed The second integral dispensing target function value；

If the second integral dispensing target function value is more than the first integral dispensing target function value, all it is in described The task of wait state runs meter according to what second point of coordination type was assigned to all being waited for for tasks On operator node.

6. method according to claim 5, it is characterised in that the first integral dispensing target function value is equal to described the During one point of coordination type, the single node distribution of each node when all task runs being waited for are on the node distributed The sum of target function value；

When the second integral dispensing target function value is equal to second point of coordination type, all task fortune being waited for The single node of each node distributes the sum of target function value when row is on the node distributed.

7. a kind of explorer, it is characterised in that the explorer includes：Receiving unit, resolving cell, estimation are single Member, determining unit and allocation unit；

The receiving unit, the operation for receiving client device submission；

The resolving cell, for being multiple tasks by the breakdown of operation, wherein, each task in the multiple task is equal It is configured with corresponding resources requirement；

The estimation unit, the run time for estimating each task；

The determining unit, for according to the corresponding resources requirement of each task and run time, being adjusted with reference to default Degree strategy, determines first point of coordination type of the multiple task, and first point of coordination type is used to indicate that the multiple task exists The distribution situation run in calculate node in the multiple calculate node, it is preferential that the scheduling strategy includes resource utilization At least one of strategy and efficiency comes first strategy；

The allocation unit, for by the multiple task according to first point of coordination type be assigned to the multiple task can Run in calculate node.

8. explorer according to claim 7, it is characterised in that if the scheduling strategy is that resource utilization is preferential Strategy, then first point of coordination type is so that the list for running each calculate node in calculate node of the multiple task The maximum point coordination type of node resource utilization rate.

9. explorer according to claim 8, it is characterised in that if the scheduling strategy is efficiency comes first strategy, Then first point of coordination type is causes most fast point coordination type of overall execution speed of the operation.

10. the explorer according to claim any one of 7-9, it is characterised in that the estimation unit specifically for：

For each task, handled according to the operation below for first task：

11. the explorer according to claim any one of 7-10, it is characterised in that will be described in the allocation unit Multiple tasks are assigned to after the running in calculate node of the multiple task according to first point of coordination type, described to determine Unit, is additionally operable to：

The allocation unit, if being additionally operable to the second integral dispensing target function value more than the first integral dispensing target letter Numerical value, all being waited for for tasks is assigned to according to second point of coordination type described all in wait shape The task of state is run in calculate node.

12. explorer according to claim 11, it is characterised in that described first integral dispensing target function value etc. When first point of coordination type, the list of each node when all task runs being waited for are on the node distributed The sum of node distribution target function value；

13. a kind of explorer, it is characterised in that the explorer includes：Processor, memory, bus and communication Interface；

The memory is used to store computer executed instructions, and the processor is connected with the memory by the bus, When the explorer is run, the computer executed instructions of memory storage described in the computing device, so that The explorer performs the resource allocation methods in the distributed computing system as described in claim any one of 1-6.

14. a kind of Distributed Computer System, it is characterised in that the Distributed Computer System include multiple calculate nodes and Explorer described in claim any one of 7-12；

Or, the Distributed Computer System includes the explorer described in multiple calculate nodes and claim 13.