CN107621973A - A kind of method for scheduling task and device across cluster - Google Patents
A kind of method for scheduling task and device across cluster Download PDFInfo
- Publication number
- CN107621973A CN107621973A CN201610550845.7A CN201610550845A CN107621973A CN 107621973 A CN107621973 A CN 107621973A CN 201610550845 A CN201610550845 A CN 201610550845A CN 107621973 A CN107621973 A CN 107621973A
- Authority
- CN
- China
- Prior art keywords
- task
- cluster
- feature
- scheduler
- scheduling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application is related to a kind of method for scheduling task and device across cluster, and methods described includes：The task feature for each task that the one or more projects obtained in the operation of the first cluster include, task feature database is established according to the task feature of each task；Task scheduling strategy is determined according to the clustering feature of the second cluster；Being matched with the task scheduling strategy of the task is chosen from the task feature database and treats the list of scheduler task to generate, the list when scheduler task according to the list in scheduler task for will treat that scheduler task is run from first colony dispatching to second cluster described in list.The application can realize task granularity across colony dispatching, flexibility is strong, reduces the wasting of resources, effectively increases data-handling efficiency.
Description
Technical field
The application is related to field of computer technology, more particularly to a kind of method for scheduling task and device across cluster.
Background technology
With the growth of data processing scale, traditional unit computation schema can not meet growing information clothes
Business demand.Cluster (cluster) is one group of computer interconnected independently of each other, by express network, and they constitute one
Calculating group, and can be managed in a uniform manner.Cluster can realize very high arithmetic speed, complete the meter of macrooperation amount
Calculate, there is higher responding ability, overall O＆M cost can be reduced, therefore obtained increasing application.
However, with the expansion of off-line calculation task data scale in the development of big data technology and Data Warehouse for Enterprises,
Individually build that the cost that computing cluster is brought is more and more higher, increasing user tends to no longer be offline calculating service
Cluster and computer room are individually established, but will be in line service and offline business interspersion in same cluster.Interspersion cluster only undertakes meter
Calculation task is without undertaking store tasks, to reach cost-effective purpose.At this moment, it is necessary to which a part run in offline cluster
Calculating task is dispatched on interspersion cluster and run.Therefore, what the task scheduling how realized across cluster merited attention as one
Problem.
In the prior art, a kind of across colony dispatching method be present, can by all tasks under specific transactions or project from
One colony dispatching is on another cluster.This mode is merely able to realize the scheduling of business or project level, dispatches granularity
It is very thick, the defects of flexibility is not strong be present.If in addition, there is an abnormal task it is necessary to be wrapped in whole business in the business
All tasks for including the task of normal operation are cancelled, and all tasks of whole business are moved into weight on original cluster again
New operation.This mode causes computing resource and waste of time, and reduces data-handling efficiency.
The content of the invention
The purpose of the application is to provide a kind of method for scheduling task and device across cluster, it is possible to achieve task granularity
Across colony dispatching, flexibility is strong, reduces the wasting of resources, effectively increases data-handling efficiency.
In a first aspect, this application provides a kind of task scheduling system, the system includes task scheduling apparatus and control
Cluster server, wherein：The task scheduling apparatus is used to obtain what the one or more projects run in the first cluster included
The task feature of each task, task feature database is established according to the task feature of each task；It is special according to the cluster of the second cluster
Sign determines task scheduling strategy；Matching with the task scheduling strategy for task is chosen from the task feature database to treat to generate
The list of scheduler task；The control cluster server is used for the list that scheduler task is treated according to, treats that scheduling is appointed by described
Business is run from first colony dispatching to second cluster.
Alternatively, the control cluster server is additionally operable to configuration task operation result output policy, the task run
As a result output policy be used for configure by the operation result for each task run in second cluster be sent to first cluster,
Configure the operation result as described in second cluster-based storage or configure second cluster and first cluster by each task
Operation result be sent to control cluster server.
Alternatively, the control cluster server is additionally operable to according to the dependence between task, in advance by previous task
Operation result send latter task where cluster.
Second aspect, this application provides a kind of method for scheduling task across cluster, methods described includes：Obtain first
The task feature for each task that one or more projects of cluster operation include, establishes according to the task feature of each task and appoints
Business feature database；Task scheduling strategy is determined according to the clustering feature of the second cluster；From the task feature database choose with it is described
The task of task scheduling strategy matching treats the list of scheduler task to generate, and the list for treating scheduler task is used to appoint in scheduling
It will treat that scheduler task is transported from first colony dispatching to second cluster described in list according to the list during business
OK.
The third aspect, this application provides a kind of task scheduling apparatus across cluster, described device includes：Task feature database
Unit is established, the task feature for each task that one or more projects for obtaining in the operation of the first cluster include, according to institute
The task feature for stating each task establishes task feature database；Policy determining unit, for being determined according to the clustering feature of the second cluster
Task scheduling strategy；Task chooses unit, for choosing what is matched with the task scheduling strategy from the task feature database
Task treats the list of scheduler task to generate, and the list when scheduler task is used for will according to the list in scheduler task
Treat that scheduler task is run from first colony dispatching to second cluster described in list.
Fourth aspect, it is used for the device across the task scheduling of cluster this application provides a kind of, includes memory, and
One or more than one program, one of them or more than one program storage are configured to by one in memory
Individual either more than one computing device is one or more than one program bag contains the instruction for being used for being operated below：Obtain
The task feature of each task that one or more projects in the operation of the first cluster include is taken, it is special according to the task of each task
Sign establishes task feature database；Task scheduling strategy is determined according to the clustering feature of the second cluster；Selected from the task feature database
Take being matched with the task scheduling strategy for task and treat the list of scheduler task to generate, the list for treating scheduler task is used for
According to the list scheduler task will be treated described in list from first colony dispatching to described second in scheduler task
Run on cluster.
5th aspect, this application provides a kind of method for scheduling task, methods described includes：Cluster server is controlled to obtain
Treat scheduler task list；Control cluster server treats the list of scheduler task according to, treats scheduler task from described by described
First colony dispatching is to running on second cluster.
6th aspect, this application provides one kind to control cluster server, including：Acquiring unit, wait to dispatch for obtaining
Task list；Scheduling unit, for treating the list of scheduler task according to, treat scheduler task from first cluster by described
It is dispatched on second cluster and runs.
7th aspect, this application provides one kind to control cluster server, includes memory, and one or one
More than program, one of them or more than one program storage in memory, and be configured to by one or one with
Upper computing device is one or more than one program bag contains the instruction for being used for being operated below：Scheduler task is treated in acquisition
List；According to the list for treating scheduler task, treat scheduler task from first colony dispatching to the described second collection by described
Run on group.
The method for scheduling task and device across cluster that the embodiment of the present application provides, can be according to the cluster spy of the second cluster
Sign determines task scheduling strategy, the task feature choosing of each task included according to the one or more projects run in the first cluster
Take being matched with the task scheduling strategy for task and treat the list of scheduler task to generate, be easy to that scheduler task collection will be treated by described
Task in conjunction is run from the first colony dispatching to the second cluster.The embodiment of the present application can realize across the cluster tune of task granularity
Degree, flexibility is strong, reduces the wasting of resources, effectively increases data-handling efficiency.
Brief description of the drawings
In order to illustrate more clearly of the technical scheme in the embodiment of the present application, make required in being described below to embodiment
Accompanying drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the present application, for
For those of ordinary skill in the art, without having to pay creative labor, it can also be obtained according to these accompanying drawings
His accompanying drawing.
Fig. 1 is the adaptable illustrative application scene of the embodiment of the present application；
Fig. 2 is a kind of method for scheduling task flow chart across cluster that the embodiment of the present application provides；
Fig. 3 be the embodiment of the present application provide another across cluster method for scheduling task flow chart；
Fig. 4 is a kind of task scheduling apparatus schematic diagram across cluster that the embodiment of the present application provides；
Fig. 5 is a kind of block diagram of the device of task scheduling for across cluster according to an exemplary embodiment；
Fig. 6 is the task scheduling system schematic diagram that the embodiment of the present application provides；
Fig. 7 is the method for scheduling task schematic diagram that the embodiment of the present application provides；
Fig. 8 is a kind of control cluster server schematic diagram that the embodiment of the present application provides；
Fig. 9 is another control cluster server schematic diagram that the embodiment of the present application provides.
Embodiment
The embodiment of the present application provide the method for scheduling task method and device across cluster, it is possible to achieve task granularity across
Colony dispatching, flexibility is strong, reduces the wasting of resources, effectively increases data-handling efficiency.
To enable present invention purpose, feature, advantage more obvious and understandable, below in conjunction with the application
Accompanying drawing in embodiment, the technical scheme in the embodiment of the present application is described, it is clear that described embodiment is only this
Application part of the embodiment, and not all embodiments.Based on the embodiment in the application, those of ordinary skill in the art are not having
The every other embodiment obtained under the premise of creative work is made, belongs to the scope of the application protection.
Referring to Fig. 1, for the exemplary application scene of the embodiment of the present application.The method and apparatus that the embodiment of the present application provides can
With applied to scene as shown in Figure 1, wherein, the method and apparatus that the embodiment of the present application provides can be by the first cluster 100
Task scheduling into the second cluster 200.As shown in figure 1, one or more projects, such as item is assigned in the first cluster in advance
Mesh (or referred to as business), such as project 101, project 102, each project of project 103 ... include some tasks again, such as
Task 1, task 2, task 3, task 4 ... similarly, some projects have also been run in the second cluster.When meeting specified conditions,
Need the first cluster task being dispatched to when being run in the second cluster, the method and apparatus that the embodiment of the present application provides can be realized
The scheduling of task granularity, you can so that the partial task in the one or more projects run or calculated in the first cluster to be dispatched to
Run in second cluster, to improve the overall utilization ratio of system.Such as by the task 1 in project 101, task 2, and project
Task 3, task 4 in 102, which are dispatched in the second cluster 200, to be run, and without dispatching whole project 101, project 102, is realized
The flexible dispatching of task.Wherein, the task scheduling apparatus 400 is used to obtain one or more items in the operation of the first cluster
The task feature for each task that mesh includes, task feature database is established according to the task feature of each task；According to the second cluster
Clustering feature determine task scheduling strategy；Being matched with the task scheduling strategy for task is chosen from the task feature database
The list of scheduler task is treated with generation；The control cluster server 300 is used for the list that scheduler task is treated according to, by institute
State and treat that scheduler task is run from first colony dispatching to second cluster.It should be noted that above-mentioned application scenarios
It is for only for ease of and understands the application and show, presently filed embodiment is unrestricted in this regard.On the contrary, the application
Embodiment can apply to applicable any scene.
With reference first to Fig. 6, the task scheduling system schematic diagram provided for the embodiment of the present application.As shown in fig. 7, the application is real
Applying the task scheduling system of example offer includes task scheduling apparatus 400 and control cluster server 300, wherein：
The task scheduling apparatus 400 is used to obtain each task that the one or more projects run in the first cluster include
Task feature, task feature database is established according to the task feature of each task；Determined according to the clustering feature of the second cluster
Task scheduling strategy；Being matched with the task scheduling strategy for task is chosen from the task feature database and treats that scheduling is appointed to generate
The list of business.
The control cluster server 300 is used to treat the list of scheduler task according to, by it is described treat scheduler task from
First colony dispatching is to running on second cluster.
In some embodiments, the control cluster server 300 is additionally operable to configuration task operation result output policy,
The task run result output policy, which is used to configure, is sent to the operation result for each task run in second cluster
First cluster, configuration operation result or configuration second cluster and described the as described in second cluster-based storage
The operation result of each task is sent to control cluster server by one cluster.For example, control cluster server 300 is used to match somebody with somebody
Put task run result output policy.For example, control cluster server 300 can be configured by the second cluster-based storage in the second cluster
The operation result of each task of operation, or, configure and the operation result of partial task be sent to the first cluster by the second cluster,
Wherein, the partial task is treating in scheduler task list for the task.Or control cluster server 300 can match somebody with somebody
Put and the operation result of operating on respective cluster for task is sent to by domination set by first cluster and the second cluster respectively
Group's server, the operation result for the task that projects include is collected by control cluster server.
In some embodiments, the control cluster server is additionally operable to according to the dependence between task, in advance
The operation result of previous task is sent to the cluster where latter task.For example, have project 101 in the operation of the first cluster,
Project 101 includes task 1, task 2, task 3, task 4, wherein, task 2, task 3, the fortune for performing dependence task 1 of task 4
Row result.When determining task scheduling strategy according to the clustering feature of the second cluster, by appointing in the project 101 of the first cluster operation
Business 2, task 3, task 4 are dispatched to when being run on the second cluster, can be in task implementation procedure, in advance by the operation of task 1
As a result the second cluster is sent to, is used when being performed so as to task 2, task 3, task 4.Such processing mode, without task 2,
Respectively to the operation result of the first cluster request task 1, treatment effeciency is high, reduces the wasting of resources for task 3, task 4.
The scheme of the application is illustrated on the whole from task scheduling system above, below from task scheduling apparatus
Angle illustrates to the dispatching method of the application.Specifically, below in conjunction with accompanying drawing 2 to accompanying drawing 3 to the exemplary reality of the application
The method for scheduling task across cluster exemplified is applied to be introduced.
It is the method for scheduling task flow chart across cluster that the embodiment of the present application provides, as shown in Fig. 2 this Shen referring to Fig. 2
Please the method for scheduling task across cluster of embodiment include：
S201, the task feature for obtaining each task that the one or more projects run in the first cluster include, according to institute
The task feature for stating each task establishes task feature database.
During specific implementation, the first cluster is former cluster, and the second cluster is target cluster, the method that the embodiment of the present application provides
Task scheduling in first cluster can be run into the second cluster.For convenience of description, hereafter using the first cluster as from
Line cluster, the second cluster be interspersion cluster exemplified by illustrate.Certainly, the first cluster can also be interspersion cluster, the second cluster
It can also be offline cluster.The application is not limited to this.It should be noted that allocated in advance one or more projects to
First cluster, the method that the application provides can formulate scheduling strategy in one or more of project operations, calculating process
With migration task, part task immigration is extracted from least a certain project to the second cluster, is imitated with improving overall the utilizing of system
Rate.When obtaining the task feature of each task, the feature for the task that all items that the first cluster is run include can be obtained,
The feature for each task that at least one project that can be obtained in the operation of the first cluster includes.
In some embodiments, each task included in the one or more projects obtained in the operation of the first cluster is appointed
Before feature of being engaged in, it can also include：It is extracted in appointing for each task that one or more projects that first cluster is run include
Business fingerprint.Wherein, the task fingerprint of each task is used for each task of unique mark.During specific implementation, the task fingerprint of extraction
It is the unique mark of each calculating task, is also based on least unit of the task granularity progress across colony dispatching.Due to cluster
Even if any change does not occur for the calculating task calculating logic of daily timing operation in, as the factor such as run time is not
It is consistent to cause the daily code really performed different, it is therefore desirable to the variable in task code is removed, calculating is found out and appoints
It is invariable in business to quantify, it is used for one calculating times of unique mark to generate one uniquely and stablize constant task fingerprint
Business.Specifically, task fingerprint can be extracted from the running log of task.
In some embodiments, each for being extracted in one or more projects that first cluster is run and including
The task fingerprint of business includes：The code of acquisition task, the variable in the code of the task is removed, the task after acquisition processing
Code；Computing is carried out to the code of the task after the processing using first method, it is uniquely right with the code of the task to obtain
Task fingerprint of the operation result answered as the task.For example, the code of task is obtained from task run daily record, will
All variables remove in task code, and the variable for example can be the date.
For example, for same section of code, it is in the code that on January 27th, 2016 performs：
Select*from a where ds=20160127
It is in the code that on January 26th, 2016 performs：
Select*from a where ds=20160126
When handling task code, date variable will can be identified wherein as " 20160127 ",
And delete, only remaining ' the such one section of text that will not change of select*from a where ds=', after processing
Task code.Computing then is carried out to the code of the task after the processing using first method, obtain with it is described uniquely right
The operation result answered, as the task fingerprint of the task.For example, the first method can be that (English full name is MD5
Message Digest Algorithm 5, Chinese full name are Message Digest Algorithm 5) computing is carried out to it, to generate only
One task fingerprint.It is of course also possible to use other method, such as (English full name is Message Digest to MD3
Algorithm3, Chinese full name are the Message Digest 5 third edition), (English full name is Message Digest to MD4
Algorithm 4, Chinese full name are Message Digest 5 fourth edition) etc., herein without limiting.The task fingerprint can be only
The one mark task.
In some embodiments, each task that the acquisition includes in one or more projects of the first cluster operation
Task feature, establishing task feature database according to the task feature of each task includes：Obtain one in the operation of the first cluster
Or the task feature of each task that multiple projects include, preserve the task fingerprint of each task and the corresponding relation of the task feature
To establish task feature database.During specific implementation, the operation characteristic of each task using task fingerprint as granularity, can be extracted and calculated
Data collect one task feature database of generation as its task feature, and by each task feature, are stored in the task feature database
There is the corresponding relation of task fingerprint and task feature.
Wherein, the task feature can include it is following in any one or more：
(1) the input data amount of task.From the reading data of different computer rooms during being specifically as follows each task run
Amount, this feature are associated with bandwidth accounting when across colony dispatching reading data.
(2) amount of computational resources of task consumption.It is specifically as follows the CPU core taken during the operation of each calculating task
The number * times.Scale exemplified by interspersion cluster determines the calculating total amount of being scheduled for for task.
(3) intermediate result storage (shuffle) data volume of task.In task run, some intermediate results can be cached,
This feature is associated with the local disk capacity of interspersion cluster.
(4) the EMS memory occupation amount of task.It is specifically as follows the internal memory accounting of each task, this feature and interspersion cluster
Memory amount be associated
(5) output data quantity of task.It is specifically as follows the final output data volume of each task, this feature determines
Bandwidth accounting when task write-back original cluster.The output data quantity of task and the input data amount of task and value should not
More than the bandwidth limit of interspersion cluster.
S202, task scheduling strategy determined according to the clustering feature of the second cluster.
During specific implementation, it can determine that the cluster of the second cluster is special according to the type and network type of second cluster
Sign, the clustering feature of second cluster include amount of bandwidth, disk size, memory size and/or calculate capacity.Illustrate
It is bright, disk size, memory size can be determined according to the type of the second cluster and/or calculates capacity, can be according to the second cluster
Network type determine amount of bandwidth.
In some embodiments, according to the clustering feature of the second cluster determine task scheduling strategy can include it is following in
Any one or more：
(1) when the bandwidth of second cluster is less than the first predetermined threshold value, priority scheduling calculates input than high task；
Wherein, it is described to calculate input than the amount of computational resources and the ratio of the input data amount of task for task consumption.Wherein, first is pre-
If threshold value can rule of thumb with need to set.If for example, the bandwidth of second cluster is less than the first predetermined threshold value,
Bottleneck will be turned into by illustrating the bandwidth of the second cluster, and input at this moment can be calculated with priority scheduling than high task.Wherein, calculate defeated
Enter has the characteristics of high calculating, low bandwidth take than high task, the task of this feature of priority scheduling, can mitigate the second collection
The bandwidth pressure of group.
(2) when the disk size of second cluster is less than the second predetermined threshold value, priority scheduling calculates storage than high
Task；Wherein, it is described to calculate storage than the amount of computational resources and the ratio of the intermediate result storage data quantity of task for task consumption
Value.Wherein, the second predetermined threshold value can rule of thumb with need to set.If for example, disk size of second cluster
Less than the second predetermined threshold value, then bottleneck will be turned into by illustrating the local disk capacity of the second cluster, at this moment can be in terms of priority scheduling
Storage is calculated than high task.Wherein, calculate storage has the characteristics of high calculating, low disk take, priority scheduling than high task
The task of this feature, the disk pressure of the second cluster can be mitigated.
(3) when the memory size of second cluster is less than three predetermined threshold values, priority scheduling EMS memory occupation amount is low
Task.Wherein, the 3rd predetermined threshold value can rule of thumb with need to set.If for example, the memory size is less than the 3rd
Predetermined threshold value, then bottleneck will be turned into by illustrating the memory size of the second cluster, at this moment can be low with priority scheduling EMS memory occupation amount
Task.Wherein, the low task of EMS memory occupation amount has the characteristics of high calculating, low EMS memory occupation, and this feature of priority scheduling is appointed
Business, the memory pressure of the second cluster can be mitigated.
(4) rush hour and ebb moment of the second cluster task operation are determined, is preferentially adjusted at the ebb moment
Spend the task of the first cluster.For example, task run has rush hour and ebb moment, rush hour computing resource takes
More, resource-constrained, rush hour computing resource occupancy is less, resource is relatively sufficient.Therefore, adjusted by the task of the first cluster
Spend to when running on the second cluster, can be with flood peak staggered regulation, such as ebb moment in the operation of the second cluster task, the collection of scheduling first
The task of group improves the overall operational efficiency of system to running on the second cluster, with this, makes full use of computing resource.
It should be noted that above-mentioned task scheduling strategy can be separately from can also combine applicable.Certainly, with
Upper is only the exemplary illustration of task scheduling strategy, is not intended as the limitation to the application.
It should be noted that S201 and S202 order can be performed reversedly, or it is performed in parallel.
S203, the task that selection matches with the task scheduling strategy from the task feature database treat that scheduling is appointed to generate
The list of business, the list when scheduler task are used to wait to dispatch described in list according to the list in scheduler task
Task is run from first colony dispatching to second cluster.
, can be according to the task scheduling strategy determined in S202, according to each in the task feature database during specific implementation
The task feature of business, the task of the task scheduling strategy is determined for compliance with, to generate candidate tasks set.Then, further according to institute
State the clustering feature of the second cluster and the task feature of each task in the candidate tasks set determines task scheduling scale, from
The scheduler task for the treatment of with the task scheduling Size Match is chosen in the candidate tasks set, the list of scheduler task is treated in generation
In some embodiments, can be according to described when it is determined that task scheduling scale is the quantity of scheduler task
The amount of computational resources of each task consumption determines task scheduling scale in the calculating capacity of two clusters and the candidate tasks set.
If for example, the task scheduling strategy be priority scheduling calculate input than high task when, can be according to each
The task feature of task, the calculating input ratio of each task is calculated, and the size that each task is inputted into ratio according to calculating is from high to low
It is ranked up.It should be noted that the 4th predetermined threshold value can be set, input is calculated than appointing more than the 4th predetermined threshold value
Business can enter candidate tasks set.Then, further according to the amount of calculation one for calculating capacity and each task chosen of interspersion cluster
Rise and determine scheduler task scale.Assuming that interspersion cluster has 100 machines altogether, each 32 CPU cores of machine, the total capacity of cluster is just
It is 3200 core * days；If the computing resource consumption of the task in each candidate tasks set is 1 core * days, task scheduling scale can
To be 3200 tasks.
If, can basis again for example, when the task scheduling strategy calculates storage than high task for priority scheduling
The task feature of each task, calculate each task calculating storage ratio, and by each task according to calculate store ratio size from it is high to
It is low to be ranked up.It should be noted that the 5th predetermined threshold value can be set, storage is calculated than more than the 5th predetermined threshold value
Task can enter candidate tasks set.Then, further according to the amount of calculation for calculating capacity and each task chosen of interspersion cluster
Scheduler task scale is determined together, such as the task scale of determination is 100, then priority scheduling calculates storage preceding 100 than coming
The task of position, scheduler task set is treated with generation.
If, can basis again for example, when the task scheduling strategy is priority scheduling EMS memory occupation amount low task
The task feature of each task, calculates the EMS memory occupation amount of each task, and by each task according to EMS memory occupation amount size from it is low to
Height is ranked up.It should be noted that the 6th predetermined threshold value can be set, EMS memory occupation amount is less than the 6th predetermined threshold value
Task can enter candidate tasks set.Then, further according to the amount of calculation for calculating capacity and each task chosen of interspersion cluster
Determine scheduler task scale together, such as the task scale of determination is 200, then priority scheduling EMS memory occupation amount according to from it is small to
Big order sequence, the coming first 200 of the task, scheduler task set is treated with generation.
It is of course also possible to according to the bandwidth of the second cluster, EMS memory occupation amount, disk size and/or calculating capacity and respectively
The task feature of task integrates the scheduling scale of determination task, and the application is to this without limiting.
In generation after the list of scheduler task, control cluster server is used for the list that scheduler task is treated according to,
Treat that scheduler task is run from first colony dispatching to second cluster by described.Specifically, wait to dispatch determining
After task list, you can to treat the task fingerprint of each task in scheduler task row according to, by the task from described the
One colony dispatching is to running on second cluster.
In the embodiment of the present application, it is possible to achieve task granularity it is strong across colony dispatching, flexibility.Due to can be according to
The clustering feature of two clusters determines suitable scheduling strategy and matching task, therefore can realize filling for computing resource
Divide and utilize, reduce the wasting of resources, and effectively increase data-handling efficiency.Further, since be the scheduling of task granularity, therefore,
If there is abnormal task, it is only necessary to cancel the abnormal task, and influence the operation of other normal tasks, improve and be
The fault-tolerant ability of system, reduce the consuming of time and resource.
Be more clearly understood that embodiment of the application under concrete scene for the ease of those skilled in the art, below with
The application embodiment is introduced one specific example.It should be noted that the specific example is only to cause this area skill
Art personnel more clearly understand the application, but presently filed embodiment is not limited to the specific example.
Referring to Fig. 3, another provided for the embodiment of the present application is across the method for scheduling task flow chart of cluster, shown method
It can include：
S301, the task fingerprint of each task of offline cluster is extracted from task run daily record.
S302, using task fingerprint as granularity, the task feature of each task is extracted and calculated, generates task feature database.
Wherein, the task feature database saves the corresponding relation of task fingerprint and task feature.
S303, the clustering feature of interspersion cluster is determined according to the type of the interspersion cluster and network type.
S304, task scheduling strategy is determined according to the clustering feature of the interspersion cluster.
S305, according to the task scheduling strategy, the task feature of each task, selection and institute from the task feature database
The task of task scheduling strategy matching is stated, generates candidate tasks set.
S306, the calculating that each task consumes in capacity and the candidate tasks set is calculated according to the interspersion cluster
Stock number determines task scheduling scale.
S307, the scheduler task for the treatment of with the task scheduling Size Match, generation are chosen from the candidate tasks set
Treat scheduler task set.
S308, treating in scheduler task set for the task is transported from the offline colony dispatching to the interspersion cluster
OK.
It should be noted that the specific implementation of above steps is referred to embodiment illustrated in fig. 2 progress.
The dispatching method of the application is illustrated from the angle of task scheduling apparatus above.Below in conjunction with accompanying drawing 7 from
The method for scheduling task across cluster shown in the application exemplary embodiment is introduced the angle of control cluster server.
Method shown in Fig. 7 is applied to control cluster server, and shown method can for example include：
S701, control cluster server, which obtains, treats scheduler task list.
During specific implementation, control cluster server can obtain the list for treating scheduler task from task scheduling apparatus.Obtain
Mode can be very flexible, can be the task scheduling apparatus in generation after scheduler task list, actively send
To control cluster server or control cluster server pulled to task scheduling apparatus and described treat that scheduler task arranges
Table.The list for treating scheduler task is by obtaining the one or more run in the first cluster by the task scheduling apparatus
The task feature for each task that project includes, task feature database is established according to the task feature of each task, according to the second collection
The clustering feature of group determines task scheduling strategy, and times matched with the task scheduling strategy is chosen from the task feature database
It is engaged in and generates.The generating process for treating scheduler task list specifically refers to the description of Fig. 2 and embodiment illustrated in fig. 3,
This is repeated no more.
S702, control cluster server treat the list of scheduler task according to, treat scheduler task from described the by described
One colony dispatching is to running on second cluster.
During specific implementation, the control server can refer to according to the treating each task in scheduler task row of the task
Line, the task is run from first colony dispatching to second cluster.
In some embodiments, the control server can be with configuration task operation result output policy, described
Business operation result output policy is used for the output for controlling the operation result of each task of each cluster operation.
In some embodiments, the control cluster server configuration task operation result output policy is specially：Institute
State control cluster server configuration and the operation result for each task run in second cluster is sent to first cluster.
For example, the control cluster server can configure and the operation result of partial task is sent into the first collection by the second cluster
Group, wherein, the partial task is treating in scheduler task list for the task.For example, task scheduling apparatus is waited to adjust generating
After the list of degree task, control cluster server can be according to treating the list of scheduler task by the task in list from the first cluster
It is dispatched in the second cluster and runs, and configures and be sent to the final operation result of the task in the list by the second cluster
First cluster.This mode, task run result can be obtained by the first cluster by being interacted without the first cluster with the second cluster,
It is particularly advantageous especially for the different task in same project to be distributed to the scene run in different clusters.Former cluster
Operation result of the scheduler task in purpose cluster can be obtained, and can be by each in the operation of different clusters under same project
The operation result of business collects, and effectively increases the overall data-handling efficiency of system and concertedness.
In some embodiments, the control cluster server configuration task operation result output policy is specially：Institute
State the operation result for each task that control cluster server configuration is run by second cluster-based storage in second cluster.
In this embodiment, the operation result obtained after the task run in scheduler task list can be stored in by the second cluster
It is local.The result of the first cluster or other systems need not be more particularly synchronized to, is effectively reduced between cluster or each
Data transfer pressure between system.
In some embodiments, the control cluster server configuration task operation result output policy is specially：Institute
State control cluster server configuration second cluster and the operation result of each task is sent to the control by first cluster
Cluster server processed.In these embodiments, will be operated on respective cluster respectively by first cluster and the second cluster
The operation result of task be sent to control cluster server, the fortune for the task that projects include is collected by control cluster server
Row result.In this embodiment, stored by control cluster server centered, manage each task run result, and can root
Handled accordingly according to task run result.For example, the first cluster, the second cluster are by task run failure or successfully tie
Fruit is sent to control cluster server.When controlling cluster server to find a certain task run failure, other moneys can be dispatched
Source come handle operation failure task or by run failure task deleted in time from cluster, avoid it to other tasks
Influence.Certainly, specific processing mode can be very flexible, herein without limiting.
In some embodiments, the control cluster server can also be according to the dependence between task, in advance
The operation result of previous task is sent to the cluster where latter task.For example, have project 101 in the operation of the first cluster,
Project 101 includes task 1, task 2, task 3, task 4, wherein, task 2, task 3, the fortune for performing dependence task 1 of task 4
Row result.When determining task scheduling strategy according to the clustering feature of the second cluster, by appointing in the project 101 of the first cluster operation
Business 2, task 3, task 4 are dispatched to when being run on the second cluster, can be in task implementation procedure, in advance by the operation of task 1
As a result the second cluster is sent to, is used when being performed so as to task 2, task 3, task 4.Such processing mode, without task 2,
Respectively to the operation result of the first cluster request task 1, treatment effeciency is high, reduces the wasting of resources for task 3, task 4.
Above is the embodiment of the present application is provided across cluster method for scheduling task carry out detailed description, below it is right
The task scheduling apparatus across cluster that the application provides is described in detail.
Fig. 4 is the task scheduling apparatus schematic diagram across cluster that the embodiment of the present application provides, as shown in figure 4, the application
Include across the task scheduling apparatus of cluster：Task feature database establishes unit 401, policy determining unit 402 and task and chooses unit
403, wherein：
The task feature database establish unit 401 be used for obtain one or more projects for being run in the first cluster include
The task feature of each task, task feature database is established according to the task feature of each task.
The policy determining unit 402 is used to determine task scheduling strategy according to the clustering feature of the second cluster.
The task chooses unit 403 and is used to choose what is matched with the task scheduling strategy from the task feature database
Task treats the list of scheduler task to generate, and the list when scheduler task is used for will according to the list in scheduler task
Treat that scheduler task is run from first colony dispatching to second cluster described in list.
Wherein, described device also includes：
Task fingerprint extracting unit, for each task included in the one or more projects obtained in the operation of the first cluster
Task feature before, be extracted in the task fingerprint of each task that one or more projects of first cluster operation include；
The task fingerprint of each task is used for each task of unique mark；
The task feature database is established unit and is specifically used for：
The task feature for each task that the one or more projects obtained in the operation of the first cluster include, preserves each task
The corresponding relation of task fingerprint and the task feature is to establish task feature database；
Wherein, the task fingerprint extracting unit includes：
Subelement is handled, for obtaining the code of task, the variable in the code of the task is removed, after acquisition processing
The code of task；
Computing subelement, for carrying out computing, acquisition and institute to the code of the task after the processing using first method
State the code of task uniquely task fingerprint of the corresponding operation result as the task.
Wherein, the policy determining unit include it is following in any one or more subelements：
First tactful determination subelement, for when the bandwidth of second cluster is less than the first predetermined threshold value, preferentially adjusting
Degree calculates input than high task；Wherein, it is described to calculate input than the amount of computational resources and the input number of task for task consumption
According to the ratio of amount；
Second tactful determination subelement, it is excellent for when the disk size of second cluster is less than the second predetermined threshold value
First scheduling calculates storage than high task；Wherein, it is described to calculate storage than in the amount of computational resources and task for task consumption
Between result storage data quantity ratio；
3rd tactful determination subelement, it is excellent for when the memory size of second cluster is less than three predetermined threshold values
First dispatch the low task of EMS memory occupation amount.
4th tactful determination subelement, for determining the rush hour and ebb moment of the second cluster task operation,
The task of first cluster is preferentially dispatched at the ebb moment.
Wherein, the task chooses unit and includes candidate tasks set generation unit and treat that scheduler task list generation is single
Member, wherein, the candidate tasks set generation unit is used for according to the task scheduling strategy, is selected from the task feature database
Being matched with the task scheduling strategy for task is taken to generate candidate tasks set；It is described to treat that scheduler task list generation unit is specific
Task feature for each task in the clustering feature according to second cluster and the candidate tasks set determines task
Scheduling scale, chosen from the candidate tasks set and treat scheduler task with the task scheduling Size Match, generated and wait to adjust
The list of degree task
In some embodiments, it is described to treat that scheduler task list generation unit is specifically used for：
The computing resource that each task consumes in capacity and the candidate tasks set is calculated according to second cluster
Amount determines task scheduling scale.
The function of above-mentioned each unit may correspond to the above-mentioned method for scheduling task across cluster of Fig. 2~Fig. 3 detailed descriptions
Processing step, repeated no more in this.
It is a kind of block diagram of device for task scheduling according to an exemplary embodiment referring to Fig. 5.The number
Include according to processing unit 500：At least one processor 501 (such as CPU), memory 502 and at least one communication bus 503,
For realizing the connection communication between these devices.Processor 501 is used to perform the executable module stored in memory 502,
Such as computer program.Memory 502 may include high-speed random access memory (RAM：Random Access Memory),
Non-labile memory (non-volatile memory), for example, at least a magnetic disk storage may also also be included.One
Or more than one program storage is in memory, and be configured to by one or more than one processor 501 perform it is described
One or more than one program bag contain the instruction for being used for being operated below：
The task feature for each task that the one or more projects obtained in the operation of the first cluster include, according to described each
The task feature of business establishes task feature database；
Task scheduling strategy is determined according to the clustering feature of the second cluster；
Being matched with the task scheduling strategy for task is chosen from the task feature database and treats scheduler task to generate
List, the list when scheduler task are used to that scheduler task will to be treated described in list according to the list in scheduler task
Run from first colony dispatching to second cluster.
Wherein, the processor 501 is specific is additionally operable to perform one or more than one program bag containing for carrying out
The instruction operated below：
It is extracted in the task fingerprint for each task that one or more projects that first cluster is run include；Described each
The task fingerprint of business is used for each task of unique mark；
The task feature for each task that the one or more projects obtained in the operation of the first cluster include, preserves each task
The corresponding relation of task fingerprint and the task feature is to establish task feature database；
According to the task fingerprint for treating each task in scheduler task set, the task is adjusted from first cluster
Spend on second cluster and run.
Wherein, the processor 501 is specific is additionally operable to perform one or more than one program bag containing for carrying out
The instruction operated below：
The code of acquisition task, remove the variable in the code of the task, the code of the task after acquisition processing；
Computing is carried out to the code of the task after the processing using first method, obtained unique with the code of the task
Task fingerprint of the corresponding operation result as the task.
Wherein, the processor 501 is specific is additionally operable to perform one or more than one program bag containing for carrying out
The instruction operated below：
Obtain each task of the first cluster input data amount, consumption amount of computational resources, intermediate result storage data quantity,
Any one or more in EMS memory occupation amount, output data quantity is as task feature.
Wherein, the processor 501 is specific is additionally operable to perform one or more than one program bag containing for carrying out
The instruction operated below：
The clustering feature of the second cluster is determined according to the type of second cluster and network type, second cluster
Clustering feature includes amount of bandwidth, disk size, memory size and/or calculates capacity.
Wherein, the processor 501 is specific is additionally operable to perform one or more than one program bag containing for carrying out
The instruction operated below：
When the bandwidth of second cluster is less than the first predetermined threshold value, priority scheduling calculates input than high task；Its
In, it is described to calculate input than the amount of computational resources and the ratio of the input data amount of task for task consumption；And/or
When the disk size of second cluster is less than the second predetermined threshold value, priority scheduling calculates storage and appointed than high
Business；Wherein, it is described to calculate storage than the amount of computational resources and the ratio of the intermediate result storage data quantity of task for task consumption；
And/or
When the memory size of second cluster is less than three predetermined threshold values, priority scheduling EMS memory occupation amount is low to appoint
Business；And/or
The rush hour and ebb moment of the second cluster task operation are determined, preferentially dispatches institute at the ebb moment
State the task of the first cluster.
Wherein, the processor 501 is specific is additionally operable to perform one or more than one program bag containing for carrying out
The instruction operated below：
The computing resource that each task consumes in capacity and the candidate tasks set is calculated according to second cluster
Amount determines task scheduling scale.
Referring to Fig. 8, a kind of control cluster server schematic diagram provided for the embodiment of the present application.
One kind control cluster server 800, including：
Acquiring unit 801, scheduler task list is treated for obtaining；
Scheduling unit 802, for treating the list of scheduler task according to, treat scheduler task from the described first collection by described
Group, which is dispatched on second cluster, to be run.
In some embodiments, the control cluster server 800 also includes：Dispensing unit, transported for configuration task
Row result output policy, the task run result output policy are used for the defeated of each task run result for controlling each cluster operation
Go out.
In some embodiments, the dispensing unit is specifically used for each task that configuration will be run in second cluster
Operation result be sent to first cluster.
In some embodiments, the dispensing unit is specifically used for configuration by second cluster-based storage described second
The operation result of each task of cluster operation.
In some embodiments, the dispensing unit is specifically used for configuring second cluster and first cluster will
The operation result of each task is sent to the control cluster server.
In some embodiments, the control cluster server 800 also includes：
Transmitting element, for according to the dependence between task, in advance sending the operation result of previous task latter
Cluster where task.
Referring to Fig. 9, another control cluster server schematic diagram provided for the embodiment of the present application.One kind control cluster clothes
Business device 900 includes：At least one processor 901 (such as CPU), memory 902, and at least one communication bus 903, for reality
Connection communication between these existing devices.Processor 901 is used to perform the executable module stored in memory 902, such as counts
Calculation machine program.Memory 902 may include high-speed random access memory (RAM：Random Access Memory), it is also possible to
Also include non-labile memory (non-volatile memory), for example, at least a magnetic disk storage.One or one
Individual procedure above is stored in memory, and be configured to by one or more than one processor 901 perform it is one or
More than one program bag of person contains the instruction for being used for being operated below：
Scheduler task list is treated in acquisition；
According to the list for treating scheduler task, treat scheduler task from first colony dispatching to described second by described
Run on cluster
Wherein, the processor 901 is specific is additionally operable to perform one or more than one program bag containing for carrying out
The instruction operated below：
Configuration task operation result output policy, the task run result output policy are used to control each cluster operation
The output of each task run result.
Wherein, the processor 901 is specific is additionally operable to perform one or more than one program bag containing for carrying out
The instruction operated below：
The operation result for each task run in second cluster is sent to first cluster by configuration.
Wherein, the processor 901 is specific is additionally operable to perform one or more than one program bag containing for carrying out
The instruction operated below：
Configure the operation result for each task run by second cluster-based storage in second cluster.
Wherein, the processor 901 is specific is additionally operable to perform one or more than one program bag containing for carrying out
The instruction operated below：
Configure second cluster and first cluster and the operation result of each task is sent to the control cluster clothes
Business device.
Wherein, the processor 901 is specific is additionally operable to perform one or more than one program bag containing for carrying out
The instruction operated below：
According to the dependence between task, the operation result of previous task is sent to the collection where latter task in advance
Group.
Professional should further appreciate that, each example described with reference to the embodiments described herein
Unit and algorithm steps, it can be realized with electronic hardware, computer software or the combination of the two, it is hard in order to clearly demonstrate
The interchangeability of part and software, the composition and step of each example are generally described according to function in the above description.
These functions are performed with hardware or software mode actually, application-specific and design constraint depending on technical scheme.
Professional and technical personnel can realize described function using distinct methods to each specific application, but this realization
It is not considered that exceed scope of the present application.
The method that is described with reference to the embodiments described herein can use hardware, computing device the step of algorithm
Software module, or the two combination are implemented.Software module can be placed in random access memory (RAM), internal memory, read-only storage
(ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technical field
In any other form of storage medium well known to interior.
Above-described embodiment, the purpose, technical scheme and beneficial effect of the application are carried out further
Describe in detail, should be understood that the embodiment that the foregoing is only the application, be not used to limit the application
Protection domain, all any modification, equivalent substitution and improvements within spirit herein and principle, done etc., all should include
Within the protection domain of the application.
Claims (23)
- A kind of 1. task scheduling system, it is characterised in that the system includes task scheduling apparatus and control cluster server, its In：The task scheduling apparatus is used for the obtaining each task that the one or more projects run in the first cluster include of the task Feature, task feature database is established according to the task feature of each task；Determine that task is adjusted according to the clustering feature of the second cluster Degree strategy；The task that selection matches with the task scheduling strategy from the task feature database treats the row of scheduler task to generate Table；The control cluster server is used for the list that scheduler task is treated according to, treats scheduler task from described first by described Colony dispatching is to running on second cluster.
- 2. system according to claim 1, it is characterised in that the control cluster server is additionally operable to configuration task operation As a result output policy, the task run result output policy are used for the fortune for configuring each task that will be run in second cluster Row result is sent to first cluster, configuration operation result or configuration described second as described in second cluster-based storage The operation result of each task is sent to control cluster server by cluster and first cluster.
- 3. system according to claim 1, it is characterised in that the control cluster server is additionally operable to according between task Dependence, the operation result of previous task is sent to the cluster where latter task in advance.
- 4. a kind of method for scheduling task, it is characterised in that methods described includes：The task feature for each task that the one or more projects obtained in the operation of the first cluster include, according to each task Task feature establishes task feature database；Task scheduling strategy is determined according to the clustering feature of the second cluster；The task that selection matches with the task scheduling strategy from the task feature database treats the list of scheduler task to generate, The list when scheduler task is used to scheduler task will be treated from institute described in list according to the list in scheduler task The first colony dispatching is stated to run to second cluster.
- 5. according to the method for claim 4, it is characterised in that obtaining one or more projects in the operation of the first cluster Comprising each task task feature before, methods described also includes：It is extracted in the task fingerprint for each task that one or more projects that first cluster is run include；Each task Task fingerprint is used for each task of unique mark；The task feature for obtaining each task that the one or more projects run in the first cluster include, according to described each The task feature of business, which establishes task feature database, to be included：The task feature for each task that the one or more projects obtained in the operation of the first cluster include, preserve the task of each task The corresponding relation of fingerprint and the task feature is to establish task feature database；It is described that treating in scheduler task set for the task is run to bag from first colony dispatching to second cluster Include：According to the task fingerprint for treating each task in scheduler task set, by the task from first colony dispatching to Run on second cluster.
- 6. according to the method for claim 5, it is characterised in that be extracted in the first cluster operation one or more The task fingerprint for each task that individual project includes includes：The code of acquisition task, remove the variable in the code of the task, the code of the task after acquisition processing；Computing is carried out to the code of the task after the processing using first method, obtained uniquely corresponding with the code of the task Task fingerprint of the operation result as the task.
- 7. according to the method for claim 4, it is characterised in that the task feature include it is following in any one or it is more Kind：The input data amount of task；The amount of computational resources of task consumption；The intermediate result storage data quantity of task；The EMS memory occupation amount of task；The output data quantity of task.
- 8. according to the method for claim 4, it is characterised in that the clustering feature of second cluster is according to the described second collection The type and network type of group determines that the clustering feature of second cluster includes amount of bandwidth, disk size, memory size And/or calculate capacity.
- 9. according to the method described in claim 4 to 8 any one, it is characterised in that the cluster according to the second cluster is special Sign determine task scheduling strategy include it is following in any one or more：When the bandwidth of second cluster is less than the first predetermined threshold value, priority scheduling calculates input than high task；Wherein, institute State and calculate input than the amount of computational resources and the ratio of the input data amount of task for task consumption；When the disk size of second cluster is less than the second predetermined threshold value, priority scheduling calculates storage than high task；Its In, it is described to calculate storage than the amount of computational resources and the ratio of the intermediate result storage data quantity of task for task consumption；When the memory size of second cluster is less than three predetermined threshold values, the low task of priority scheduling EMS memory occupation amount；The rush hour and ebb moment of second cluster task operation are determined, preferentially in ebb moment scheduling described the The task of one cluster.
- 10. according to the method for claim 4, it is characterised in that described to be chosen and described from the task feature database The task of business scheduling strategy matching treats that the list of scheduler task includes to generate：According to the task scheduling strategy, the task life matched with the task scheduling strategy is chosen from the task feature database Into candidate tasks set；Task is determined according to the task feature of each task in the clustering feature of second cluster and the candidate tasks set Scheduling scale, chosen from the candidate tasks set and treat scheduler task with the task scheduling Size Match, generated and wait to adjust The list of degree task.
- 11. according to the method for claim 10, it is characterised in that the clustering feature according to second cluster and The task feature of each task determines that task scheduling scale includes in the candidate tasks set：It is true according to the amount of computational resources for calculating each task consumption in capacity and the candidate tasks set of second cluster Determine task scheduling scale.
- 12. a kind of method for scheduling task, it is characterised in that methods described includes：Control cluster server, which obtains, treats scheduler task list；Control cluster server treats the list of scheduler task according to, treats scheduler task from first colony dispatching by described Run on to second cluster.
- 13. according to the method for claim 12, it is characterised in that methods described also includes：Cluster server configuration task operation result output policy is controlled, the task run result output policy is used to control respectively The output of each task run result of cluster operation.
- 14. according to the method for claim 13, it is characterised in that the control cluster server configuration task operation result Output policy is specially：The operation result for each task run in second cluster is sent to described the by the control cluster server configuration One cluster.
- 15. according to the method for claim 13, it is characterised in that the control cluster server configuration task operation result Output policy is specially：The operation for each task that the control cluster server configuration is run by second cluster-based storage in second cluster As a result.
- 16. according to the method for claim 13, it is characterised in that the control cluster server configuration task operation result Output policy is specially：The control cluster server configures second cluster and the operation result of each task is sent to by first cluster The control cluster server.
- 17. according to the method for claim 12, it is characterised in that methods described also includes：The control cluster server in advance sends the operation result of previous task latter according to the dependence between task Cluster where task.
- 18. a kind of task scheduling apparatus, it is characterised in that described device includes：Task feature database establishes unit, and each task that one or more projects for obtaining in the operation of the first cluster include is appointed Business feature, task feature database is established according to the task feature of each task；Policy determining unit, for determining task scheduling strategy according to the clustering feature of the second cluster；Task chooses unit, for choosing being matched with the task scheduling strategy of the task from the task feature database to generate Treat the list of scheduler task, the list when scheduler task is used for the institute in list in scheduler task according to the list State and treat that scheduler task is run from first colony dispatching to second cluster.
- 19. one kind control cluster server, it is characterised in that including：Acquiring unit, scheduler task list is treated for obtaining；Scheduling unit, for treating the list of scheduler task according to, treat scheduler task from first colony dispatching by described Run on to second cluster.
- 20. a kind of be used for the device across the task scheduling of cluster, it is characterised in that includes memory, and one or one Program more than individual, one of them or more than one program storage are configured to by one or one in memory Above computing device is one or more than one program bag contains the instruction for being used for being operated below：The task feature for each task that the one or more projects obtained in the operation of the first cluster include, according to each task Task feature establishes task feature database；Task scheduling strategy is determined according to the clustering feature of the second cluster；The task that selection matches with the task scheduling strategy from the task feature database treats the list of scheduler task to generate, The list when scheduler task is used to scheduler task will be treated from institute described in list according to the list in scheduler task The first colony dispatching is stated to run to second cluster.
- 21. device according to claim 20, the processing implement body be additionally operable to perform it is one or more than one Program bag contains the instruction for being used for being operated below：When the bandwidth of second cluster is less than the first predetermined threshold value, priority scheduling calculates input than high task；Wherein, institute State and calculate input than the amount of computational resources and the ratio of the input data amount of task for task consumption；When the disk size of second cluster is less than the second predetermined threshold value, priority scheduling calculates storage than high task；Its In, it is described to calculate storage than the amount of computational resources and the ratio of the intermediate result storage data quantity of task for task consumption；When the memory size of second cluster is less than three predetermined threshold values, the low task of priority scheduling EMS memory occupation amount.
- 22. device according to claim 20, the processing implement body be additionally operable to perform it is one or more than one Program bag contains the instruction for being used for being operated below：It is true according to the amount of computational resources for calculating each task consumption in capacity and the candidate tasks set of second cluster Determine task scheduling scale.
- 23. one kind control cluster server, it is characterised in that include memory, and one or more than one program, One of them or more than one program storage is configured to by one or more than one computing device in memory One or more than one program bag contains the instruction for being used for being operated below：Scheduler task list is treated in acquisition；According to the list for treating scheduler task, treat scheduler task from first colony dispatching to second cluster by described Upper operation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610550845.7A CN107621973B (en) | 2016-07-13 | 2016-07-13 | Cross-cluster task scheduling method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610550845.7A CN107621973B (en) | 2016-07-13 | 2016-07-13 | Cross-cluster task scheduling method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107621973A true CN107621973A (en) | 2018-01-23 |
| CN107621973B CN107621973B (en) | 2021-10-26 |
Family
ID=61087098
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610550845.7A Active CN107621973B (en) | 2016-07-13 | 2016-07-13 | Cross-cluster task scheduling method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107621973B (en) |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108519910A (en) * | 2018-03-21 | 2018-09-11 | 四川斐讯信息技术有限公司 | A kind of data processing method and system based on single phase in even-numbered days |
| CN108920265A (en) * | 2018-06-27 | 2018-11-30 | 平安科技（深圳）有限公司 | A kind of task executing method and server based on server cluster |
| CN109032804A (en) * | 2018-08-01 | 2018-12-18 | 阿里巴巴集团控股有限公司 | Data processing method and device, server |
| CN109145053A (en) * | 2018-08-01 | 2019-01-04 | 阿里巴巴集团控股有限公司 | Data processing method and device, client, server |
| CN109324896A (en) * | 2018-08-22 | 2019-02-12 | 北京奇虎科技有限公司 | The dispatching method and device of database instance |
| CN110120979A (en) * | 2019-05-20 | 2019-08-13 | 华为技术有限公司 | A kind of dispatching method, device and relevant device |
| CN110597609A (en) * | 2019-09-17 | 2019-12-20 | 深圳市及响科技有限公司 | Cluster migration and automatic recovery method and system |
| TWI687875B (en) * | 2018-02-02 | 2020-03-11 | 香港商阿里巴巴集團服務有限公司 | Distributed transaction processing method and device |
| CN111158909A (en) * | 2019-12-27 | 2020-05-15 | 中国联合网络通信集团有限公司 | Cluster resource allocation processing method, device, equipment and storage medium |
| CN111597024A (en) * | 2020-05-14 | 2020-08-28 | 科东(广州)软件科技有限公司 | Cross-domain cluster processing method and device, electronic equipment and storage medium |
| WO2020192065A1 (en) * | 2019-03-22 | 2020-10-01 | 苏宁云计算有限公司 | Method for achieving cross-cluster high availability, apparatus, system, and device |
| CN112214299A (en) * | 2020-09-30 | 2021-01-12 | 深圳云天励飞技术股份有限公司 | Multi-core processor and task scheduling method and device thereof |
| CN112269641A (en) * | 2020-11-18 | 2021-01-26 | 网易（杭州）网络有限公司 | Scheduling method, scheduling device, electronic equipment and storage medium |
| CN112925619A (en) * | 2021-02-24 | 2021-06-08 | 深圳依时货拉拉科技有限公司 | Big data real-time computing method and platform |
| CN113391902A (en) * | 2021-06-22 | 2021-09-14 | 未鲲(上海)科技服务有限公司 | Task scheduling method and device and storage medium |
| CN116866438A (en) * | 2023-09-04 | 2023-10-10 | 金网络(北京)数字科技有限公司 | Cross-cluster task scheduling method and device, computer equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080256223A1 (en) * | 2007-04-13 | 2008-10-16 | International Business Machines Corporation | Scale across in a grid computing environment |
| CN103207814A (en) * | 2012-12-27 | 2013-07-17 | 北京仿真中心 | Decentralized cross cluster resource management and task scheduling system and scheduling method |
| CN103870322A (en) * | 2012-12-17 | 2014-06-18 | 联发科技股份有限公司 | Method for controlling task migration, other-than-temporary computer readable medium, heterogeneous multi-core system |
| CN104461740A (en) * | 2014-12-12 | 2015-03-25 | 国家电网公司 | Cross-domain colony computing resource gathering and distributing method |
-
2016
- 2016-07-13 CN CN201610550845.7A patent/CN107621973B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080256223A1 (en) * | 2007-04-13 | 2008-10-16 | International Business Machines Corporation | Scale across in a grid computing environment |
| CN103870322A (en) * | 2012-12-17 | 2014-06-18 | 联发科技股份有限公司 | Method for controlling task migration, other-than-temporary computer readable medium, heterogeneous multi-core system |
| CN103207814A (en) * | 2012-12-27 | 2013-07-17 | 北京仿真中心 | Decentralized cross cluster resource management and task scheduling system and scheduling method |
| CN104461740A (en) * | 2014-12-12 | 2015-03-25 | 国家电网公司 | Cross-domain colony computing resource gathering and distributing method |
Cited By (26)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI687875B (en) * | 2018-02-02 | 2020-03-11 | 香港商阿里巴巴集團服務有限公司 | Distributed transaction processing method and device |
| CN108519910A (en) * | 2018-03-21 | 2018-09-11 | 四川斐讯信息技术有限公司 | A kind of data processing method and system based on single phase in even-numbered days |
| CN108920265A (en) * | 2018-06-27 | 2018-11-30 | 平安科技（深圳）有限公司 | A kind of task executing method and server based on server cluster |
| CN109145053A (en) * | 2018-08-01 | 2019-01-04 | 阿里巴巴集团控股有限公司 | Data processing method and device, client, server |
| CN109145053B (en) * | 2018-08-01 | 2021-03-23 | 创新先进技术有限公司 | Data processing method and device, client and server |
| CN109032804A (en) * | 2018-08-01 | 2018-12-18 | 阿里巴巴集团控股有限公司 | Data processing method and device, server |
| US11563805B2 (en) | 2018-08-01 | 2023-01-24 | Advanced New Technologies Co., Ltd. | Method, apparatus, client terminal, and server for data processing |
| US11106488B2 (en) | 2018-08-01 | 2021-08-31 | Advanced New Technologies Co., Ltd. | Blockchain read/write data processing method, apparatus, and server |
| CN109324896A (en) * | 2018-08-22 | 2019-02-12 | 北京奇虎科技有限公司 | The dispatching method and device of database instance |
| WO2020192065A1 (en) * | 2019-03-22 | 2020-10-01 | 苏宁云计算有限公司 | Method for achieving cross-cluster high availability, apparatus, system, and device |
| CN110120979A (en) * | 2019-05-20 | 2019-08-13 | 华为技术有限公司 | A kind of dispatching method, device and relevant device |
| CN110120979B (en) * | 2019-05-20 | 2023-03-10 | 华为云计算技术有限公司 | Scheduling method, device and related equipment |
| WO2020233120A1 (en) * | 2019-05-20 | 2020-11-26 | 华为技术有限公司 | Scheduling method and apparatus, and related device |
| EP3866441A4 (en) * | 2019-05-20 | 2022-02-16 | Huawei Technologies Co., Ltd. | Scheduling method and apparatus, and related device |
| CN110597609A (en) * | 2019-09-17 | 2019-12-20 | 深圳市及响科技有限公司 | Cluster migration and automatic recovery method and system |
| CN111158909A (en) * | 2019-12-27 | 2020-05-15 | 中国联合网络通信集团有限公司 | Cluster resource allocation processing method, device, equipment and storage medium |
| CN111158909B (en) * | 2019-12-27 | 2023-07-25 | 中国联合网络通信集团有限公司 | Cluster resource allocation processing method, device, equipment and storage medium |
| CN111597024B (en) * | 2020-05-14 | 2022-02-18 | 科东(广州)软件科技有限公司 | Cross-domain cluster processing method and device, electronic equipment and storage medium |
| CN111597024A (en) * | 2020-05-14 | 2020-08-28 | 科东(广州)软件科技有限公司 | Cross-domain cluster processing method and device, electronic equipment and storage medium |
| CN112214299A (en) * | 2020-09-30 | 2021-01-12 | 深圳云天励飞技术股份有限公司 | Multi-core processor and task scheduling method and device thereof |
| CN112269641A (en) * | 2020-11-18 | 2021-01-26 | 网易（杭州）网络有限公司 | Scheduling method, scheduling device, electronic equipment and storage medium |
| CN112269641B (en) * | 2020-11-18 | 2023-09-15 | 网易（杭州）网络有限公司 | Scheduling method, scheduling device, electronic equipment and storage medium |
| CN112925619A (en) * | 2021-02-24 | 2021-06-08 | 深圳依时货拉拉科技有限公司 | Big data real-time computing method and platform |
| CN113391902A (en) * | 2021-06-22 | 2021-09-14 | 未鲲(上海)科技服务有限公司 | Task scheduling method and device and storage medium |
| CN116866438A (en) * | 2023-09-04 | 2023-10-10 | 金网络(北京)数字科技有限公司 | Cross-cluster task scheduling method and device, computer equipment and storage medium |
| CN116866438B (en) * | 2023-09-04 | 2023-11-21 | 金网络(北京)数字科技有限公司 | Cross-cluster task scheduling method and device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107621973B (en) | 2021-10-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107621973A (en) | A kind of method for scheduling task and device across cluster | |
| CN101938416B (en) | Cloud computing resource scheduling method based on dynamic reconfiguration virtual resources | |
| Venkataraman et al. | The power of choice in {Data-Aware} cluster scheduling | |
| CN104657220B (en) | Scheduling model and method based on deadline and expense restriction in mixed cloud | |
| CN103092698B (en) | Cloud computing application automatic deployment system and method | |
| CN103336808B (en) | A kind of real-time diagram data processing system and method based on BSP models | |
| CN103309738B (en) | User job dispatching method and device | |
| US9323580B2 (en) | Optimized resource management for map/reduce computing | |
| CN103716381B (en) | Control method and management node of distributed system | |
| CN109491790A (en) | Industrial Internet of Things edge calculations resource allocation methods and system based on container | |
| CN104301404B (en) | A kind of method and device of the adjustment operation system resource based on virtual machine | |
| CN108510082A (en) | The method and device that machine learning model is handled | |
| CN104199912B (en) | A kind of method and device of task processing | |
| CN103049330B (en) | A kind of trustship type distributed task dispatching method and system | |
| CN101986272A (en) | Task scheduling method under cloud computing environment | |
| CN114138486A (en) | Containerized micro-service arranging method, system and medium for cloud edge heterogeneous environment | |
| Shin et al. | Deadline-guaranteed scheduling algorithm with improved resource utilization for cloud computing | |
| CN102541858A (en) | Data equality processing method, device and system based on mapping and protocol | |
| CN107864211A (en) | Cluster resource dispatching method and system | |
| Kaliaev | Multiagent approach for building distributed adaptive computing system | |
| CN109726004A (en) | A kind of data processing method and device | |
| CN110300188A (en) | Data transmission system, method and apparatus | |
| CN107704322A (en) | Ask distribution method and device | |
| CN115277692A (en) | Automatic operation and maintenance method, device and system for edge network computing terminal equipment | |
| CN104239520B (en) | A kind of HDFS data block Placement Strategies based on historical information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |