CN105868222A - Task scheduling method and device - Google Patents
Task scheduling method and device Download PDFInfo
- Publication number
- CN105868222A CN105868222A CN201510595417.1A CN201510595417A CN105868222A CN 105868222 A CN105868222 A CN 105868222A CN 201510595417 A CN201510595417 A CN 201510595417A CN 105868222 A CN105868222 A CN 105868222A
- Authority
- CN
- China
- Prior art keywords
- monitoring data
- cluster
- historical record
- record
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Software Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the invention provides a task scheduling method and device. A scheduling system obtains latest monitoring data which is acquired; the latest monitoring data which is acquired is input into a decision tree prediction model, and the decision tree prediction model outputs the busy degree of a cluster at a current moment; and the scheduling system schedules tasks according to the busy degree of the cluster. When the technical scheme of the invention is adopted, the scheduling system analyzes the monitoring data which can reflect the state information of a management node at the current moment, and determines the busy degree of the cluster at the current moment according to the decision tree prediction model so as to carry out task scheduling according to the busy degree of the cluster, the task scheduling only can be carried out under a situation that resources which can be distributed in the cluster are sufficient, data processing efficiency is improved, and system performance is effectively improved.
Description
Technical field
The present embodiments relate to computer application field, particularly relate to a kind of method for scheduling task and device.
Background technology
Hadoop is a software frame that mass data can carry out distributed treatment, and it can be with one
Plant mode reliable, efficient, telescopic and carry out data process.Hadoop is elementary composition by several, as
MapReduce、HDFS(Hadoop Distributed File System；Hadoop distributed file system)
Deng, wherein, MapReduce is the core element of Hadoop, and this MapReduce is used for large-scale data
The concurrent operation of collection (1TB can be more than)；MapReduce engine comprise management node (jobtracker) and
Task node (tasktracker) two category node, wherein, jobtracker is host node, is used for managing all
MapReduce calculates the operation of operation, and tasktracker is from node.
In Hadoop1.0 application scenarios, the resource of each tasktracker node is divided by Hadoop1.0
For multiple slot；For any one MapReduce operation, the data of this any one MapReduce operation
Processor active task (hereinafter referred to as task) can be assigned in node on the slot of configuration perform, all described joints
Point one cluster of composition.At present, when the task in Hadoop is scheduling by dispatching patcher, be only capable of by
Task is together in series, and makes all tasks perform in a certain order, can not carry out task run environment
Analyze, thus cannot know the assignable resource of current time (such as untapped slot number) whether with treat
The resource processing required by task to be taken matches so that in current time cluster during allowable resource deficiency,
Dispatching patcher still can carry out task distribution, now, will appear from data operation task run speed slow, at data
Manage inefficient problem.
In sum, at present in dispatching patcher carries out task assignment procedure, there is task unreasonable distribution,
The problem that data-handling efficiency is low.
Summary of the invention
The embodiment of the present invention provides a kind of method for scheduling task and device, in order to solve to enter in dispatching patcher at present
In the assigning process of row task, there is task unreasonable distribution, the problem that data-handling efficiency is low.
The concrete technical scheme that the embodiment of the present invention provides is as follows:
The embodiment of the present invention provides a kind of method for scheduling task, and being applied to dispatching patcher is that cluster carries out task
During scheduling, including:
Real-time Collection monitoring data；Wherein, described monitoring data are used for characterizing management joint described in current time
The status information of point；
In the decision tree forecast model that the up-to-date monitoring data collected input is pre-build, obtain described
The cluster duty tag along sort of decision tree forecast model output；Wherein, described cluster duty is divided
Class label is for characterizing the busy extent of current time cluster；
According to the cluster duty tag along sort that described monitoring data are corresponding, waiting task is adjusted
Degree.
The embodiment of the present invention provides a kind of task scheduling apparatus, and being applied to dispatching patcher is that cluster carries out task
During scheduling, including:
Monitoring data capture unit, monitors data for Real-time Collection；Wherein, described monitoring data are used for table
Levy the status information managing node described in current time；
Tag along sort acquiring unit, for inputting the decision tree pre-build by the up-to-date monitoring data collected
In forecast model, obtain the cluster duty tag along sort of described decision tree forecast model output；Its
In, described cluster duty tag along sort is for characterizing the busy extent of current time cluster；
Scheduling unit, for the cluster duty tag along sort corresponding according to described monitoring data, treats place
Reason task is scheduling.
The method for scheduling task of embodiment of the present invention offer and device, by dispatching patcher acquisition monitoring data；
And in the up-to-date monitoring data input decision tree forecast model that will collect, this decision tree forecast model export
The busy extent of current time cluster；Task, according to cluster busy extent, is scheduling by dispatching patcher.Adopt
With technical solution of the present invention, the dispatching patcher monitoring to current time jobtracker status information can be reflected
Data are analyzed, and according to decision tree forecast model, determine the busy extent of current time cluster, thus
Task scheduling can be carried out, the most in the cluster in the case of allowable resource abundance according to the busy extent of cluster
Carry out task scheduling, improve data-handling efficiency, be effectively increased systematic function.
Accompanying drawing explanation
Fig. 1 is dispatching patcher structural representation in the embodiment of the present invention；
Fig. 2 is that in the embodiment of the present invention, dispatching patcher is scheduling flow chart to task；
Fig. 3 is the status information table managing node in the embodiment of the present invention；
Fig. 4 is the monitoring data that in the embodiment of the present invention, a certain moment collects；
Fig. 5 is the status information ratio index table managing node in the embodiment of the present invention；
Fig. 6 is decision tree forecast model schematic diagram in the embodiment of the present invention；
Fig. 7 is the function file call relation schematic diagram in the embodiment of the present invention in lisp file；
Fig. 8 is task scheduling apparatus structural representation in the embodiment of the present invention.
Detailed description of the invention
For making the purpose of the embodiment of the present invention, technical scheme and advantage clearer, below in conjunction with the present invention
Accompanying drawing in embodiment, is clearly and completely described the technical scheme in the embodiment of the present invention, it is clear that
Described embodiment is a part of embodiment of the present invention rather than whole embodiments.Based in the present invention
Embodiment, those of ordinary skill in the art obtained under not making creative work premise all its
His embodiment, broadly falls into the scope of protection of the invention.
Refering to shown in Fig. 1, for dispatching patcher configuration diagram in the embodiment of the present invention, this dispatching patcher includes
Acquisition server, for acquisition monitoring data；Storage server, is used for storing monitoring data and historical record；
Dispatch server, is used for using monitoring data to set up decision tree forecast model, and predicts mould according to decision tree
The result of type output determines the need for carrying out task scheduling.
Below in conjunction with Figure of description, the embodiment of the present invention is described in further detail.
Refering to shown in Fig. 2, in the embodiment of the present invention, the process that task is scheduling by dispatching patcher, including:
Step 200: dispatching patcher Real-time Collection monitoring data；Wherein, these monitoring data are used for characterizing currently
The status information of moment jobtracker.
In the embodiment of the present invention, acquisition server is according to predetermined period acquisition monitoring data, this predetermined period
Can pre-set according to concrete application scenarios；The monitoring data collected are passed through to gather by acquisition server
Rest API (Application Programming Interface between server and storage server；Application
Program becomes interface) store to storage server.
Concrete, above-mentioned monitoring data comprise the status information of current time jobtracker, wherein, is somebody's turn to do
Status information includes at least Map task capacity (Map Task Capacity), Reduce task capacity
(Reduce Task Capacity), online Map Slot number (Occupied Map Slots), online
Reduce Slot number (Occupied Reduce Slots), interstitial content (Nodes), blacklist list
The node (Blacklisted/Graylisted Nodes) comprised in the list of/gray list, is in the task of run mode
Number (Running Jobs), additionally, this status information can also comprise MapReduce engine status
(State), whether MapReduce engine is safe mode (Safe Mode), show refering to Fig. 3
Status information table in the embodiment of the present invention.
Optionally, acquisition server can be by newlisp monitoring script according to predetermined period acquisition monitoring number
According to.Such as, refering to shown in Fig. 4, the monitoring data collected for the moment a certain in the embodiment of the present invention.
Owing in task in the process of implementation, dispatch server will generate the history note corresponding with this task
Record, wherein, this historical record is used for characterizing some moment task run situation, and this operation conditions is at least
Including running duration；Therefore, further, the historical record of generation is taken by dispatch server by scheduling
Historical record is stored to storage server by the Rest api interface between business device and storage server.
Step 210: the above-mentioned up-to-date monitoring data collected are inputted the decision tree forecast model pre-build
In, obtain the cluster duty tag along sort of this decision tree forecast model output；Wherein, this cluster work
Make state classification label for characterizing the busy extent of current time cluster.
In the embodiment of the present invention, the up-to-date monitoring data collected are inputted the decision-making pre-build by dispatching patcher
In tree forecast model, wherein, this model is for being analyzed monitoring data, to obtain this monitoring data pair
The cluster duty tag along sort answered；Dispatching patcher obtains the cluster work of this decision tree forecast model output
State classification label.
In above process, owing to, in dispatching patcher, task performs according to predetermined period, therefore, adjust
Degree system can be chosen and specify number a task, specifies number the history run of a task according to this,
Set up the decision tree prediction mould between cluster duty tag along sort and the historical record that monitoring data are corresponding
Type.
Concrete, dispatching patcher sets up the process of this decision tree forecast model, including: dispatching patcher is chosen
Specify number an appointed task；And determine preset time period；Dispatching patcher, from historical record, is chosen
State the historical record in the preset time period that each appointed task is corresponding, as sample history, example
As, dispatching patcher chooses 20 appointed tasks, and the preset time period chosen is for from 1 day March in 2015
60 days, then to choose these 20 appointed tasks corresponding from 1 day March in 2015 respectively for dispatching patcher
Historical record in 60 days, using the historical record chosen as sample history；The sample obtained will be chosen
This historical record is divided into exception history record and normal historical record；According to this exception history record and
Normal historical record, determines the cluster duty tag along sort that corresponding sample monitoring data are corresponding.
Wherein, described cluster duty tag along sort can only comprise idle and busy two kinds, described collection
Group's duty tag along sort can also be busy extent grade；According to the collection that this sample monitoring data are corresponding
The jobtracker status information comprised in group's duty tag along sort, and sample monitoring data, calculates
Prediction Parameters；According to calculated Prediction Parameters, set up decision tree forecast model.
Optionally, dispatching patcher can be connect by the Rest API between dispatch server and storage server
Mouth obtains the historical record preserved in storage server.
Optionally, historical record is divided into exception history record and the process of normal historical record, tool
Body includes: owing to, in dispatching patcher, task is to perform according to predetermined period, and the fortune of each task
Row duration is relatively stable, therefore, and the fortune recorded according to each the historical record that each task is corresponding
Row duration, can calculate appointment time period this task interior averagely runs duration；According to calculated flat
All run duration, and the operation duration parameters preset, calculate and run duration threshold value, run as average
Shi Changwei a, operation duration parameters is b, runs duration threshold value and is (a × b), wherein, this operation
The value that according to duration parameters, concrete application scenarios pre-sets, as this operation duration parameters can be chosen for
30%；If growing up when the historical record got exists the operation that any one historgraphic data recording recorded
In running duration threshold value, then this any one historical data is defined as exception history record；Otherwise, will
This any one historical record is defined as normal historical record.
Optionally, dispatching patcher, according to exception history record and normal historical record, determines that sample monitors number
According to the process of corresponding cluster duty tag along sort, specifically include: remember for any one exception history
Record, performs following operation, obtains the operation time period of this any one exception history record；Monitor from sample
In data, choose the sample run in the time period monitoring data that this any one exception history record is corresponding,
Cluster duty tag along sort corresponding for the sample chosen monitoring data is defined as busy；For arbitrarily
One normal historical record, performs following operation, obtains the operation time of this any one normal historical record
Section；From sample monitoring data, choose corresponding the running in the time period of this any one normal historical record
Sample monitoring data, are defined as cluster duty tag along sort corresponding for the sample chosen monitoring data
Idle.
Further, dispatching patcher determines the cluster duty classification that corresponding sample monitoring data are corresponding
After label, sample can also be monitored in data by dispatching patcher, and cluster duty tag along sort is empty
Not busy sample monitoring data are bisected into the first Idle state sample monitoring data and the second Idle state sample monitoring number
According to；Being monitored by sample in data, the sample monitoring data that cluster duty tag along sort is busy are divided equally
It is first busy aspect this monitoring data and second busy aspect this monitoring data, and by the first Idle state
Sample monitoring data and first busy aspect this monitoring data are combined, as training sample set, and will
Second Idle state sample monitoring data and second busy aspect this monitoring data are combined, as test specimens
This set.Such as, the total number of sample monitoring data is 1440, wherein, and busy aspect this monitoring number
According to number be 134, Idle state sample monitoring data number be 1266；From busy this prison of aspect
Control data in randomly select 67 as first busy aspect this monitoring data, by busy aspect this monitoring number
According to, remaining 67 busy aspect this monitoring data are as second busy aspect this monitoring data, in like manner,
From Idle state sample monitoring data, randomly select 633 monitor data as the first Idle state sample, by sky
In not busy aspect this monitoring data, remaining 633 Idle state samples monitoring data are supervised as the second Idle state sample
Control data；Dispatching patcher the just first Idle state sample monitoring data and first busy aspect this monitoring data
It is combined, as training sample set, by the second Idle state sample monitoring data and the second busy aspect
This monitoring data are combined, as test sample set.
Optionally, test sample set is trained, generates decision tree forecast model.
Using technique scheme, sample is monitored data and is divided into test sample set and training by dispatching patcher
Sample set, and test sample set and training sample set all comprise busy aspect this monitoring data and
Idle state sample monitoring data, dispatching patcher is according to the sample prison monitor database comprised in training sample set
Get decision tree forecast model, and use the sample comprised in test sample set monitoring data to decision-making
Tree forecast model is tested, and further ensure that the accuracy of the decision-tree model of foundation.
Optionally, according to the cluster duty tag along sort that this sample monitoring data are corresponding, and sample
The jobtracker status information comprised in monitoring data, uses C4.5 algorithm to calculate Prediction Parameters, calculates pre-
Survey parameter.Wherein, due to the Map Task Capacity, Reduce that comprise in jobtracker status information
Task Capacity, Occupied Map Slots, Occupied Reduce Slots, Nodes,
The cluster scale of Blacklisted/Graylisted Nodes with Hadoop is relevant, if at a time adding
The node comprised in Hadoop, the most above-mentioned jobtracker status information will change, the decision tree of foundation
The problem that forecast model is little by there is range of applicability；In order to make the decision tree forecast model of foundation be applicable to
The Hadoop of various cluster scales, can change jobtracker status information, be converted to ratio value
State (hereinafter referred to as status information ratio index), the jobtracker status information ratio obtained after conversion
Index is refering to shown in Fig. 5.
Based on above-mentioned training result, can obtain, in jobtracker status information ratio index, only comprising
Reduce Slot occupancy, map Slots occupancy, the online rate of node, jobtracker memory usage four
Item is relevant to the busy extent of monitoring data；Based on this, the calculated Prediction Parameters of dispatching patcher is
The parameter of above-mentioned four jobtracker status information ratio indexs.Based on Prediction Parameters obtained above, build
Vertical decision tree forecast model as shown in Figure 6.
Step 220: according to the cluster duty tag along sort that above-mentioned monitoring data are corresponding, to pending
Business is scheduling.
In the embodiment of the present invention, when monitoring cluster duty tag along sort corresponding to data and being busy,
Waiting task is not scheduling by current time dispatching patcher；When the cluster work shape that monitoring data are corresponding
When state tag along sort is idle, current time dispatching patcher scheduling waiting task.
Optionally, dispatching patcher also comprises some general lisp files, is used for providing basic function, its bag
Containing Timer, Result, DecisionTree etc.；Refering to shown in Fig. 7, for lisp literary composition in the embodiment of the present invention
Function file call relation schematic diagram in part, wherein, Timer file provides Time form transformation merit
Can, Result provides the analytical capabilities that basis storage services module returns result, and DecisionTree carries
Supply the function that a decision tree based on C4.5 algorithm is trained, shows, preserves, classifies.
Based on technique scheme, refering to shown in Fig. 8, the embodiment of the present invention provides a kind of task scheduling apparatus,
Single including monitoring data capture unit 80, cluster duty tag along sort acquiring unit 81, and scheduling
Unit 82, wherein:
Monitoring data capture unit 80, monitors data for Real-time Collection；Wherein, described monitoring data are used for
Characterize the status information managing node described in current time；
Tag along sort acquiring unit 81, for inputting the decision-making pre-build by the up-to-date monitoring data collected
In tree forecast model, obtain the cluster duty tag along sort of described decision tree forecast model output；Its
In, described cluster duty tag along sort is for characterizing the busy extent of current time cluster；
Scheduling unit 82, for the cluster duty tag along sort corresponding according to described monitoring data, treats
Process task is scheduling.
Further, described device also includes that decision tree forecast model sets up unit 83, is used for: obtains and presets
Number task；And from historical record, choose the Preset Time that each obtaining of task is corresponding respectively
Historical record in Duan；Wherein, the record generated during described historical record is described tasks carrying；
The described historical record chosen is divided into exception history record and normal historical record；According to described different
Often historical record and described normal historical record, determines the sample monitoring data pair that described historical record is corresponding
The cluster duty tag along sort answered；Divide according to the cluster duty that described sample monitoring data are corresponding
The status information of the management node comprised in class label, and described sample monitoring data, calculates prediction ginseng
Number；According to described Prediction Parameters, set up decision tree forecast model.
Optionally, described decision tree forecast model is set up unit 83 and is divided into by the described historical record chosen
Exception history record and normal historical record, specifically include: for the task of any one acquisition, perform
Following operation: obtain corresponding each of the task of described any one acquisition respectively and choose the history note obtained
The operation duration that record is recorded；The operation duration that the historical record obtained is recorded is chosen according to each,
Calculate described any one acquisition task averagely run duration；During according to calculated average operation
Long, and the operation duration parameters preset, calculate and run duration threshold value；If choosing the history note obtained
Record exists operation duration that any one historical record recorded more than described operation duration threshold value, then will
Described any one historical record is defined as exception history record；Otherwise, by true for described any one historical record
It is set to normal historical record.
Optionally, described decision tree forecast model sets up unit 83 according to described exception history record and described
Normal historical record, determines the cluster duty tag along sort that described sample monitoring data are corresponding, specifically
Including: for any one exception history record, perform following operation: obtain described any one exception history
The operation time period recorded；From sample monitoring data, choose described any one exception history note
The sample run in the time period monitoring data that record is recorded, by corresponding for the described sample monitoring data chosen
Cluster duty tag along sort be defined as busy；For any one normal historical record, perform as follows
Operation: obtain the operation time period that described any one normal historical record is recorded；Data are monitored from sample
In, choose the sample run in the time period monitoring data that described any one normal historical record is recorded,
Cluster duty tag along sort corresponding for the described sample chosen monitoring data is defined as the free time.
Optionally, described scheduling unit 82, specifically for: when the cluster work that described monitoring data are corresponding
When state classification label is busy, described waiting task is not scheduling by current time；When described prison
When control cluster duty tag along sort corresponding to data be idle, described pending of current time scheduling
Business.
In sum, in the embodiment of the present invention, dispatching patcher obtains the up-to-date monitoring data collected；This prison
Control data are used for characterizing current time jobtracker status information；By the above-mentioned up-to-date monitoring data collected
In the decision tree forecast model that input pre-builds, obtain the cluster state of this decision tree forecast model output
Tag along sort；Wherein, this tag along sort is for characterizing the busy extent of dispatching patcher described in current time；
According to the cluster state tag along sort that above-mentioned monitoring data are corresponding, waiting task is scheduling.Use
Technical solution of the present invention, dispatching patcher is according to the monitoring that can reflect current time jobtracker status information
Data, and decision tree forecast model, determine the busy extent of current time cluster, due to jobtracker
It is managed, therefore, by jobtracker status information for MapReduce being calculated the operation of operation
I.e. it is capable of the analysis to MapReduce engine such that it is able to determine the busy extent of cluster, and then
Only in the case of dispatching patcher allowable resource abundance, carry out task scheduling, improve data-handling efficiency,
It is effectively increased systematic function.
Device embodiment described above is only schematically, wherein said illustrates as separating component
Unit can be or may not be physically separate, and the parts shown as unit can be or also
Can not be physical location, i.e. may be located at a place, or can also be distributed on multiple NE.
Some or all of module therein can be selected according to the actual needs to realize the mesh of the present embodiment scheme
's.Those of ordinary skill in the art, in the case of not paying performing creative labour, are i.e. appreciated that and implement.
Through the above description of the embodiments, those skilled in the art is it can be understood that arrive each enforcement
Mode can add the mode of required general hardware platform by software and realize, naturally it is also possible to pass through hardware.
Based on such understanding, the part that prior art is contributed by technique scheme the most in other words is permissible
Embodying with the form of software product, this computer software product can be stored in computer-readable storage medium
In matter, such as ROM/RAM, magnetic disc, CD etc., including some instructions with so that a computer equipment
(can be personal computer, server, or the network equipment etc.) performs each embodiment or embodiment
The method described in some part.
Last it is noted that above example is only in order to illustrate the technical scheme of the embodiment of the present invention, rather than
It is limited；Although the embodiment of the present invention being described in detail with reference to previous embodiment, this area
Those of ordinary skill is it is understood that the technical scheme described in foregoing embodiments still can be repaiied by it
Change, or wherein portion of techniques feature is carried out equivalent；And these amendments or replacement, do not make phase
The essence answering technical scheme departs from the spirit and scope of the embodiment of the present invention each embodiment technical scheme.
Claims (10)
1. a method for scheduling task, being applied to dispatching patcher is during cluster carries out task scheduling,
It is characterized in that, including:
Real-time Collection monitoring data；Wherein, described monitoring data are used for characterizing management joint described in current time
The status information of point；
In the decision tree forecast model that the up-to-date monitoring data collected input is pre-build, obtain described
The cluster duty tag along sort of decision tree forecast model output；Wherein, described cluster duty is divided
Class label is for characterizing the busy extent of current time cluster；
According to the cluster duty tag along sort that described monitoring data are corresponding, waiting task is adjusted
Degree.
Method the most according to claim 1, it is characterised in that set up described decision tree prediction mould
Type, specifically includes:
Obtain preset number task；And
From historical record, choose the history in the preset time period that each obtaining of task is corresponding respectively
Record；Wherein, the record generated during described historical record is described tasks carrying；
The described historical record chosen is divided into exception history record and normal historical record；
According to described exception history record and described normal historical record, determine that described historical record is corresponding
The cluster duty tag along sort that sample monitoring data are corresponding；
According to the cluster duty tag along sort that described sample monitoring data are corresponding, and described sample prison
The status information of the management node comprised in control data, calculates Prediction Parameters；
According to described Prediction Parameters, set up decision tree forecast model.
Method the most according to claim 2, it is characterised in that the described historical record chosen is drawn
It is divided into exception history record and normal historical record, specifically includes:
For the task of any one acquisition, perform to operate as follows:
Obtain corresponding each of the task of described any one acquisition respectively to choose the historical record obtained and remembered
The operation duration of record；
Choose, according to each, the operation duration that the historical record obtained is recorded, calculate described any one obtain
Taking of task averagely run duration；
According to calculated average operation duration, and the operation duration parameters preset, when calculating operation
Long threshold value；
The historical record obtained exists operation duration that any one historical record recorded more than institute if choosing
State operation duration threshold value, then described any one historical record is defined as exception history record；Otherwise,
Described any one historical record is defined as normal historical record.
Method the most according to claim 2, it is characterised in that according to described exception history record and
Described normal historical record, determines the cluster duty tag along sort that described sample monitoring data are corresponding,
Specifically include:
For any one exception history record, perform following operation: obtain described any one exception history note
The operation time period that record is recorded；From sample monitoring data, choose described any one exception history record
The sample run in the time period monitoring data recorded, by corresponding for the described sample monitoring data chosen
Cluster duty tag along sort is defined as busy；
For any one normal historical record, perform following operation: obtain described any one normal history note
The operation time period that record is recorded；From sample monitoring data, choose described any one normal historical record
The sample run in the time period monitoring data recorded, by corresponding for the described sample monitoring data chosen
Cluster duty tag along sort is defined as the free time.
5. according to the method described in any one of claim 1-4, it is characterised in that according to described monitoring number
According to corresponding cluster duty tag along sort, waiting task is scheduling, specifically includes:
When the cluster duty tag along sort that described monitoring data are corresponding is busy, current time is the most right
Described waiting task is scheduling；
When the cluster duty tag along sort that described monitoring data are corresponding is idle, current time is dispatched
Described waiting task.
6. a task scheduling apparatus, being applied to dispatching patcher is during cluster carries out task scheduling,
It is characterized in that, including:
Monitoring data capture unit, monitors data for Real-time Collection；Wherein, described monitoring data are used for table
Levy the status information managing node described in current time；
Tag along sort acquiring unit, for inputting the decision tree pre-build by the up-to-date monitoring data collected
In forecast model, obtain the cluster duty tag along sort of described decision tree forecast model output；Its
In, described cluster duty tag along sort is for characterizing the busy extent of current time cluster；
Scheduling unit, for the cluster duty tag along sort corresponding according to described monitoring data, treats place
Reason task is scheduling.
Device the most according to claim 6, it is characterised in that also include that decision tree forecast model is built
Vertical unit, is used for:
Obtain preset number task；And from historical record, choose each task pair obtained respectively
Historical record in the preset time period answered；Wherein, during described historical record is described tasks carrying
The record generated；
The described historical record chosen is divided into exception history record and normal historical record；
According to described exception history record and described normal historical record, determine that described historical record is corresponding
The cluster duty tag along sort that sample monitoring data are corresponding；
According to the cluster duty tag along sort that described sample monitoring data are corresponding, and described sample prison
The status information of the management node comprised in control data, calculates Prediction Parameters；
According to described Prediction Parameters, set up decision tree forecast model.
Device the most according to claim 7, it is characterised in that described decision tree forecast model is set up
The described historical record chosen is divided into exception history record and normal historical record by unit, specifically wraps
Include:
For the task of any one acquisition, perform to operate as follows:
Obtain corresponding each of the task of described any one acquisition respectively to choose the historical record obtained and remembered
The operation duration of record；
Choose, according to each, the operation duration that the historical record obtained is recorded, calculate described any one obtain
Taking of task averagely run duration；
According to calculated average operation duration, and the operation duration parameters preset, when calculating operation
Long threshold value；
The historical record obtained exists operation duration that any one historical record recorded more than institute if choosing
State operation duration threshold value, then described any one historical record is defined as exception history record；Otherwise,
Described any one historical record is defined as normal historical record.
Device the most according to claim 7, it is characterised in that described decision tree forecast model is set up
Unit, according to described exception history record and described normal historical record, determines that described sample monitors data pair
The cluster duty tag along sort answered, specifically includes:
For any one exception history record, perform following operation: obtain described any one exception history note
The operation time period that record is recorded；From sample monitoring data, choose described any one exception history record
The sample run in the time period monitoring data recorded, by corresponding for the described sample monitoring data chosen
Cluster duty tag along sort is defined as busy；
For any one normal historical record, perform following operation: obtain described any one normal history note
The operation time period that record is recorded；From sample monitoring data, choose described any one normal historical record
The sample run in the time period monitoring data recorded, by corresponding for the described sample monitoring data chosen
Cluster duty tag along sort is defined as the free time.
10. according to the device described in any one of claim 6-9, it is characterised in that described scheduling unit,
Specifically for:
When the cluster duty tag along sort that described monitoring data are corresponding is busy, current time is the most right
Described waiting task is scheduling；
When the cluster duty tag along sort that described monitoring data are corresponding is idle, current time is dispatched
Described waiting task.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510595417.1A CN105868222A (en) | 2015-09-17 | 2015-09-17 | Task scheduling method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510595417.1A CN105868222A (en) | 2015-09-17 | 2015-09-17 | Task scheduling method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105868222A true CN105868222A (en) | 2016-08-17 |
Family
ID=56624271
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510595417.1A Pending CN105868222A (en) | 2015-09-17 | 2015-09-17 | Task scheduling method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105868222A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108280780A (en) * | 2018-01-18 | 2018-07-13 | 深圳市麦斯杰网络有限公司 | The optimization method and device of property quality product control |
| CN109815154A (en) * | 2019-02-22 | 2019-05-28 | 深圳市网心科技有限公司 | A kind of test method, device, system and medium |
| CN109960573A (en) * | 2018-12-29 | 2019-07-02 | 天津南大通用数据技术股份有限公司 | A kind of cross-domain calculating task dispatching method and system based on Intellisense |
| WO2019128598A1 (en) * | 2017-12-29 | 2019-07-04 | Oppo广东移动通信有限公司 | Application processing method, electronic device, and computer readable storage medium |
| CN110413397A (en) * | 2019-07-31 | 2019-11-05 | 北京智行者科技有限公司 | A kind of method for scheduling task towards automatic Pilot |
| CN111381970A (en) * | 2020-03-16 | 2020-07-07 | 第四范式（北京）技术有限公司 | Cluster task resource allocation method and device, computer device and storage medium |
| CN112019581A (en) * | 2019-05-30 | 2020-12-01 | 华为技术有限公司 | Method and device for scheduling task processing entities |
| CN113760496A (en) * | 2020-12-29 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Container scheduling method and scheduler |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPS62290958A (en) * | 1986-06-11 | 1987-12-17 | Hitachi Ltd | Task schedule system |
| CN101075304A (en) * | 2006-05-18 | 2007-11-21 | 河北全通通信有限公司 | Method for constructing decision supporting system of telecommunication industry based on database |
| CN104102673A (en) * | 2013-04-12 | 2014-10-15 | 腾讯科技（深圳）有限公司 | Webpage state monitoring method and device |
| CN104583789A (en) * | 2012-04-11 | 2015-04-29 | 爱德万测试公司 | Creation and scheduling of a decision and execution tree of a test cell controller |
-
2015
- 2015-09-17 CN CN201510595417.1A patent/CN105868222A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPS62290958A (en) * | 1986-06-11 | 1987-12-17 | Hitachi Ltd | Task schedule system |
| CN101075304A (en) * | 2006-05-18 | 2007-11-21 | 河北全通通信有限公司 | Method for constructing decision supporting system of telecommunication industry based on database |
| CN104583789A (en) * | 2012-04-11 | 2015-04-29 | 爱德万测试公司 | Creation and scheduling of a decision and execution tree of a test cell controller |
| CN104102673A (en) * | 2013-04-12 | 2014-10-15 | 腾讯科技（深圳）有限公司 | Webpage state monitoring method and device |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019128598A1 (en) * | 2017-12-29 | 2019-07-04 | Oppo广东移动通信有限公司 | Application processing method, electronic device, and computer readable storage medium |
| CN108280780A (en) * | 2018-01-18 | 2018-07-13 | 深圳市麦斯杰网络有限公司 | The optimization method and device of property quality product control |
| CN109960573A (en) * | 2018-12-29 | 2019-07-02 | 天津南大通用数据技术股份有限公司 | A kind of cross-domain calculating task dispatching method and system based on Intellisense |
| CN109960573B (en) * | 2018-12-29 | 2021-01-08 | 天津南大通用数据技术股份有限公司 | Cross-domain computing task scheduling method and system based on intelligent perception |
| CN109815154A (en) * | 2019-02-22 | 2019-05-28 | 深圳市网心科技有限公司 | A kind of test method, device, system and medium |
| CN112019581A (en) * | 2019-05-30 | 2020-12-01 | 华为技术有限公司 | Method and device for scheduling task processing entities |
| CN112019581B (en) * | 2019-05-30 | 2022-02-25 | 华为技术有限公司 | Method and device for scheduling task processing entities |
| CN110413397A (en) * | 2019-07-31 | 2019-11-05 | 北京智行者科技有限公司 | A kind of method for scheduling task towards automatic Pilot |
| CN110413397B (en) * | 2019-07-31 | 2021-09-10 | 北京智行者科技有限公司 | Task scheduling method for automatic driving |
| CN111381970A (en) * | 2020-03-16 | 2020-07-07 | 第四范式（北京）技术有限公司 | Cluster task resource allocation method and device, computer device and storage medium |
| CN111381970B (en) * | 2020-03-16 | 2023-07-25 | 第四范式（北京）技术有限公司 | Cluster task resource allocation method and device, computer device and storage medium |
| CN113760496A (en) * | 2020-12-29 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Container scheduling method and scheduler |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105868222A (en) | Task scheduling method and device | |
| CN104317658B (en) | A kind of loaded self-adaptive method for scheduling task based on MapReduce | |
| CN110389820B (en) | Private cloud task scheduling method for resource prediction based on v-TGRU model | |
| Legato et al. | Simulation-based optimization for discharge/loading operations at a maritime container terminal | |
| CN109992404A (en) | PC cluster resource regulating method, device, equipment and medium | |
| CN105978960A (en) | Cloud scheduling system and method based on mass video structured processing | |
| CN106547882A (en) | A kind of real-time processing method and system of big data of marketing in intelligent grid | |
| Chan* et al. | Fuzzy goal-programming model with an artificial immune system (AIS) approach for a machine tool selection and operation allocation problem in a flexible manufacturing system | |
| CN112685153A (en) | Micro-service scheduling method and device and electronic equipment | |
| CN103177035A (en) | Data query device and data query method in data base | |
| CN114819213A (en) | Intelligent operation and maintenance management method and system for electric automobile public charging facility | |
| CN113822585A (en) | Intelligent smelting factory informatization management system | |
| Roshani et al. | A simulated annealing approach for multi-manned assembly line balancing problem type II | |
| CN116109139A (en) | Wind control strategy generation method, decision method, server and storage medium | |
| Zahmani et al. | A data mining based dispatching rules selection system for the job shop scheduling problem | |
| Fontes et al. | On multi-objective evolutionary algorithms | |
| CN111339052A (en) | Unstructured log data processing method and device | |
| CN117435306A (en) | Cluster container expansion and contraction method, device, equipment and storage medium | |
| CN107871055A (en) | A kind of data analysing method and device | |
| CN117076077A (en) | Planning and scheduling optimization method based on big data analysis | |
| CN110188258B (en) | Method and device for acquiring external data by using crawler | |
| CN110413682A (en) | A kind of the classification methods of exhibiting and system of data | |
| Shen et al. | Blocking flow shop scheduling based on hybrid ant colony optimization | |
| CN109766181A (en) | A kind of RMS schedulability determination method and device based on deep learning | |
| EP3855316A1 (en) | Optimizing breakeven points for enhancing system performance |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20160817 |