CN103049330B - A kind of trustship type distributed task dispatching method and system - Google Patents
A kind of trustship type distributed task dispatching method and system Download PDFInfo
- Publication number
- CN103049330B CN103049330B CN201210517389.8A CN201210517389A CN103049330B CN 103049330 B CN103049330 B CN 103049330B CN 201210517389 A CN201210517389 A CN 201210517389A CN 103049330 B CN103049330 B CN 103049330B
- Authority
- CN
- China
- Prior art keywords
- work
- task
- workstation
- destination file
- computing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 68
- 230000008569 process Effects 0.000 claims abstract description 56
- 238000013528 artificial neural network Methods 0.000 claims abstract description 26
- 238000012545 processing Methods 0.000 claims description 18
- 230000000694 effects Effects 0.000 claims description 16
- 238000012795 verification Methods 0.000 claims description 11
- 238000009826 distribution Methods 0.000 claims description 6
- 230000005540 biological transmission Effects 0.000 claims description 5
- 230000000977 initiatory effect Effects 0.000 claims description 4
- 238000007689 inspection Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000000151 deposition Methods 0.000 claims description 3
- 238000011156 evaluation Methods 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 claims description 2
- 238000010327 methods by industry Methods 0.000 claims description 2
- 230000006870 function Effects 0.000 description 22
- 238000005457 optimization Methods 0.000 description 7
- 238000012217 deletion Methods 0.000 description 3
- 230000037430 deletion Effects 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 238000012805 post-processing Methods 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 238000011084 recovery Methods 0.000 description 2
- 230000007115 recruitment Effects 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 230000003750 conditioning effect Effects 0.000 description 1
- 238000005314 correlation function Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000008571 general function Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a kind of distributed task dispatching method of trustship type, there are following steps: a certain decomposable Task-decomposing is become multiple work transferring to workstation to complete by S100.; S200. described multiple work are processed according to the pre-multiple workstations all giving the computing of participation task at random of quantity; S300. carry out at described workstation in the process of work disposal, estimated by the calculated capacity of artificial neural network to pending work, by work large for calculated capacity, priority allocation is to the strong workstation of computing power; S400., after any operative station completes the work described in one in the workstation of described participation computing, result treatment is carried out to this work; S500. after described multiple workstations complete all work, gather whole working results, generate task result.
Description
Technical field
The present invention relates to task scheduling and policy techniques field, particularly relate to a kind of trustship type, unattended distributed task dispatching system and method, relate to Patent classificating number G06 and calculate; Calculate; Counting G06F electricity Digital data processing G06F9/00 presetting apparatus, such as, controller G06F9/06 apply stored in program, namely the storage inside applying treatment facility is carried out reception program and keeps the G06F9/46 multiprogramming device G06F9/50 Resourse Distribute of program, such as, CPU (central processing unit).
Background technology
Computer applied algorithm to be actually by computing machine to perform a series of work, such as, copy a file, start a process, close a window etc.Along with computer applied algorithm becomes increasingly complex, the calculated amount required for it is also more and more surprising, if transfer to a computing machine to complete so complicated program, needs to consume a large amount of time, loses more than gain.Relatively more current solution is, use distributed task scheduling system, the work that the subtask of a complexity and huge computer applied algorithm being resolved into multiple suitable size can be able to be completed within the suitable time by a computing machine/workstation in other words.Then, dispatching system is by these work allocations to the workstation of the some in network, and the collaborative work of use multiple stage workstation solves a huge computer applied algorithm, and efficiency has had very large change.
But existing distributed task scheduling distribution/dispatching system, more, lays particular emphasis on and how more effectively divides work and share out the work.And when workstation process is transferred in the work distributed, dispatching system is then not too concerned about problems such as the treatment situation of each workstation and treatment effeciencies, dispatching system or seldom can not can carry out management and running to the work that each workstation processes, we know the algorithm that divides no matter in advance and share out the work how careful precision, also there will be because the uneven situation of work allocation, and then cause the low of system-computed efficiency.
Summary of the invention
The present invention is directed to the proposition of above problem, and the distributed task dispatching method of a kind of trustship type of development, there are following steps:
S100. a certain decomposable Task-decomposing is become multiple work transferring to workstation to complete;
S200. described multiple work are processed according to the pre-multiple workstations all giving the computing of participation task at random of quantity;
S300. carry out at described workstation in the process of work disposal, estimated by the calculated capacity of artificial neural network to pending work, by work large for calculated capacity, priority allocation is to the strong workstation of computing power;
S400., after any operative station completes the work described in one in the workstation of described participation computing, result treatment is carried out to this work;
S500. after described multiple workstations complete all work, gather whole working results, generate task result.
Described step S300 specifically comprises step:
S310. the hardware parameter of the workstation of all participation tasks process is added up, the ability to work of each workstation described in evaluation;
S320. for completed work, set its calculated capacity=work consuming time × complete the computing power of the workstation of this work;
The parameter of the work S330. completed at first in selecting system and calculated capacity, as the sample of artificial neural network, obtain initial neural network relation function;
S340. using the parameter of work that completes successively in system and calculated capacity as the artificial neural network described in sample substitutes into, constantly improve the relation function of artificial neural network;
S350. the parameter of work to be allocated is substituted in current artificial neural network, estimate the calculated capacity of all work to be allocated; By work large for calculated capacity, distribute to the workstation that in the workstation of participation task process, computing power is strong.
In described step S310, in multiple workstations of setting participation task process, the computing power of the workstation that computing power is the strongest is 100, and the computing power of all the other each workstations is the number of 0-100.
Described step S400 specifically comprises the steps:
S410. result verification: when any operative station completes the computing of a job in all working station participating in computing, the result of this work is at least comprised: whether operation result exists the verification whether met the demands with operation result designated value;
S420. data encasement: if the result of work is by verification, carries out rename destination file, mobile destination file is to specified path and/or extract the operation of data to the file of specified format;
S430. described path of depositing through the destination file of data encasement is recorded;
S440. repeat step S410-430, until last computing that worked of this workstation, generate the destination file that comprises all working handled by this workstation.
Described step S500 specifically comprises:
S510., after a workstation completes distributed whole work, the operation result file of this workstation or the store path of destination file is received;
S520. repeat step S510, until all working station of the work of participation completes the process shared out the work separately, the destination file at all working station is gathered, generate the destination file of task or the address of task result, complete the computing of described distributed task scheduling.
A distributed task dispatching system for trustship type, comprises at least one server and multiple workstation:
Described server receive client's client upload task, Task-decomposing is become multiple work after, then give multiple workstation by work allocation and in the process of workstation process, the distribution of work adjusted in real time; Workstation receives the work by server-assignment, processes, after processing, result is back to server, process of finishing the work;
Described server has: task matching scheduler module and the whole processing module of task;
Described workstation has: work disposal module and working result pretreatment module;
During work:
Task matching scheduler module: receive the task that client uploads, Task-decomposing is become multiple work that can be completed separately process by any one workstation in described multiple workstation; Each workstation is all given at random in advance according to number by decomposing the multiple work obtained;
Workstation startup is to after the process of described work, and the computing power of task matching scheduler module to the workstation of the task of participation process is assessed, for each workstation sets the numerical value that one represents the ability that it calculates;
Described task scheduling modules, according to completed work, estimates the calculated capacity of work to be allocated by artificial neural network;
Described task scheduling modules is by work large for calculated capacity, and priority allocation processes to the workstation that computing power is strong;
Described work of being decomposed by task scheduling modules stores in the server, when described workstation needs to process work, work to be processed need be handed down to workstation by task scheduling modules;
Each workstation receives the work by described server-assignment, carries out calculation process by the processing module in this workstation, generates destination file; After any one in the multiple work be assigned with that this workstation receives has worked and obtain destination file, described working result pretreatment module has at least comprised to this destination file the inspection that whether destination file exists and in destination file, whether particular value meets the demands;
After inspection, working result pretreatment module carries out data encasement, and this data encasement at least comprises: the conversion of the rename of destination file, movement and/or file layout;
After all having worked in this workstation, the whole processing module of the task that the virtual route that the destination file of all working is deposited uploads onto the server by described working result pretreatment module;
When a workstation completes distributed whole work, the whole processing module of described task store this workstation upload described in the part virtual route of result literary composition of finishing the work or download destination file;
When each workstation relating to a process complete task completes all working that it is assigned with, all destination files gather by the whole processing module of described task, generate download path and/or generate task result files; Described download path and/or task result files are sent to the data transmission port of server, complete the computing of distributed task scheduling.
Described server also has task real-time informing module, and this notification module at least uses Email and note form that the progress msg of task process is informed to user.
To the database of user transparent, this database at least comprises:
The task management tables of data of the current Processing tasks of register system and historic task;
At least record the work management tables of data of the numbering of each work become by task division, the corresponding station number of this work, running parameter, duty, the startup of work and deadline and mission number corresponding to this work; And
Record the real-time informing management data list of the task process progress msg that described task real-time informing module transmits to user/supvr, in task process engineering, described task real-time informing module calls the information recorded in this real-time informing management data list, sends to the user specified.
Also have: accept user instruction, assisting users completes and at least comprises: newly-built task, submission task, initiating task and the client of checking latest activity and having finished the work.
Have consistent with described server capability server.
In work disposal process, described task processing module estimates the calculated capacity of each workstation residue work according to the computing power that workstation completes work on hand, according to the remaining displacement volume in different operating station, and the distribution of real-time adjustment work.
Accompanying drawing explanation
In order to the technical scheme of clearer explanation embodiments of the invention or prior art, introduce doing one to the accompanying drawing used required in embodiment or description of the prior art simply below, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
The distributed task dispatching system network topology of Fig. 1 trustship type
Fig. 2 is general function structured flowchart of the present invention
Fig. 3 is overall operation flow process of the present invention
Fig. 4 is invention client station list structure figure
Fig. 5 is database structure figure of the present invention
Fig. 6 is task management data list structure figure in database
Fig. 7 is work management data list structure figure in database
Real-time informing management data list structure in Fig. 8 database
Fig. 9 is client task list functions of mouse right key menu of the present invention
Embodiment
For making the object of embodiments of the invention, technical scheme and advantage clearly, below in conjunction with the accompanying drawing in the embodiment of the present invention, clear complete description is carried out to the technical scheme in the embodiment of the present invention:
A distributed task dispatching method for trustship type, mainly comprises the steps:
S100. a certain decomposable distributed task scheduling is resolved into multiple work transferring to workstation to complete: the calculated amount of this work is usually huge especially, if give single computing machine or workstation carry out computing may periods of months or the time of several years.
Task described in the present invention, is formed by multiple parameter combinations, principal character be can with suitable granulometric become can by the fritter of computing machine independent operating; If can not divided task, be not suitable for carrying out Distributed Calculation.Accordingly, the specific parameter combinations become by Task-decomposing is exactly work of the present invention.
Embodiment 1, suppose to have the distributed task scheduling that to be solved, this task is according to four environmental parameters: wind-force size, wind direction, temperature and humidity carry out weather forecast.Known conditions: environmental parameter span is wind-force: 1-4, wind direction: 0-90, temperature: 15-25 and humidity: 40-65.
Application program (i.e. forecasting software) used by weather forecasting: WFC (abbreviation of Weather Forecast), with above-mentioned four parameters, for input parameter, (each parameter needs particular value to this program, and as wind-force gets 2, wind direction gets 30, temperature gets 20, humidity gets 45), export weather forecasting information, output format is as follows: clear to cloudy, component 2 grades, wind direction 30 °, temperature 20 DEG C, humidity 15%.
For convenience of description, here using wind-force as parameter X1, wind direction is as parameter X2, and temperature is as parameter X3, and humidity is as parameter X4; In practical problems, the span of parameters is continuous print (this is realistic physical phenomenon also), but carry out scientific algorithm to use computing machine, must problem be carried out abstract, the value of parameter is carried out discrete, here we are according to the request for utilization of WFC software simultaneously for ease of explaining " task ", " work " these two nouns, and we have done following setting:
The value of parameters is:
X1(wind-force): 1,2,3,4(is totally 4 data points)
X2(wind direction): 0,30,60,90(is totally 4 data points)
X3(temperature): 15,20,25(is totally 3 data points)
X4(humidity): 40,45,50,55,60,65(is totally 6 data points)
Like this, four parameters (X1, X2, X3, X4) form 4*4*3*6=288 parameter combinations altogether, as shown in the table:
These 288 parameter combinations just define " task " in the present invention, and the just corresponding work of each parameter combinations, each work can be carried out computing independently by WFC application program, such as, work Job1(and X1=1 that first group of parameter combinations is corresponding, X2=0, X3=15, X4=40), WFC application program carries out computing according to the parameter of Job1, obtains operation result Result1 after the some time.
S200. multiple workstations that described multiple work all give the computing of participation task according to quantity are processed: after obtaining 288 described work, described 288 work are all given each workstation according to number mean random, namely meets difference≤1 of " work " quantity that any 2 WA are assigned to.Suppose that the quantity of the workstation of participation task computing has 10, then according to mean random apportion design, each workstation can assign to 28 or 29 work.Meanwhile, the method adopting mean random to distribute, also ensure that the situation being distributed in a workstation in the working set avoiding causing calculated amount large.
Further, the conveniently optimize allocation of task, the work described in the present invention, before issuing workstation process, is all be stored in described server; The distribution of described work can be described as just setting up a pending work sequence for a workstation, determines which has by the work of this workstation process.Only have when a certain workstation completes a upper job in sequence, when preparing to process next task, the task in sequence is just issued to workstation by server, processes.So just facilitate system according to the situation of the real time execution of each workstation, adjust the work sequence of each workstation, and then reach the object optimized and share out the work.
S310. the hardware parameter of the workstation of all participation tasks process is added up, the ability to work of each workstation described in evaluation: in order to carry out real-time optimize allocation to work, before distributed component arithmetic system, first in system, or be described as the hardware performance of the workstation participating in distributed task scheduling computing, assess.
The key hardware of allocation of computer (CPU, internal memory, buffer memory etc.) performance parameter is used to weigh the computing power of this computing machine in the present invention.During practical operation, operating system can be utilized to carry or popular hardware assessment software each workstation is given a mark.
In the present invention, for arbitrary workstation, its computing power is exactly an intrinsic known attribute.Conveniently computing, in all working station end computing machine participating in certain task, by the computing power of the strongest that computing machine of digital 100 calibrated and calculated abilities, for other each workstation end computing machines, the computing power (be greater than 0 and be less than or equal to 100, namely numerical value is larger, represents that the computing power of this computing machine is stronger) between 0 to 100 of its correspondence.
S320: by described step S200, first task in the sequence of the pending work of each workstation obtained by leading random all point-scores, when working beginning, is first issued to predetermined each workstation, process, the work of recording each work is consuming time.Consuming time with this work, the computing power in conjunction with the workstation completing this work is defined as the calculated capacity of this work:
For each work in same task, under identical computing environment, (adopt same computer to carry out computing), certain work computing complete required time shorter (each work is continuously uninterrupted computing), then the calculated capacity that this work is corresponding is less.
The calculated capacity of work can be subject to the impact of each combination parameter.Be combined into the parameters of a described job, the value of any one parameter all can have influence on the calculated capacity of work, and meanwhile, different parameters is not identical to the influence degree of the calculated capacity of same work yet.
Any one be operated in do not have computing complete before, we can not obtain the exact time needed for this work of computing, namely can not obtain the calculated capacity of this task.But from whole task run management view, we can go to estimate according to those work completed in task the calculated capacity being in the work treating compute mode.
The time needed for a job run, being the computing power of the workstation end computing machine running this work on the one hand, was the calculated capacity of this work itself on the other hand mainly by the impact of two aspect factors.
Therefore, in order to utilize complete work to estimate the calculated capacity of the work treating compute mode, the impact of the computing power of workstation end computing machine first must be eliminated.In the present invention, run complete work for certain workstation, when recruitment is made trouble, the computing power of * workstation end computing machine is to characterize the calculated capacity of this work.The computing power making station end computing machine characterizes the calculated capacity of this work.Namely
Calculated capacity=work consuming time × computing power
Visible, the calculated capacity to working consuming time that works is directly proportional, and is inversely proportional to the computing power of computing machine.
In order to estimate the calculated capacity of the pending work in each work sequence, employing artificial neural network algorithm realizes the assessment to the calculated capacity being in the work treating compute mode.
The parameter of the work S330. completed at first in selecting system and calculated capacity, the sample as artificial neural network:
In the present invention, computing is completed the input Xt of combination parameter as artificial neural network of (first job in the pending work sequence of namely described each workstation) part work, calculated capacity (computing power of the * computing machine consuming time that works) corresponding to part work computing completed exports Yt as the target of artificial neural network, m group sample (Xt is obtained by the part work that computing completes, Yt), t=1,2,3,,, m.Namely as follows:
Application BP neural network algorithm is analyzed above-mentioned relation, finally obtains the relation function of calculated capacity Y and the parameters of composition work, that is, Y=f (X1, X2, X3,, Xi,,, Xn).
Constantly learnt in order to described artificial neural network can be made, use step S340. in the present invention using the parameter of the work completed successively in system (first job in the respective sequence that each workstation completes) and calculated capacity as the artificial neural network described in sample substitutes into, constantly make artificial neural network learn.
Same, continue the example of weather forecast, according to foregoing description, this task comprises 288 groups of parameter combinations altogether, often organizes corresponding one " work " of parameter combinations, here our hypothesis should complete 100 " work ", therefore we also just define 100 groups of samples, because the information of each sample (parameter combinations that sample is corresponding, computing is consuming time, the computing power etc. of corresponding computing machine) all known, therefore we can obtain following information easily:
Utilize these 100 samples and then obtain relation function Y.
Here we suppose the relation function Y=10X1+5X2+8X3+9X4+100 of this task.
Utilize this relation function, the parameter information that (X1=40, X2=90, X3=25, X4=65) (suppose that this work is in and treat compute mode) works is brought in relation function by we can obtain calculated capacity corresponding to this work.
S350. other are in the work treating compute mode and obtain corresponding calculated capacity according to which equally, therefore, define a calculated capacity tables of data for the work of compute mode for the treatment of all in this task.By the task scheduling modules in server by work large for calculated capacity, distribute to the workstation that in the workstation of participation task process, computing power is strong, the real-time allotment to work of finishing the work in process.
S410. result verification: when any operative station completes the computing of a job in all working station participating in computing, the result of this work is at least comprised: whether operation result exists the verification whether met the demands with operation result designated value;
S420. data encasement: if the result of work is by verification, carries out rename destination file, mobile destination file is to specified path and/or extract the operation of data to the file of specified format;
S430. described path of depositing through the destination file of data encasement is recorded;
S440. repeat step S410-430, until last computing that worked of this workstation, generate the destination file that comprises all working handled by this workstation.
S510., after a workstation completes distributed whole work, the operation result file of this workstation or the store path of destination file is received;
S520. repeat step S510, until all working station of the work of participation completes the process shared out the work separately, the destination file at all working station is gathered, generate the destination file of task or the address of task result, complete the computing of described distributed task scheduling.
As shown in figs 1-9:
The distributed task dispatching system of trustship type, comprises client, server and workstation.
Client is the remote subscriber interface of server-side application, is arranged on and can be connected on the computing machine of server by network arbitrarily.Client assisting users complete such as newly-built task, submission task, initiating task and check latest activities, the function such as to finish the work.
User can arrange sending short messages in groups device property parameters in server under task real-time informing module by client, and Main Function is the port attribute arranging sending short messages in groups device.
The SerialPort control under .NET Framework 4.0 platform is used to realize serial communication in the present invention.According to the relevant information of sending short messages in groups device hardware device, its (sending short messages in groups device) serial ports attribute is set.The parameters such as serial ports attribute mainly contains: port numbers, baud rate, parity check bit, data bit length, Handshake Protocol, position of rest, reads time-out, write time-out.
In the present invention when this sending short messages in groups device serial ports setup of attribute, provide convenient, flexible human oriented design: when the arranging of uncertain serial ports property parameters, client is provided with default setting and carrys out auxiliary parameter setting, automatic search function is provided in program, the correct setting of parameters can be realized easily.
After optimum configurations, can confirmatory connection be carried out, if each property parameters of sending short messages in groups device all arranges correctly, then point out optimum configurations correct, complete sending short messages in groups and arrange.If sending short messages in groups device has at least a property parameters to arrange wrong, then client can eject dialog box, and prompting optimum configurations is wrong, now reexamines and revises optimum configurations, carrying out confirmatory connection, until optimum configurations is correct.
The computer name of the corresponding computing machine of all right display server end of client, Computer IP address, and udp broadcast group.
Client also can send scan instruction to server, is mainly used to scan online workstation, to coordinate submission, initiating task use.
Server, after scanning completes, sends online station list to client, and the workstation acquiescence in list is all selected state, can share out the work, if user wants to cancel certain workstation, clicks in lists and cancels.
Same, in station list, user can select to select flexibly to start, suspend and reclaim corresponding work.The effect starting the work suspended in the workstation chosen is that the work that will suspend in the workstation chosen starts, and the work that after this this workstation is corresponding just brings into operation calculating successively, until run complete or suspended.The effect suspending the work of queue in the workstation chosen is by the work stoppage of queue in the workstation chosen; The effect of reclaiming the work suspended in the workstation chosen is reclaimed at the work suspended in the workstation chosen, and makes it be in holding state.
In the present embodiment, client is in order to realize the correlation function of above-mentioned control work, there is station list functions of mouse right key menu, this menu is made up of 11 function menus, be respectively [newly-built task], [work suspended in the task that startup is chosen], [work of queue in the task that time-out is chosen], [work standby in the task that submission is chosen], [work suspended in the task that recovery is chosen], [task is chosen in deletion], [removing completed task], [starting whole task], [suspending whole task], [attribute], 11 function menus such as [task management settings].
The effect of [newly-built task] creates new task, can eject newly-built task dialogue frame when clicking [newly-built task], and user can complete creation task operation easily;
The Main Function of [work suspended in the task that startup is chosen] is that the work that will be in halted state in choosing of task starts, and the work of after this queue in this task just brings into operation calculating successively, until run complete or suspended;
The effect of [work of queue in the task that time-out is chosen] is the work stoppage by being in quene state in choosing of task;
The effect of [work standby in the task that submission is chosen] is that each workstation is submitted in work standby in choosing of task, makes it be in halted state;
The effect of [work suspended in the task that recovery is chosen] is reclaimed at the work being in halted state in choosing of task, makes it be in holding state;
The effect of [task that deletion is chosen] is deleted at choosing of task, and the task of being about to choose moves to Old_Task tables of data from the Task tables of data database D B;
The effect of [removing completed task] be by run complete all tasks all from database D B Task tables of data move to Old_Task tables of data;
The effect of [starting whole task] is calculated by the whole startup optimization of all tasks being in halted state;
The effect of [suspending whole task] is that being all in queue of task is conjugated halted state by quene state;
The effect of [attribute] is the relevant information of checking choosing of task, is made up of 2 parts.Part I is ' routine ' item, the establishment of work that the workstation number having task names, task run state, task to enable, task agent contain (the work number in the work number of operating work number, the work number do not submitted to, time-out, queue, run complete work number and the number that must work), time cumulation, actual run time, speed-up ratio, queue prediction and task, beginning, deadline.。Part II is ' parameter ' item, and 3 parts such as, optimum configurations capable by task order, order preview form.
The effect of [task management setting] is the correlation parameter arranging task real-time informing, is made up of 3 parts.Part I is ' notice is arranged ' item, is made up of notice form, announcement period.Wherein notify that form provides 2 kinds of selections, is respectively SMS notification (needing the support of sending short messages in groups device hardware), mail notification; Announcement period provides 2 kinds of selections, is respectively the time interval, percentage-proportion intervals.Namely the time interval carries out the real-time informing of a Task Progress every the selected time period, namely percentage-proportion intervals carries out the real-time informing of Task Progress according to selected percentage-proportion intervals.The corresponding selection that user can select inside notice form, announcement period according to user demand.
Server-side application is installed on a server computer, at running background after unlatching.The main functional modules of server-side application has: central processing module, task scheduling modules, result post-processing module.The function of modules is:
Central processing module: the Main Function of this module is the instruction by receiving from client, carries out corresponding operating, realize the final function such as such as creation task, submission, startup, deletion to database DB.
Task scheduling modules: the Main Function of this module is that computing power according to each workstation end is (for same work, the time completed needed for this work is shorter, then the computing power of this workstation end computing machine is stronger), the distribution worked in real-time optimization task, reach best operational efficiency, namely complete the time optimization of a required by task.
The core methed realizing task scheduling has following 2 parts, and Part I is the predistribution taking to work in task, and Part II is the Optimized Operation taking to work in task.
Work predistribution:
In the present invention, before " work " startup optimization in task, first take randomly ordered for " work " in task, concentrate on probability on same workstation to avoid high " work " consuming time; Then take equal point-score in advance, " work " that a task comprised all gives each workstation end WA according to number, namely meets difference≤1 of " work " quantity that any 2 WA are assigned to.(work predistribution completed in ' newly-built task ' time, completed before the startup optimization that works in task.）
Optimization is dispatched:
After " work " startup optimization in task, task scheduling modules will according to certain algorithm, according to the real-time dynamic conditioning work allocation of the computing power of workstation end computing machine, to make required minimal time of finishing the work.For each workstation end computing machine, the time completed needed for same work is shorter, then the computing power of this workstation end computing machine is stronger.
Weigh the computing power of this computing machine by key hardware (CPU, internal memory, the buffer memory etc.) performance parameter of allocation of computer in the present invention.Therefore, in the present invention, for arbitrary computing machine, its computing power is a build-in attribute.In all working station end computing machine participating in certain task, by the computing power of the strongest that computing machine of digital 100 calibrated and calculated abilities, for the computing power of other each workstation end computing machines, the computing power (be greater than 0 and be less than or equal to 100, namely numerical value is larger, represents that the computing power of this computing machine is stronger) between 0 to 100 of its correspondence.
In same task, the calculated capacity of a job is subject to the impact of each running parameter, and the calculated capacity of work is weighed in the combination of the present invention's parameter.Be combined into the parameters of work, the value of any one parameter all can have influence on the calculated capacity of work, meanwhile, different parameters on the calculated capacity of same work to affect intensity not identical yet.
Any one be operated in do not have computing complete before, we can not obtain the exact time needed for this work of computing, namely can not obtain the calculated capacity of this task.But from whole task run management view, we can go to predict according to those work completed in task the calculated capacity being in the work treating compute mode.Run any one time needed for work mainly by the impact of two aspect factors, being the computing power of the workstation end computing machine running this work on the one hand, is the calculated capacity of this work itself on the other hand.Therefore, in order to utilize complete work to predict the calculated capacity of the work treating compute mode, the impact of the computing power of workstation end computing machine first must be eliminated.In the present invention, run complete work for certain workstation, when recruitment is made trouble, the computing power of * workstation end computing machine is to characterize the calculated capacity of this work.Namely
A mistake! Do not find Reference source.Calculated capacity=work consuming time × computing power
Visible, the calculated capacity to working consuming time that works is directly proportional, and is inversely proportional to the computing power of computing machine.
The assessment of calculated capacity that artificial neural network algorithm realizes being in the work treating compute mode is adopted in the present invention.(essence of artificial neural network embodies a kind of funtcional relationship between network input and output.By choosing different model structures and activation function, various different artificial neural network can be formed, obtain the relational expression between different input and output, and reaching different purposes of design, completing different functions.）
Result post-processing module: the Main Function of this module completes to each workstation end the destination file that " work " produce and processes.
Result treatment is divided into two stages, the process eventually of the pre-service of workstation end and server end.
The pre-service of workstation end:
1) trigger condition 1: any one " work " runs complete;
2) pre-service: perform the following step successively;
A. result verification: whether check result file exists, whether the value of assigned address meets the demands in destination file; Then the completion status of " work " be set to " completing " by verification and enter next step, otherwise just the completion status of " work " be set to " failure " and jump out pre-service;
B. data encasement: rename destination file, mobile destination file, to extract in data to the file of specified format (support plain text, MS-EXCEL, MS-ACCESS) or what does not do;
3) trigger condition 2: on current mechanism end last " work " complete pre-service;
4) preparation is uploaded: send the virtual route deposited through pretreated destination file to server end.Server end is process eventually:
1) trigger condition 1: any one workstation end completes uploads preparation;
2) data are downloaded: download the destination file of workstation end or the virtual route of only event memory file;
3) trigger condition 2: the destination file on last workstation end is downloaded virtual route that is complete or destination file and stored complete;
4) process eventually: if download destination file, then gather and finally process, final destination file is distributed on the data transmission interface of server end; If storing virtual path, then mapped on the data transmission interface of server end; (data transmission interface of server end supports the multiple transport protocols such as http, ftp)
5) notify: use the various ways such as Email, note notice user task to complete.
User directly downloads final destination file by client, also therefrom can extract download link, uses third party's instrument to download.
Database D B is arranged on computing machine corresponding to server end, is used for storing the information of all creation tasks.Be illustrated in figure 5 the structural drawing of database D B, this database is made up of 6 tables of data, is respectively Task tables of data, Old_Task tables of data, Job tables of data, Old_Job tables of data, Inform tables of data, Old_Inform tables of data.
Task tables of data is used for storing the information of institute creation task, and Old_Task tables of data is used for backing up the task of deleting from Task tables of data.Job tables of data is used for storing the information of the Job that institute creation task comprises, and Old_Job tables of data is used for backing up the task of deleting from Job tables of data.In the present invention, [job_status] field in the Job tables of data in database D B is used for indicating the running status of job, and following table is depicted as Job state value table.Inform tables of data is used for the real-time follow-up parameters information of store tasks, and Old_Inform tables of data is used for backing up the task of deleting from Inform tables of data.
The above; be only the present invention's preferably embodiment; but protection scope of the present invention is not limited thereto; anyly be familiar with those skilled in the art in the technical scope that the present invention discloses; be equal to according to technical scheme of the present invention and inventive concept thereof and replace or change, all should be encompassed within protection scope of the present invention.
Claims (9)
1. a distributed task dispatching method for trustship type, has following steps:
S100. a certain decomposable Task-decomposing is become multiple work transferring to workstation to complete;
S200. described multiple work are processed according to the pre-multiple workstations all giving the computing of participation task at random of quantity;
S300. carry out at described workstation in the process of work disposal, estimated by the calculated capacity of artificial neural network to pending work, by work large for calculated capacity, priority allocation is to the strong workstation of computing power;
S400., after any operative station completes the work described in one in the workstation of described participation task, result treatment is carried out to this work;
S500. after described multiple workstations complete all work, gather whole working results, generate task result;
Described step S300 specifically comprises step:
S310. the hardware parameter of the workstation of all participation tasks process is added up, the ability to work of each workstation described in evaluation;
S320. for completed work, set its calculated capacity=work consuming time × complete the computing power of the workstation of this work;
The parameter of the work S330. completed at first in selecting system and calculated capacity, as the sample of artificial neural network, obtain initial neural network relation function;
S340. using the parameter of work that completes successively in system and calculated capacity as the artificial neural network described in sample substitutes into, constantly improve the relation function of artificial neural network;
S350. the parameter of work to be allocated is substituted in current artificial neural network, estimate the calculated capacity of all work to be allocated; By work large for calculated capacity, distribute to the workstation that in the workstation of participation task process, computing power is strong.
2. the distributed task dispatching method of a kind of trustship type according to claim 1, be further characterized in that: in described step S310, in multiple workstations of setting participation task process, the computing power of the workstation that computing power is the strongest is 100, and the computing power of all the other each workstations is the number of 0-100.
3. the distributed task dispatching method of a kind of trustship type according to claim 1, is further characterized in that: described step S400 specifically comprises the steps:
S410. result verification: any operative station completes the computing of a job in all working station of the task of participation, at least comprises to the result of this work the verification that whether working result exists and whether operation result designated value meets the demands;
S420. data encasement: if the result of work is by verification, carries out rename destination file, mobile destination file is to specified path and/or extract the operation of data to the file of specified format;
S430. described path of depositing through the destination file of data encasement is recorded;
S440. repeat step S410-430, until last computing that worked of this workstation, generate the destination file that comprises all working handled by this workstation.
4. the distributed task dispatching method of a kind of trustship type according to claim 2, is further characterized in that: described step S500 specifically comprises:
S510., after a workstation completes distributed whole work, the operation result file of this workstation or the store path of destination file is received;
S520. step S510 is repeated, until all working station of the work of participation completes the process shared out the work separately, the destination file at all working station is gathered, generates the destination file of task or the address of task result files, complete the computing of described distributed task scheduling.
5. a distributed task dispatching system for trustship type, comprises at least one server and multiple workstation:
Described server receive client's client upload task, Task-decomposing is become multiple work after, then give multiple workstation by work allocation and in the process of workstation process, the distribution of work adjusted in real time; Workstation receives the work by server-assignment, processes, after processing, result is back to server, process of finishing the work; It is characterized in that
Described server has: task matching scheduler module and the whole processing module of task;
Described workstation has: work disposal module and working result pretreatment module;
During work:
Task matching scheduler module: receive the task that client uploads, Task-decomposing is become multiple work that can be completed separately process by any one workstation in described multiple workstation; Each workstation is all given at random in advance according to number by decomposing the multiple work obtained;
Workstation startup is to after the process of described work, task matching scheduler module is assessed the computing power of software to the workstation of the task of participation process by operating system or hardware and is assessed, for each workstation sets the numerical value that one represents the ability that it calculates;
Described task matching scheduler module, according to completed work, estimates the calculated capacity of work to be allocated by artificial neural network;
Described task matching scheduler module is by work large for calculated capacity, and priority allocation processes to the workstation that computing power is strong;
Described work of being decomposed by task matching scheduler module stores in the server, when described workstation needs to process work, work to be processed need be handed down to workstation by task matching scheduler module;
Each workstation receives the work by described server-assignment, carries out calculation process by the work disposal module in this workstation, generates destination file; After any one in the multiple work be assigned with that this workstation receives has worked and obtain destination file, described working result pretreatment module has at least comprised to this destination file the inspection that whether destination file exists and in destination file, whether particular value meets the demands;
After inspection, working result pretreatment module carries out data encasement, and this data encasement at least comprises: the conversion of the rename of destination file, movement and/or file layout;
After all having worked in this workstation, the whole processing module of the task that the virtual route that the destination file of all working is deposited uploads onto the server by described working result pretreatment module;
When a workstation completes distributed whole work, the whole processing module of described task store this workstation upload described in the virtual route of destination file of finishing the work or download destination file;
When each workstation relating to a process complete task completes all working that it is assigned with, all destination files gather by the whole processing module of described task, generate download path and/or generate task result files; Described download path and/or task result files are sent to the data transmission port of server, complete the computing of distributed task scheduling.
6. the distributed task dispatching system of a kind of trustship type according to claim 5, be further characterized in that: described server also has task real-time informing module, this notification module at least uses Email and note form that the progress msg of task process is informed to user.
7. the distributed task dispatching system of a kind of trustship type according to claim 6, be further characterized in that described server also has: to the database of user transparent, this database at least comprises:
The task management tables of data of the current Processing tasks of register system and historic task;
At least record the work management tables of data of the numbering of each work become by task division, the corresponding station number of this work, running parameter, duty, the startup of work and deadline and mission number corresponding to this work; And
Record the real-time informing management data list of the task process progress msg that described task real-time informing module transmits to user/supvr, in task process engineering, described task real-time informing module calls the information recorded in this real-time informing management data list, sends to the user specified.
8. the distributed task dispatching system of a kind of trustship type according to claim 7, be further characterized in that and also have: accept user instruction, assisting users completes the client at least comprising newly-built task, submission task, initiating task and check latest activity and finished the work.
9. the distributed task dispatching system of a kind of trustship type according to claim 5, its feature also with have consistent with server with described server capability.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210517389.8A CN103049330B (en) | 2012-12-05 | 2012-12-05 | A kind of trustship type distributed task dispatching method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210517389.8A CN103049330B (en) | 2012-12-05 | 2012-12-05 | A kind of trustship type distributed task dispatching method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103049330A CN103049330A (en) | 2013-04-17 |
| CN103049330B true CN103049330B (en) | 2015-08-12 |
Family
ID=48061980
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210517389.8A Active CN103049330B (en) | 2012-12-05 | 2012-12-05 | A kind of trustship type distributed task dispatching method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103049330B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11853043B1 (en) * | 2018-11-16 | 2023-12-26 | Ai Technologies, Inc. | Controlling operation of machine tools using artificial intelligence |
Families Citing this family (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104181873A (en) * | 2013-05-28 | 2014-12-03 | 深圳市菲恩测控科技有限公司 | Method, device and system for achieving digital product detection |
| US20150100531A1 (en) * | 2013-10-09 | 2015-04-09 | Qualcomm Incorporated | Method and apparatus to control and monitor neural model execution remotely |
| CN104281449A (en) * | 2014-09-16 | 2015-01-14 | 上海卫星工程研究所 | Managed spacecraft task management system and management method |
| US9778957B2 (en) * | 2015-03-31 | 2017-10-03 | Stitch Fix, Inc. | Systems and methods for intelligently distributing tasks received from clients among a plurality of worker resources |
| CN107742183A (en) * | 2017-10-10 | 2018-02-27 | 海尔集团公司 | task scheduling system, method and storage medium |
| CN109976809B (en) * | 2017-12-28 | 2020-08-25 | 中科寒武纪科技股份有限公司 | Scheduling method and related device |
| CN108710530A (en) * | 2018-02-24 | 2018-10-26 | 深圳市艾龙电子有限公司 | Task distribution formula processing method, device, network-termination device and storage medium |
| CN108897608B (en) * | 2018-05-31 | 2021-09-07 | 中国科学院软件研究所 | Data-driven extensible intelligent general task scheduling system |
| CN109343939B (en) * | 2018-07-31 | 2022-01-07 | 国家电网有限公司 | Distributed cluster and parallel computing task scheduling method |
| US11379727B2 (en) * | 2019-11-25 | 2022-07-05 | Shanghai United Imaging Intelligence Co., Ltd. | Systems and methods for enhancing a distributed medical network |
| CN110968065A (en) * | 2019-12-18 | 2020-04-07 | 北京云杉世界信息技术有限公司 | Fresh food processing control method and device |
| CN112016830A (en) * | 2020-08-27 | 2020-12-01 | 广东电网有限责任公司 | Patent file evaluation task allocation method and device |
| EP4277173A4 (en) * | 2021-01-13 | 2024-02-28 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Node determination method and apparatus of distributed task, device, and medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521044A (en) * | 2011-12-30 | 2012-06-27 | 北京拓明科技有限公司 | Distributed task scheduling method and system based on messaging middleware |
-
2012
- 2012-12-05 CN CN201210517389.8A patent/CN103049330B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521044A (en) * | 2011-12-30 | 2012-06-27 | 北京拓明科技有限公司 | Distributed task scheduling method and system based on messaging middleware |
Non-Patent Citations (2)
| Title |
|---|
| 于学军,程博.基于BP神经网络的工作量估算模型.《计算机科学》.2012,第39卷(第10期),第98页第2栏. * |
| 基于MPI的动态负载均衡任务分配方法设计实现;杨文有;《淮南师范学院学报》;20101231;第12卷(第3期);第36页第1栏第第2段、第2栏第2-3段，第37页第1栏第1-2段、第2栏第1段 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11853043B1 (en) * | 2018-11-16 | 2023-12-26 | Ai Technologies, Inc. | Controlling operation of machine tools using artificial intelligence |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103049330A (en) | 2013-04-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103049330B (en) | A kind of trustship type distributed task dispatching method and system | |
| CN102254246B (en) | Workflow managing method and system | |
| CN104780214B (en) | Cloud manufacture system based on cloud computing and 3 D-printing | |
| CN103309946B (en) | Multimedia file processing method, Apparatus and system | |
| CN103617062B (en) | The render farm Dynamic Deployment System of a kind of flexibility and method | |
| CN107621973A (en) | A kind of method for scheduling task and device across cluster | |
| CN104407912B (en) | A kind of virtual machine configuration method and device | |
| CN106844198A (en) | Distributed dispatching automation test platform and method | |
| De Coninck et al. | Dynamic auto-scaling and scheduling of deadline constrained service workloads on IaaS clouds | |
| CN104778076B (en) | A kind of cloud service workflow schedule method | |
| CN103838627B (en) | Workflow dispatching method based on workflow throughput maximization | |
| CN101303749A (en) | Method for scheduling workshop work facing to client requirement | |
| CN111209091B (en) | Scheduling method of Spark task containing private data in mixed cloud environment | |
| CN106775932B (en) | Real-time workflow scheduling method triggered by random event in cloud computing system | |
| CN108803553A (en) | Work order executes method and device, medium, numerically-controlled machine tool and production management system | |
| CN104615487A (en) | System and method for optimizing parallel tasks | |
| JP2011170496A (en) | Device and method for supporting plant construction plan | |
| CN104346214B (en) | Asynchronous task managing device and method for distributed environment | |
| CN105808347A (en) | Cluster server deployment calculation method and device | |
| De et al. | Task management in the new ATLAS production system | |
| CN116954873A (en) | Heterogeneous computing system, and method, device, equipment and medium for selecting power nodes of heterogeneous computing system | |
| EP2685693A1 (en) | Method for gathering queue information and job information in computation environment | |
| CN117331668A (en) | Job scheduling method, device, equipment and storage medium | |
| CN102799474A (en) | Cloud resource fault-tolerant scheduling method based on reliability drive | |
| CN104735134B (en) | A kind of method and apparatus serviced for providing calculating |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |