CN101576831A - Distributed calculating system and realization method - Google Patents
Distributed calculating system and realization method Download PDFInfo
- Publication number
- CN101576831A CN101576831A CNA2008100370569A CN200810037056A CN101576831A CN 101576831 A CN101576831 A CN 101576831A CN A2008100370569 A CNA2008100370569 A CN A2008100370569A CN 200810037056 A CN200810037056 A CN 200810037056A CN 101576831 A CN101576831 A CN 101576831A
- Authority
- CN
- China
- Prior art keywords
- formation
- execution
- time
- input
- task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention relates to a distributed calculating system, taking input type-input scale-output type-parameter as a reference frame and estimating the resource needs to carry out task scheduling. Further, when receiving operation, the system returns to a queue according to the estimated execution time and the condition of a current execution queue, wherein the operation is adhered to the queue. The operation queue comprises an instantaneous execution queue, a timely execution queue and a background execution queue.
Description
Technical field
The present invention relates to a kind of distributed computing system, is reference frame with input type-input size-output type-parameter, the estimation resource requirement, thus carry out task scheduling, belong to the Distributed Calculation field.
Background technology
The task scheduling technology in the operating system aspect, has many achievements in research, and the solution of practice.But, in Distributed Calculation, relevant achievement in research, and the solution of practice and few.
Distributed Calculation except the factor that the operating system task scheduling is considered, also relates to distinctive problems such as network communication.In this field, adopt load-balancing technique to handle concurrent visit usually.And current current load balancing scheme mostly is with the decision foundation of visit capacity as equilibrium, rather than serves as the decision foundation with accurate relatively computational resource occupancy.For the website, and common operation system, this is reasonably, because the shared computational resource of different access is more or less the same usually, and operation consuming time can be carried out when system start-up and when idle usually.
Along with improving constantly of the level of informatization, the demand of real-time analysis is obvious gradually, and in rising trend.This type systematic, the shared computational resource of different access may differ greatly, and the scheme of carrying out load balancing based on visit capacity can not finely satisfy this class demand.
Dispatching algorithm in the current practice is a target with the equilibrium propelling etc. that increases the computational resource utilization factor, shortens performance period and many group jobs normally.This can be suitable in the system that finishes or be exclusively used in batch job than the time in short for most of operations.But, the instant analysis system, the shared computational resource of different access may differ greatly, and spends the relative legacy system of operation of a large amount of computational resources, and is much more.
Summary of the invention
In view of above actual conditions, the object of the present invention is to provide a kind of distributed computing system, be reference frame with computation module, input metadata-input size-input type-output type-parameter value, the estimation resource requirement, thus carry out job scheduling.Wherein, an operation can relate to a plurality of calculation tasks with execution order, and a calculation task is the example of certain computation module, and the input metadata is meant the data structure of input data.Like this, can solve the load balancing scheme with the decision foundation of visit capacity as equilibrium, rather than with the accurate relatively computational resource occupancy drawback that serves as the decision foundation.
For batch processing system, system is not very crucial to final user's response time.Common non-batch processing system when design, can carry out the special processing of user interface, such as design schedule bar, job queue etc. usually in advance for the operation of consumes resources.For the instant analysis system, be difficult to carry out this class during design and handle, because identical operations, because the difference of input size may cause bigger computational resource to take difference, and input size is just known generally will move the time.And the hardware environment adaptability of these class processing means is relatively poor.Therefore, the present invention further proposes, and system is when receiving operation, unless return results immediately, otherwise return estimated execution time immediately, so that making corresponding user interfaces, client handles, put into backstage formation etc. such as progress displaying bar, prompting user.
Singularity in view of the instant analysis system, the invention discloses a kind of method of being convenient to estimate the calculation task resource requirement, each computation module is realized such method: parameter comprises the hash of input port name-metadata, the hash of input port name-input size, the hash of input port name-input type, the hash of output type, the hash of parameter-parameter value, and rreturn value is resource requirement.
For the ease of realizing computation module, and the resource requirement of estimating a calculation task, the present invention proposes the data type scheme of standardized calculation assembly I/O.Its requires input to comprise four types of metadata, form, program, model; Output comprises metadata, form, program, model, report and displaying.Wherein, form types can be the form of delegation's one row, also can comprise form in row; Form, program, types of models all have the metadata part that indicates its data structure.
The resource requirement of a calculation task, main and input metadata, input size, input type, output type, parameter value are relevant, and when importing when being not form, input size can not be considered.Therefore, based on above-mentioned data type scheme, input size just is simplified shown as the size of form.The relation of the resource requirement of computation module and metadata, parameter, input size is confirmable when computation module is realized, also can sum up out by concrete history run.When having only metadata, the resource requirement of calculation task almost can be ignored; When input comprised form, computational resource requirements was relevant with the form line number with the form metadata, sometimes also and parameter obvious relation is arranged; When input only comprised program, computational resource requirements was relevant with program structure usually, but when output also was program, the resource requirement of calculation task almost can be ignored usually.An operation generally includes a plurality of tasks, may also have a large amount of Network Transmission sometimes.Therefore, the present invention proposes the method for the computational resource requirements of a computational tasks:
(a) if exist calculation task to obtain the form input from client or another server, then increase corresponding network transmission resource demand, total demand is relevant with the transmission scale;
(b) if exist calculation task need export form to client, then increase corresponding network transmission resource demand, total demand is relevant with the transmission scale;
(c) above-mentioned resource requirement and, add the resource requirement of each task in the operation, constitute the resource requirement of this operation.
At last, the invention discloses a kind of job scheduling scheme that realizes native system:
(a) computational resource requirements of estimation operation;
(b) if operation can instantaneously be finished, the instantaneous execution formation of then preferential adding if instantaneous execution formation is full, then preferentially adds and in time carries out formation, also expires if in time carry out formation, then adds backstage execution formation;
(c) if operation can in time be finished, in time carry out formation then preferential the adding, if it is full in time to carry out formation, then adds the backstage and carry out formation;
(d) when carrying out the operation of in time carrying out in the formation, reduce the capacity of instantaneous execution formation, when executing an operation, recover the respective volume of instantaneous formation again;
(e) formation is carried out on the backstage, with task is minimum thread, when carrying out task carrying out the operation in the formation in the backstage, change the capacity of instantaneous execution formation according to the computational resource requirements situation of this task, change the capacity of in time carrying out formation in case of necessity, during this task termination, recover the respective volume of instantaneous execution formation and execution formation in time;
(f) in time carry out in the formation computing time demand and the longest preferential execution of stand-by period sum;
(g) job priority of instantaneous execution formation is carried out, and the operation of formation is in time carried out in execution then, carries out the operation in the execution formation of backstage when idle.
Description of drawings
Accompanying drawing has been described the task management formation and has been carried out the treatment scheme that adds new operation.
Embodiment
The summary of the invention chapters and sections, by the agency of the interface that need realize of computation module, the data type scheme of computation module I/O, the computational resource requirements computing method and the job scheduling scheme of computational tasks, introduce module, server end workflow, the client workflow of server section and client part here.
The server end module
Server section comprises session manager, computational resource requirements estimation device, job queue management device, job scheduler and task actuator.The session manager maintenance is connected with client, receives request, sends the result.Session manager is set up and is kept and being connected of client, and receives request, sends the result.Computational resource requirements estimation device, resource requirement estimation request is accepted in the mapping of maintenance calculations assembly and resource estimation interface thereof, returns the estimation result.The job queue management device is safeguarded instantaneous execution formation, is in time carried out formation and backstage and carry out formation, accepts to add operation, cancels operation, revises the operation executing state, the deletion operation, revises the request of capacity of queue.Job scheduler from job queue according to dispatching algorithm, the instantaneous execution formation of scheduled for executing and in time carry out operation in the formation, or the task of the operation in the formation is carried out on the backstage.Actual each computational tasks of execution of task actuator or task, and the result sent to client, passes to next task or be temporarily stored in server end by session manager.
The expression of operation in carrying out formation comprises that operation itself, session manager quote, and adds the information that scheduling such as the time of formation, the resource requirement of estimation are correlated with.If the Java realization to quoting of session manager, can not be strongly reference, in order to avoid disturb refuse collection.Carry out formation for three, by job queue management device and job scheduler visit.
The server end workflow
Client and server end are set up session, and session manager is handed to the job queue management device after receiving the computation requests of client transmission.As shown in drawings, the job queue management device calls computational resource requirements estimation device estimation resource requirement, if instantaneous computation requirement, and instantaneous execution formation less than, then join instantaneous execution formation, otherwise attempt joining timely execution formation,, then join the backstage and carry out formation if it is full in time to carry out formation; If timely computation requirement, and in time carry out formation less than, then join timely execution formation, carry out formation otherwise join the backstage; If the backstage computation requirement then joins the backstage and carries out formation.When the job queue management device adds corresponding execution formation with operation, if not adding instantaneous execution formation, then return the queue type that operation adds to client by session manager, if in time carry out formation, then can in return message, add the computational resource requirements of estimating simultaneously.Session manager receives after cancellation backstage that client sends carries out the request of operation, hand to the job queue management device, if the still non-delivery execution of operation of request cancellation, then directly carry out the formation and delete it from the backstage, if this operation has had the part task carrying out, the task that then first termination is being carried out, the resource of release busy is deleted this operation then from formation.According to dispatching algorithm, the calling task actuator is carried out the operation in instantaneous execution formation and the execution formation in time to job scheduler, or the task of the operation in the formation is carried out on the backstage from job queue.The task actuator executes after the operation, by quoting of session manager, the result is sent to client by session manager; If session disconnects, then the result is temporarily stored in server end, wait for that the user obtains the result in the future.The task actuator executes the backstage and carries out after the non-termination task of the operation in the formation, and the result passes to next task, and to the last a task termination sends to client to the result by session manager then, or temporary to server end.
Client modules
Client part comprises backstage assemblies such as connection manager, execution result audiomonitor, and be used to edit, the work space of preservation, submit job, demonstration execution result and check the user front ends such as background job manager of background job executing state, cancellation background job.
The client workflow
Client is connected to server end by connection manager, lands after the success, sets up the session of client-server end.After the user is submitted to server by front-end interface, the execution result audiomonitor, the monitoring client is connected with server, and the packets of information that resolution server sends it back obtains execution result.The execution queue type that if the result who returns is operation to add, and carry out queue type and in time carry out formation, then open a progress bar, or similar information, remind the user to wait for slightly, up to receiving final execution result; If carry out queue type is that formation is carried out on the backstage, then tells user task successfully to submit to, just carries out in server background.
Claims (7)
1, a kind of distributed computing system, it is characterized in that with computation module, input metadata-input type-input size-output type-parameter value be reference frame, the estimation resource requirement, thereby carry out job scheduling, wherein, an operation can relate to a plurality of calculation tasks with execution order, and a calculation task is the example of certain computation module, and the input metadata is meant the data structure of input data.
2, the system as claimed in claim 1 is characterized in that existing such 3 to carry out formation:
(a) instantaneous execution formation, the arbitrary operation in the inside all require from being submitted to end, the imperceptible wait of client;
(b) in time carry out formation, the arbitrary operation in the inside all requires from being submitted to end, and the required time is in final user's acceptable stand-by period;
(c) formation is carried out on the backstage, carries out when the operation of the inside has system idle.
3, system as claimed in claim 2 is characterized in that system when receiving operation, according to the situation of estimated execution time and current execution formation, returns the affiliated formation of operation.
4, realize the method for the described system of claim 1, it is characterized in that:
(a) the input canonical representation of computation module is the Hash table of the input type array of input port name-acceptance;
(b) the output canonical representation of computation module is the Hash table of delivery outlet name-output type;
(c) computation module is realized such method: parameter comprises the hash of input port name-metadata, the hash of input port name-input size, the hash of input port name-input type, the hash of output type, the hash of parameter-parameter value, and rreturn value is resource requirement.
5, realize the method for the described system of claim 1, it is characterized in that computation module adopts following I/O data type scheme:
(a) input comprises four types of metadata, form, program, model;
(b) output comprises metadata, form, program, model, report and displaying;
(c) form types can be the form of delegation's one row, also can comprise form in row;
(d) Program Type comprises database table/view name, SQL script, Java bytecode etc.;
(e) form, program, types of models all have the metadata part that indicates its form.
6, on the basis of claim 4, realize the method for the described system of claim 1, it is characterized in that adopting following method to calculate the computational resource requirements of an operation:
(a) if exist calculation task to obtain the form input from client or another server, then increase corresponding network transmission resource demand, total demand is relevant with the transmission scale;
(b) if exist calculation task need export form to client, then increase corresponding network transmission resource demand, total demand is relevant with the transmission scale;
(c) above-mentioned resource requirement and, add the resource requirement of each task in the operation, constitute the resource requirement of this operation.
7, realize the method for the described system of claim 2, it is characterized in that adopting following job scheduling scheme:
(a) computational resource requirements of estimation operation;
(b) if operation can instantaneously be finished, the instantaneous execution formation of then preferential adding if instantaneous execution formation is full, then preferentially adds and in time carries out formation, also expires if in time carry out formation, then adds backstage execution formation;
(c) if operation can in time be finished, in time carry out formation then preferential the adding, if it is full in time to carry out formation, then adds the backstage and carry out formation;
(d) when carrying out the operation of in time carrying out in the formation, reduce the capacity of instantaneous execution formation, when executing an operation, recover the respective volume of instantaneous formation again;
(e) formation is carried out on the backstage, with task is minimum thread, when carrying out task carrying out the operation in the formation in the backstage, change the capacity of instantaneous execution formation according to the computational resource requirements situation of this task, change the capacity of in time carrying out formation in case of necessity, during this task termination, recover the respective volume of instantaneous execution formation and execution formation in time;
(f) in time carry out in the formation computing time demand and the longest preferential execution of stand-by period sum;
(g) job priority of instantaneous execution formation is carried out, and the operation of formation is in time carried out in execution then, carries out the operation in the execution formation of backstage when idle.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2008100370569A CN101576831A (en) | 2008-05-07 | 2008-05-07 | Distributed calculating system and realization method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNA2008100370569A CN101576831A (en) | 2008-05-07 | 2008-05-07 | Distributed calculating system and realization method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN101576831A true CN101576831A (en) | 2009-11-11 |
Family
ID=41271768
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNA2008100370569A Pending CN101576831A (en) | 2008-05-07 | 2008-05-07 | Distributed calculating system and realization method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101576831A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102404211A (en) * | 2011-11-15 | 2012-04-04 | 北京天融信科技有限公司 | Method and device for realizing load balancing of processors under AMP framework |
| CN102902591A (en) * | 2012-09-04 | 2013-01-30 | 深圳市宏电技术股份有限公司 | Task execution method of not long on-line terminal and not long on-line terminal |
| CN104346220A (en) * | 2013-07-31 | 2015-02-11 | 中国科学院计算技术研究所 | Task scheduling method and system |
| CN105207948A (en) * | 2014-06-30 | 2015-12-30 | 中兴通讯股份有限公司 | Data message distribution method and apparatus |
| CN107766129A (en) * | 2016-08-17 | 2018-03-06 | 北京金山云网络技术有限公司 | A kind of task processing method, apparatus and system |
| CN108009006A (en) * | 2016-11-02 | 2018-05-08 | 华为技术有限公司 | The dispatching method and device of I/O requests |
| CN108270693A (en) * | 2017-12-29 | 2018-07-10 | 珠海国芯云科技有限公司 | The adaptive optimization leading method and device of website visiting |
| WO2021223174A1 (en) * | 2020-05-07 | 2021-11-11 | Citrix Systems, Inc. | Task shifting between computing devices |
-
2008
- 2008-05-07 CN CNA2008100370569A patent/CN101576831A/en active Pending
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102404211A (en) * | 2011-11-15 | 2012-04-04 | 北京天融信科技有限公司 | Method and device for realizing load balancing of processors under AMP framework |
| CN102902591A (en) * | 2012-09-04 | 2013-01-30 | 深圳市宏电技术股份有限公司 | Task execution method of not long on-line terminal and not long on-line terminal |
| CN104346220A (en) * | 2013-07-31 | 2015-02-11 | 中国科学院计算技术研究所 | Task scheduling method and system |
| CN104346220B (en) * | 2013-07-31 | 2017-11-03 | 中国科学院计算技术研究所 | A kind of method for scheduling task and system |
| CN105207948B (en) * | 2014-06-30 | 2019-05-31 | 中兴通讯股份有限公司 | The distribution method and device of data message |
| CN105207948A (en) * | 2014-06-30 | 2015-12-30 | 中兴通讯股份有限公司 | Data message distribution method and apparatus |
| CN107766129B (en) * | 2016-08-17 | 2021-04-16 | 北京金山云网络技术有限公司 | Task processing method, device and system |
| CN107766129A (en) * | 2016-08-17 | 2018-03-06 | 北京金山云网络技术有限公司 | A kind of task processing method, apparatus and system |
| CN108009006A (en) * | 2016-11-02 | 2018-05-08 | 华为技术有限公司 | The dispatching method and device of I/O requests |
| CN108009006B (en) * | 2016-11-02 | 2022-02-18 | 华为技术有限公司 | Scheduling method and device of I/O (input/output) request |
| CN108270693A (en) * | 2017-12-29 | 2018-07-10 | 珠海国芯云科技有限公司 | The adaptive optimization leading method and device of website visiting |
| WO2021223174A1 (en) * | 2020-05-07 | 2021-11-11 | Citrix Systems, Inc. | Task shifting between computing devices |
| US11561822B2 (en) | 2020-05-07 | 2023-01-24 | Citrix Systems, Inc. | Task shifting between computing devices |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101576831A (en) | Distributed calculating system and realization method | |
| CN101882161B (en) | Application level asynchronous task scheduling system and method | |
| CN101084680B (en) | Method for managing resources in a platform for telecommunication service and/or network management, corresponding platform and computer program product therefor | |
| CN101692208B (en) | Task scheduling method for processing real-time traffic information | |
| Yang et al. | An algorithm in SwinDeW-C for scheduling transaction-intensive cost-constrained cloud workflows | |
| WO2021159638A1 (en) | Method, apparatus and device for scheduling cluster queue resources, and storage medium | |
| CN103685309B (en) | The method for building up of Asynchronous Request queue towards map visualization tile service access | |
| CN101290668B (en) | Time sharing operation dynamic dispatching method and device | |
| CN103838621B (en) | Method and system for scheduling routine work and scheduling nodes | |
| US20050076043A1 (en) | Workload scheduler with resource optimization factoring | |
| CN101957780A (en) | Resource state information-based grid task scheduling processor and grid task scheduling processing method | |
| CN105095327A (en) | Distributed ELT system and scheduling method | |
| CN105912401A (en) | Distributed data batch processing system and method | |
| JP2008527514A5 (en) | ||
| CN102223419A (en) | Virtual resource dynamic feedback balanced allocation mechanism for network operation system | |
| CN103324534A (en) | Operation scheduling method and operation scheduler | |
| CN104298550A (en) | Hadoop-oriented dynamic scheduling method | |
| CN100385402C (en) | Dynamic online scheduling system in gridding circumstance and its scheduling method | |
| US20080221857A1 (en) | Method and apparatus for simulating the workload of a compute farm | |
| CN111782627B (en) | Task and data cooperative scheduling method for wide-area high-performance computing environment | |
| CN107430526A (en) | For dispatching the method and node of data processing | |
| JPH1083382A (en) | Decentralized system operation maintenance support device and operation maintenance supporting method | |
| CN108665157A (en) | A method of realizing cloud Workflow system flow instance balance dispatching | |
| CN103336720B (en) | A kind of job execution method with data dependence relation based on SLURM | |
| CN107038072B (en) | Method for scheduling task and device based on Hadoop system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| DD01 | Delivery of document by public notice |
Addressee: Wan Dehong Document name: Notification of before Expiration of Request of Examination as to Substance |
|
| DD01 | Delivery of document by public notice |
Addressee: Wan Dehong Document name: Notification that Application Deemed to be Withdrawn |
|
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Open date: 20091111 |