CN102763086A - Task processing system for distributed computation and task processing method for distributed computation - Google Patents
Task processing system for distributed computation and task processing method for distributed computation Download PDFInfo
- Publication number
- CN102763086A CN102763086A CN2012800001658A CN201280000165A CN102763086A CN 102763086 A CN102763086 A CN 102763086A CN 2012800001658 A CN2012800001658 A CN 2012800001658A CN 201280000165 A CN201280000165 A CN 201280000165A CN 102763086 A CN102763086 A CN 102763086A
- Authority
- CN
- China
- Prior art keywords
- task
- scheduler
- subtask
- formation
- layer scheduler
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
Embodiments of the invention provide a task processing system for distributed computation and a task processing method for the distributed computation. The task processing system comprises a first level scheduler which is used for receiving requests of executing a task, starting or selecting a second level scheduler according to the task, and transmitting the requests to the second level scheduler; and the second level scheduler which is used for decomposing the task into a plurality of subtasks according to a logical relationship of the task when receiving the requests are transmitted by the first level scheduler. The embodiments of the invention employ a two-level scheduling frameworks, with the second level scheduler corresponding to the task and the first level scheduler starting or selecting the second level scheduler corresponding to the task, so that the task processing system and the task processing method can be used in different tasks, and processing efficiency and scheduling flexibility are improved.
Description
Technical field
The embodiment of the invention relates to network communication field, and more specifically, relates to distributed computing task disposal system and task processing method.
Background technology
At present, along with Internet development, the demand of the fast processing of bulk information is become very urgent.Therefore the parallel processing of data just becomes very important.DCE provides the effective means of different soft, hardware platform resource sharings and interoperability under the network environment, becomes the framework commonly used of parallel processing.The parallel processing system (PPS) that present industry is known adopts the MapReduce framework.MapReduce is the Distributed Calculation software architecture, and it can support the distributed treatment of big data quantity.This framework originates from map (mapping) and two functions of reduce (reduction) of functional expression formula at first.Map refers to original document is handled according to self-defining mapping ruler, output intermediate result.Reduce merges middle result according to self-defining reduction rule.
In DCE, the generic structure of MapReduce comprises scheduling node and a plurality of working node.Scheduling node is responsible for task scheduling and resource management; Be responsible for the configuration according to the user, the task that the user is submitted to is decomposed into map, two kinds of subtasks of reduce, and distributes map, reduce subtask to working node.Working node is responsible for moving map, reduce subtask, keeps in communication with scheduling node.
In this parallel processing framework,, and need strictly successively to carry out the task processing according to map, the order in two steps of reduce because a scheduling node is responsible for task and resource management.If there is the processing of a lot of steps, then need accomplish through submitting many times task requests to, treatment effeciency is lower, the scheduling underaction.
Summary of the invention
The embodiment of the invention provides a kind of task processing system and task processing method, can solve the problem of treatment effeciency in the existing parallel processing framework.
On the one hand, a kind of distributed computing task disposal system is provided, has comprised: the ground floor scheduler, be used to receive the request of executing the task, start or second layer scheduler that the selection task is corresponding and transmit described request to second layer scheduler; Second layer scheduler is used for when receiving the request that the ground floor scheduler transmits, and according to the logical relation of task task is decomposed into a plurality of subtasks.
On the other hand, a kind of distributed computing task disposal route is provided, this method comprises: the ground floor scheduler starts or second layer scheduler that the selection task is corresponding when receiving the request of executing the task; The ground floor scheduler is transmitted and should be asked to second layer scheduler; Second layer scheduler is decomposed into a plurality of subtasks according to the logical relation of task with task when receiving the request that the ground floor scheduler transmits.
The embodiment of the invention adopts two-layer scheduling framework, and second layer scheduler is corresponding to task, the second layer scheduler that the ground floor scheduler starts or the selection task is corresponding, thus go for various tasks, improved treatment effeciency and dispatching flexibility.
Description of drawings
In order to be illustrated more clearly in the technical scheme of the embodiment of the invention; To do to introduce simply to the accompanying drawing of required use in embodiment or the description of the Prior Art below; Obviously, the accompanying drawing in describing below only is some embodiments of the present invention, for those of ordinary skills; Under the prerequisite of not paying creative work, can also obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is the block diagram of the task processing system of the embodiment of the invention.
Fig. 2 is the synoptic diagram of the processing framework of one embodiment of the invention.
Fig. 3 is the process flow diagram of the task processing method of one embodiment of the invention.
Fig. 4 is the schematic flow diagram of the task processes of one embodiment of the invention.
Embodiment
To combine the accompanying drawing in the embodiment of the invention below, the technical scheme in the embodiment of the invention is carried out clear, intactly description, obviously, described embodiment is the present invention's part embodiment, rather than whole embodiment.Based on the embodiment among the present invention, those of ordinary skills are not making the every other embodiment that is obtained under the creative work prerequisite, all belong to the scope of the present invention's protection.
Fig. 1 is the block diagram of the distributed computing task disposal system of the embodiment of the invention.The task processing system 10 of Fig. 1 comprises two-layer scheduler, i.e. ground floor scheduler 11 and second layer scheduler 12.
For example, when in system, not having suitable second layer scheduler, ground floor scheduler 11 can start the corresponding second layer scheduler 12 of this task.When in system, having had suitable second layer scheduler, ground floor scheduler 11 can be selected the corresponding second layer scheduler 12 of this task from these suitable second layer schedulers.
Alternatively, said ground floor scheduler also is used for said task is carried out priority management, and starts or select said second layer scheduler that said task is handled according to said priority.
The embodiment of the invention adopts two-layer scheduling framework, and second layer scheduler is corresponding to task, the second layer scheduler that the ground floor scheduler starts or the selection task is corresponding, thus go for various tasks, improved treatment effeciency and dispatching flexibility.
In existing parallel processing framework, have only one deck scheduling, handle thereby need strictly successively to carry out task, and the embodiment of the invention does not have this restriction according to map, two steps of reduce.The ground floor scheduler 11 of the embodiment of the invention can be accepted various forms of tasks, and the form of task is not necessarily limited to the map of strictness of the prior art, two steps of reduce.Second layer scheduler 12 is corresponding to task, and ground floor scheduler 11 can send to corresponding second layer scheduler 12 with various tasks and dispatch processing like this.Second layer scheduler 12 is decomposed into the subtask so that task is handled with task, for example dispatches the execution of each subtasks.Such scheduling has higher dirigibility.
In addition, needing strictly successively to carry out task according to map, the order in two steps of reduce in the prior art handles.If there is the processing of a lot of steps, then need accomplish through submitting many times task requests to, treatment effeciency is lower.The embodiment of the invention does not have this restriction.The embodiment of the invention is to task itself and execute the task or the mode of subtask does not limit.For example, the number of the subtask that is comprised in the task can be more than these two kinds of map, the reduce of prior art, like the subtask more than three or three; And be not limited to the form of map, reduce subtask.In addition, strict sequencing needn't be followed in the subtask, can carry out concurrently, carry out to serial or the execution of part parallel part serial ground.Like this, even the processing of a lot of steps, also the task requests of a less number of times of needs has improved treatment effeciency.
The number of subtask is relevant with concrete task, like tasks such as transcoding, recognition of face business.According to the logical relation of task, these tasks possibly have identical or different subtask number.Alternatively, as an embodiment, can in the description document of task, carry the logical relation of this task.For example; Task processing system 10 (particularly, for example second layer scheduler 12) can receive the description document that the user uploads, like XML (Extensible Markup Language; Extend markup language) description document of form is carried the logical relation of task in this description document.
Alternatively, second layer scheduler 12 obtains the description document of the corresponding XML form of this task when receiving the request of ground floor scheduler 11 forwardings.The logical relation of the task of carrying in the description document according to this XML form is decomposed into a plurality of subtasks with this task.
In addition, each subtask can also further be decomposed into the more subtask of small grain size.That is to say that the subtask of the embodiment of the invention can be the multilayer subtask, the mode of the further decomposition of every straton task all can be confirmed through the logical relation of carrying in the description document.For instance, subtask 1 can be decomposed into a plurality of subtasks 2, and subtask 2 also can also further be decomposed into a plurality of subtasks 3 or the like.
Alternatively, as an embodiment, the logical relation of task can be indicated the execution dependence of a plurality of subtasks.The so-called dependence of carrying out is meant between the executable operations of each subtasks whether interdepend.
For instance, suppose that subtask 2 must depend on the execution result of subtask 1, then subtask 2 should be carried out (being that subtask 1 needs serial ground to carry out with subtask 2) again after carry out subtask 1.On the other hand, if subtask 2 does not rely on whole execution results of subtask 1, then subtask 1 and subtask 2 can executed in parallel, also can serial carry out.
A nonrestrictive example carrying out dependence can comprise: two or more subtasks in a plurality of subtasks are carried out according to serial or parallel or part parallel part serial order, and are not limited to map of the prior art, these two steps of reduce.Like this, if there is the processing of a lot of steps in a certain task, then need not as the MapReduce framework, to submit to task requests many times, the embodiment of the invention possibly only need be submitted to once or task requests several times on a small quantity, thereby has improved task handling efficient.
The logical relation of task can be indicated the execution dependence between the subtask by explicitly, and for example explicitly representes that this task is to be made up of the subtask 1-3 that the priority serial is carried out.Perhaps, the logical relation of task can implicitly be indicated the execution dependence between the subtask, and for example for a certain particular task, system knows that in advance this task is to be made up of the subtask 1-3 that the priority serial is carried out.
Alternatively, as another embodiment, second layer scheduler 12 also is used to said a plurality of subtask and creates corresponding formation to store the task that said subtask comprises.When in said formation, having stored the task that said subtask comprises; Second layer scheduler 12 can also be used to said subtask application resource; And the working cell manager startup working cell of the indication resource of applying for, so that said working cell obtains task that said subtask comprises to execute the task from said formation.Alternatively, as another embodiment, the result that second layer scheduler 12 can also be used for indicating said working cell to execute the task puts into another formation or exports the said result who executes the task.
Further, as another embodiment, second layer scheduler 12 can also be used to obtain the progress msg of said formation and said working cell, to confirm said task executions progress.
In a word, the embodiment of the invention does not limit the concrete form of task.Alternatively, the logical relation setting of task or selection can be supported User Defined, for example receive user's setting or selection through Plugin Mechanism.
The task processing system 10 of the embodiment of the invention can be applicable to the cloud computing framework.Cloud computing has proposed a kind of high reliability, low cost, has used as required, flexible business model.A lot of systems can be through using cloud service, reaches high reliability, elasticity, target cheaply.
Fig. 2 is the synoptic diagram of the processing framework of one embodiment of the invention.The processing framework 20 of Fig. 2 is a kind of cloud computing frameworks, comprises the task processing system 10 of Fig. 1.Be that with the difference of Fig. 1 the task processing system 10 of Fig. 2 can comprise a plurality of second layer schedulers 12.For succinctly, only described two second layer schedulers 12 among Fig. 2, but the number of second layer scheduler 12 does not receive the restriction (can be more or less) of this example.Each second layer scheduler 12 is corresponding to a kind of task, with adaptive or support Different Calculating Models.Alternatively, a plurality of second layer schedulers 12 also can be corresponding to a kind of task, to realize the high concurrency of system call.If there is the suitable second layer scheduler 12 corresponding to task in existing a plurality of second layer scheduler 12, then ground floor scheduler 11 can select 12 pairs of tasks of this suitable second layer scheduler to handle; If not corresponding to the suitable second layer scheduler 12 of task, then ground floor scheduler 11 can start new 12 pairs of tasks of suitable second layer scheduler and handles in existing a plurality of second layer scheduler 12.
In handling framework 20, ground floor scheduler 11 can be distributed, to support high concurrency.Ground floor scheduler 11 can receive the task requests that Webservice (network service) 21 sends.Webservice 21 is responsible for the reception and the forwarding of user's web (network) request, and therefore concrete implementation can repeat no more with reference to prior art.
Alternatively; As an embodiment; When ground floor scheduler 11 received a plurality of task requests, ground floor scheduler 11 can also carry out priority management (for example carrying out prioritization) to task, and handled according to priority startup or 12 pairs of tasks of selection second layer scheduler.For example, ground floor scheduler 11 can preferentially start or select the pairing second layer scheduler 12 of the higher task of priority.
Alternatively, as another embodiment, ground floor scheduler 11 can be realized the additional functions such as priority adjustment of task.The mode of prioritization or adjustment can be supported User Defined, for example receives user's setting through Plugin Mechanism.
Particularly, second layer scheduler 12 can be created corresponding formation with the storage task that the subtask was comprised for a plurality of subtasks.Second layer scheduler 12 can be put the order of formation according to the logical relation of task in order.For example; Suppose that the subtask 1-3 (subtask 1-＞2-＞subtask, subtask 3) that task is carried out by the priority serial constitutes; Second layer scheduler 12 can be set up formation 1-3; Store the task that subtask 1-3 is comprised respectively, and the order of definite formation 1-3, promptly successively carry out comprising in the corresponding subtask of task according to the order of formation 1-＞formation 2-＞formation 3.The task action result of subtask 1 is put into formation 2, and the task action result of subtask 2 is put into formation 3, and the task action result of subtask 3 exports suitable position to, for example exports distributed storage device shown in Figure 2 24 to or returns to the user.
Alternatively, as another embodiment, when in distributed queue 22, having stored the task that the subtask comprises, second layer scheduler 12 can also be this subtask application resource, for example from explorer 25 application resources.Explorer 25 is responsible for satisfying resource bid, the release of scheduler 11 or 12.The major function of explorer 25 comprises that resource management, resource matched, resource are flexible automatically.Wherein resource matched method can adopt Plugin Mechanism, supports User Defined.In addition, so-called resource is stretched when being meant that user's allocation cluster scale is in a scope automatically, can come automatic dilatation cluster or subtract the appearance cluster according to the cluster loading condition.Therefore other implementations of explorer 25 can repeat no more with reference to prior art.For example, explorer 25 also can adopt distributed schemes, to realize high concurrency.
After setting up formation for the subtask and applying for resource; Second layer scheduler 12 can be indicated the manager 26 startup working cells 27, working cell (worker) of the resource of applying for, so that working cell 27 obtains the task that the subtask comprises and carries out this task from formation.Worker manager 26 is responsible for establishment, deletion, the monitoring of worker 27.On each node in the cloud computing framework (physical machine or virtual machine) worker manager 26 is arranged all.Therefore other implementations of worker manager 26 can repeat no more with reference to prior art.
Worker 27 responsible respective queue from distributed queue 22 are obtained user's the task that the subtask comprised; Carry out pre-service; The handling procedure of invoke user exploitation more afterwards; After the pending completion, can the result who execute the task be put into another formation or export the result who executes the task according to the order of second layer scheduler 12 determined formations.Therefore other implementations of Worker 27 can repeat no more with reference to prior art.
In addition, second layer scheduler 12 can also be realized other scheduling processing, for example task abnormity processing or Task Progress statistics etc.For example, second layer scheduler 12 can obtain formation and working cell (worker) progress msg (as whether accomplishing or what accomplished in each subtask, the subtask in each formation whether accomplish or accomplished what or the like), with the implementation progress that sets the tasks.Like this, can realize the real-time inquiry of Task Progress.For example, the user can arrive the implementation progress of second layer scheduler 12 inquiry corresponding task.Perhaps, second layer scheduler 12 can report the progress msg of task ground floor scheduler 11, so that the user is to the implementation progress of ground floor scheduler 11 inquiry corresponding task, user friendly monitoring.
For succinctly, among Fig. 2 illustration three worker managers 26 and corresponding three worker 27, but the embodiment of the invention is not limited to this object lesson, the number of worker manager 26 and worker 27 can be more or less.
The cluster management software 28 responsible robotizations of handling the cluster of parallel task are disposed with basic and are monitored, and therefore its implementation can repeat no more with reference to prior art.
Distributed queue 22, database 23 (like the nosql database), distributed storage device 24 realizes handling framework 20 required task storage, database and file storage, and therefore concrete implementation also can repeat no more with reference to prior art.For example, database 23 can be used for the information persistent storage, to satisfy system's operation needs or to realize fault tolerance.
Handle the bottom of framework 20 and support various isomerization hardwares such as physical machine or virtual machine 29, for the user uses, need not to be concerned about.Therefore the implementation of physical machine or virtual machine 29 can repeat no more with reference to prior art.
Handle the computation model that framework 20 adopts " formation-worker ", but the embodiment of the invention is not limited thereto.Handle framework 20 and also can adopt other computation models, for example, the part second layer scheduler of handling in the framework 20 12 also can adopt above-mentioned MapReduce mode, and need not formation.
Therefore; The processing framework 20 of the embodiment of the invention adopts two-layer scheduling framework, and second layer scheduler is corresponding to task, and the ground floor scheduler starts or the pairing second layer scheduler of selection task; Thereby go for various tasks, improved treatment effeciency and dispatching flexibility.And, through the computation model of above-mentioned " formation-worker ", can start a plurality of second layer schedulers of different task simultaneously, further improve concurrent performance.
In addition, the embodiment of the invention provides high-performance, parallel processing framework 20 can be supported physical machine and at present more popular cloud computing platform, support large-scale cluster, support user scheduling strategy configuration and self-defined, support Different Calculating Models flexibly.
Fig. 3 is the process flow diagram of the distributed computing task disposal route of one embodiment of the invention.The method of Fig. 3 can be carried out by the task processing system 10 of Fig. 1 and Fig. 2, therefore describes the method for Fig. 3 below in conjunction with Fig. 1 and Fig. 2, and suitably omits the description of repetition.
301, ground floor scheduler 11 starts or second layer scheduler 12 that the selection task is corresponding when receiving the request of executing the task.
For example, when in system, not having suitable second layer scheduler, ground floor scheduler 11 can start the corresponding second layer scheduler 12 of this task.When in system, having had suitable second layer scheduler, ground floor scheduler 11 can be selected the corresponding second layer scheduler 12 of this task from these suitable second layer schedulers.
302, ground floor scheduler 11 is transmitted request to second layer scheduler 12.
303, second layer scheduler 12 is decomposed into a plurality of subtasks according to the logical relation of task with task when receiving the request that the ground floor scheduler transmits.
The embodiment of the invention adopts two-layer scheduling framework, and second layer scheduler is corresponding to task, and the ground floor scheduler starts or the pairing second layer scheduler of selection task, thereby goes for various tasks, has improved treatment effeciency and dispatching flexibility.
In existing parallel processing framework, have only one deck scheduling, handle thereby need strictly successively to carry out task, and the embodiment of the invention does not have this restriction according to map, two steps of reduce.The ground floor scheduler 11 of the embodiment of the invention can be accepted various forms of tasks, and the form of task is not necessarily limited to the map of strictness of the prior art, two steps of reduce.Second layer scheduler 12 is corresponding to task, and ground floor scheduler 11 can send to corresponding second layer scheduler 12 with various tasks and dispatch processing like this.Second layer scheduler 12 is decomposed into the subtask so that task is handled with task, for example dispatches the execution of each subtasks.Such scheduling has higher dirigibility.
In addition, needing strictly successively to carry out task according to map, the order in two steps of reduce in the prior art handles.If there is the processing of a lot of steps, then need accomplish through submitting many times task requests to, treatment effeciency is lower.The embodiment of the invention does not have this restriction.The embodiment of the invention is to task itself and execute the task or the mode of subtask does not limit.For example, the number of the subtask that is comprised in the task can be more than these two kinds of map, the reduce of prior art, like the subtask more than three or three; And be not limited to the form of map, reduce subtask.In addition, strict sequencing needn't be followed in the subtask, can carry out concurrently, carry out to serial or the execution of part parallel part serial ground.Like this, even the processing of a lot of steps, also the task requests of a less number of times of needs has improved treatment effeciency.
Alternatively, as an embodiment, the logical relation of task can be indicated the execution dependence of a plurality of subtasks.The so-called dependence of carrying out is meant between the executable operations of each subtasks whether interdepend.
For instance, suppose that subtask 2 must depend on the execution result of subtask 1, then subtask 2 should be carried out (being that subtask 1 needs serial ground to carry out with subtask 2) again after carry out subtask 1.On the other hand, if subtask 2 does not rely on whole execution results of subtask 1, then subtask 1 and subtask 2 can executed in parallel, also can serial carry out.
A nonrestrictive example carrying out dependence can comprise: two or more subtasks in a plurality of subtasks are carried out according to serial or parallel or part parallel part serial order, and are not limited to map of the prior art, these two steps of reduce.Like this, if there is the processing of a lot of steps in a certain task, then need not as the MapReduce framework, to submit to task requests many times, the embodiment of the invention possibly only need be submitted to once or task requests several times on a small quantity, thereby has improved task handling efficient.
Alternatively, as another embodiment, second layer scheduler 12 also can be a plurality of subtasks and creates corresponding formation with the task that the storage subtask comprises, and puts the order of formation in order according to the logical relation of task.
Alternatively; As another embodiment; When second layer scheduler 12 also can have been stored the task that the subtask comprises in formation; Be subtask application resource, and the manager startup working cell, working cell (worker) of the indication resource of applying for, so that the working cell obtains task that the subtask comprises to execute the task from formation.
Alternatively, as another embodiment, the result that second layer scheduler 12 also can indicate the working cell to execute the task puts into another formation or exports the result who executes the task.
Alternatively, as another embodiment, second layer scheduler 12 also can obtain the progress msg of formation and working cell, with the implementation progress that sets the tasks.Like this, can realize the real-time inquiry of Task Progress, make things convenient for user monitoring.
Alternatively, as another embodiment, in step 301, ground floor scheduler 11 can carry out priority management to task, and handles according to priority startup or 12 pairs of tasks of selection second layer scheduler.
Below, in conjunction with object lesson, embodiments of the invention are described in further detail.Fig. 4 is the schematic flow diagram of the task processes of one embodiment of the invention.For example, the process of Fig. 4 can be carried out by the processing framework 20 of Fig. 2, therefore suitably omits the description of repetition.
In the example of Fig. 4, suppose that the subtask 1-3 (subtask 1-＞2-＞subtask, subtask 3) that task is carried out by the priority serial constitutes.But the embodiment of the invention is not limited to this object lesson, and the task of other any kinds all can be used the processing procedure of the embodiment of the invention similarly.Such application all falls in the scope of the embodiment of the invention.
401, network service (webservice) receives the request of executing the task that the user submits to.The logical relation of task can be by user definition.
402, the network service is transmitted to the ground floor scheduler with request.
403, the ground floor scheduler returns the response of submitting successful request to the network service.Step 403 is optional steps.
404, the ground floor scheduler is receiving of task, according to priority computing method calculating priority level and sort, selects the high task of priority.
405; The ground floor scheduler is according to selecting in the step 404 of task, starts (if system in not corresponding to such second layer scheduler) at that time or selects (have in the system available corresponding to such second layer scheduler) suitable second layer scheduler.
406, the ground floor scheduler is transmitted to task requests the second layer scheduler that in step 405, starts or select.
407, after second layer scheduler receives task requests, task is carried out pre-service according to the logical relation of task.Particularly, as a nonrestrictive example, second layer scheduler can be decomposed into a plurality of subtasks (subtask 1, subtask 2, subtask 3) with task.
408, second layer scheduler is according to " formation-worker " computation model, for subtask 1-3 creates formation 1-3.Alternatively, second layer scheduler can produce initial subtask (corresponding to subtask 1) and put it in the formation 1.At this moment, second layer scheduler can be " formation 1-＞formation 2-＞formation 3 " according to the execution sequence of the execution dependence between the 1-3 of subtask " subtask 1-＞2-＞subtask, subtask 3 " arrangement formation 1-3.
409, second layer scheduler finds in the formation 1 subtask 1 is arranged.For example, second layer scheduler can periodically be checked formation, to check whether task is arranged in the formation.But the embodiment of the invention does not limit this, and second layer scheduler can be found the subtask in the formation according to other modes.
410, second layer scheduler is subtask 1 an application resource from explorer.
411, working cell (worker) manager of the second layer scheduler indication resource of applying for starts worker and goes to handle the subtask 1 in the formation 1.
412, the worker manager starts worker, and informs that the result (corresponding to subtask 2) that worker will handle subtask 1 and obtain puts into formation 2.
After 413:worker starts, go the task that formation 1 is obtained and subtasking 1 is comprised automatically, complete after, execution result (corresponding to subtask 2) is put into formation 2.
414, second layer scheduler finds in the formation 2 subtask 2 is arranged.
415, second layer scheduler is subtask 2 application resources from explorer.
416, second layer scheduler indication working cell (worker) manager starts worker and goes to handle the subtask 2 in the formation 2.
417, the worker manager starts worker, and informs that the result (corresponding to subtask 3) that worker will handle subtask 2 and obtain puts into formation 3.
After 418:worker starts, go the task that formation 2 is obtained and subtasking 2 is comprised automatically, complete after, execution result (corresponding to subtask 3) is put into formation 3.
419, second layer scheduler finds in the formation 3 subtask 3 is arranged.
420, second layer scheduler is subtask 3 application resources from explorer.
421, second layer scheduler indication working cell (worker) manager starts worker and goes to handle the subtask 3 in the formation 3.
422, the worker manager starts worker, and informs that worker will handle the result that subtask 3 obtains and put into suitable position (as put into distributed storage device or return to the user).
423, after worker starts, go the task that formation 3 is obtained and subtasking 3 is comprised automatically, complete after, execution result is put into suitable position.
Automatically get the subtask and put the subtask through above-mentioned steps 409-423:worker.Whole like this task just can be handled according to the logical relation of task definition.In addition, carry out though among the embodiment of Fig. 4 409-423 is depicted as serial, the embodiment of the invention is not limited thereto.In other embodiments, when formation 1-3 need not to carry out in order, the execution sequence of step 409-413, step 414-418, step 419-423 might exchange or be overlapping.For example; If the subtask 2 of formation 2 does not rely on the execution result of whole subtasks 1 in the formation 1; Then when the worker of formation 1 work, the worker of formation 2 also can work, and worker that need not formation 1 executes the worker that whole subtasks 1 could start formation 2.
424, second layer scheduler can obtain the progress msg of formation or worker, judges whole task implementation progress.
425, if task is complete, then second layer scheduler reports the ground floor scheduler.Second layer scheduler also can directly supply the progress of user real time query task.
Like this; The embodiment of the invention adopts two-layer scheduling framework; Second layer scheduler is corresponding to task, and the ground floor scheduler starts or the pairing second layer scheduler of selection task, thereby goes for various tasks; Improve treatment effeciency and dispatching flexibility, and can satisfy the professional demand of multiple parallel processing.
Those of ordinary skills can recognize, the unit and the algorithm steps of each example of describing in conjunction with embodiment disclosed herein can be realized with the combination of electronic hardware or computer software and electronic hardware.These functions still are that software mode is carried out with hardware actually, depend on the application-specific and the design constraint of technical scheme.The professional and technical personnel can use distinct methods to realize described function to each certain applications, but this realization should not thought and exceeds scope of the present invention.
The those skilled in the art can be well understood to, for the convenience described with succinct, the concrete course of work of the system of foregoing description, device and unit can repeat no more at this with reference to the corresponding process among the preceding method embodiment.
In several embodiment that the application provided, should be understood that the system that is disclosed, apparatus and method can realize through other mode.For example, device embodiment described above only is schematically, for example; The division of said unit; Only be that a kind of logic function is divided, during actual the realization other dividing mode can be arranged, for example a plurality of unit or assembly can combine or can be integrated into another system; Or some characteristics can ignore, or do not carry out.Another point, the coupling each other that shows or discuss or directly coupling or communication to connect can be through some interfaces, the indirect coupling of device or unit or communication connect, and can be electrically, machinery or other form.
Said unit as separating component explanation can or can not be physically to separate also, and the parts that show as the unit can be or can not be physical locations also, promptly can be positioned at a place, perhaps also can be distributed on a plurality of NEs.Can realize the purpose of present embodiment scheme according to the needs selection some or all of unit wherein of reality.
In addition, each functional unit in each embodiment of the present invention can be integrated in the processing unit, also can be that the independent physics in each unit exists, and also can be integrated in the unit two or more unit.
If said function realizes with the form of SFU software functional unit and during as independently production marketing or use, can be stored in the computer read/write memory medium.Based on such understanding; The part that technical scheme of the present invention contributes to prior art in essence in other words or the part of this technical scheme can be come out with the embodied of software product; This computer software product is stored in the storage medium; Comprise some instructions with so that computer equipment (can be personal computer, server, the perhaps network equipment etc.) carry out all or part of step of the said method of each embodiment of the present invention.And aforesaid storage medium comprises: various media that can be program code stored such as USB flash disk, portable hard drive, ROM (read-only memory) (ROM, Read-Only Memory), RAS (RAM, Random Access Memory), magnetic disc or CD.
The above; Be merely embodiment of the present invention, but protection scope of the present invention is not limited thereto, any technician who is familiar with the present technique field is in the technical scope that the present invention discloses; Can expect easily changing or replacement, all should be encompassed within protection scope of the present invention.Therefore, protection scope of the present invention should be as the criterion by said protection domain with claim.
Claims (15)
1. a distributed computing task disposal system is characterized in that, comprising:
The ground floor scheduler is used to receive the request of executing the task, and starts or selects the corresponding second layer scheduler of said task and transmit described request to said second layer scheduler;
Second layer scheduler is used for when receiving the request that said ground floor scheduler transmits, and according to the logical relation of said task said task is decomposed into a plurality of subtasks.
2. the system of claim 1 is characterized in that, the logical relation of said task is indicated the execution dependence of said a plurality of subtasks.
3. according to claim 1 or claim 2 system is characterized in that said second layer scheduler also is used to said a plurality of subtask and creates corresponding formation to store the task that said subtask comprises.
4. system as claimed in claim 3; It is characterized in that; When in said formation, having stored the task that said subtask comprises; Said second layer scheduler also is used to said subtask application resource, and the manager startup working cell, working cell of the indication resource of applying for, so that said working cell obtains task that said subtask comprises to execute the task from said formation.
5. system as claimed in claim 4 is characterized in that, the result that said second layer scheduler also is used for indicating said working cell to execute the task puts into another formation or exports the said result who executes the task.
6. like claim 4 or 5 described systems, it is characterized in that said second layer scheduler also is used to obtain the progress msg of said formation and said working cell, to confirm said task executions progress.
7. like each described system of claim 2-6, it is characterized in that the execution dependence of said a plurality of subtasks comprises: two or more subtasks in said a plurality of subtasks are carried out according to serial or parallel order.
8. like each described system of claim 1-7, it is characterized in that said ground floor scheduler also is used for said task is carried out priority management, and start or select said second layer scheduler that said task is handled according to said priority.
9. a distributed computing task disposal route is characterized in that, said method comprises:
The ground floor scheduler starts or selects the second layer scheduler of said task correspondence when receiving the request of executing the task;
Said ground floor scheduler is transmitted described request to said second layer scheduler;
Said second layer scheduler is decomposed into a plurality of subtasks according to the logical relation of said task with said task when receiving the request that said ground floor scheduler transmits.
10. method as claimed in claim 9 is characterized in that, the logical relation of said task is indicated the execution dependence of said a plurality of subtasks.
11. like claim 9 or 10 described methods, it is characterized in that said method also comprises: said second layer scheduler is that corresponding formation is created to store the task that said subtask comprises in said a plurality of subtask.
12. method as claimed in claim 11; It is characterized in that; Said method also comprises: when said second layer scheduler has been stored the task that said subtask comprises in said formation; Be said subtask application resource, and the manager startup working cell, working cell of the indication resource of applying for, so that said working cell obtains task that said subtask comprises to execute the task from said formation.
13. method as claimed in claim 12 is characterized in that, said method also comprises: the result that said second layer scheduler indicates said working cell to execute the task puts into another formation or exports the said result who executes the task.
14., it is characterized in that also comprise: said second layer scheduler obtains the progress msg of said formation and said working cell like claim 12 or 13 described methods, to confirm said task executions progress.
15., it is characterized in that said ground floor scheduler also carries out priority management to said task like each described method of claim 9-14, and start or select said second layer scheduler that said task is handled according to said priority.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2012/070551 WO2013107012A1 (en) | 2012-01-18 | 2012-01-18 | Task processing system and task processing method for distributed computation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102763086A true CN102763086A (en) | 2012-10-31 |
Family
ID=47056377
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2012800001658A Pending CN102763086A (en) | 2012-01-18 | 2012-01-18 | Task processing system for distributed computation and task processing method for distributed computation |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN102763086A (en) |
| WO (1) | WO2013107012A1 (en) |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103064736A (en) * | 2012-12-06 | 2013-04-24 | 华为技术有限公司 | Device and method for task processing |
| CN104035817A (en) * | 2014-07-08 | 2014-09-10 | 领佰思自动化科技(上海)有限公司 | Distributed parallel computing method and system for physical implementation of large scale integrated circuit |
| CN104102949A (en) * | 2014-06-27 | 2014-10-15 | 北京奇艺世纪科技有限公司 | Distributed workflow device and method for processing workflow by distributed workflow device |
| CN104123182A (en) * | 2014-07-18 | 2014-10-29 | 西安交通大学 | Map Reduce task data-center-across scheduling system and method based on master-slave framework |
| CN104778074A (en) * | 2014-01-14 | 2015-07-15 | 腾讯科技（深圳）有限公司 | Calculation task processing method and device |
| CN105653365A (en) * | 2016-02-22 | 2016-06-08 | 青岛海尔智能家电科技有限公司 | Task processing method and device |
| CN106445681A (en) * | 2016-08-31 | 2017-02-22 | 东方网力科技股份有限公司 | Distributed task scheduling system and method |
| CN106547523A (en) * | 2015-09-17 | 2017-03-29 | 北大方正集团有限公司 | Progress bar progress display packing, apparatus and system |
| CN103870334B (en) * | 2012-12-18 | 2017-05-31 | 中国移动通信集团公司 | A kind of method for allocating tasks and device of extensive vulnerability scanning |
| US9886310B2 (en) | 2014-02-10 | 2018-02-06 | International Business Machines Corporation | Dynamic resource allocation in MapReduce |
| CN107710173A (en) * | 2015-05-29 | 2018-02-16 | 高通股份有限公司 | Multithreading conversion and affairs rearrangement for MMU |
| CN107818016A (en) * | 2017-11-22 | 2018-03-20 | 苏州麦迪斯顿医疗科技股份有限公司 | Server application design method, request event processing method and processing device |
| WO2019051942A1 (en) * | 2017-09-15 | 2019-03-21 | 平安科技（深圳）有限公司 | Task allocation method, terminal, and computer readable storage medium |
| CN109885388A (en) * | 2019-01-31 | 2019-06-14 | 上海赜睿信息科技有限公司 | A kind of data processing method and device suitable for heterogeneous system |
| CN110569252A (en) * | 2018-05-16 | 2019-12-13 | 杭州海康威视数字技术股份有限公司 | Data processing system and method |
| CN110597613A (en) * | 2018-06-12 | 2019-12-20 | 成都鼎桥通信技术有限公司 | Task processing method, device, equipment and computer readable storage medium |
| CN110750371A (en) * | 2019-10-17 | 2020-02-04 | 北京创鑫旅程网络技术有限公司 | Flow execution method, device, equipment and storage medium |
| WO2020108303A1 (en) * | 2018-11-30 | 2020-06-04 | 中兴通讯股份有限公司 | Heterogeneous computing-based task processing method and software-hardware framework system |
| CN111506409A (en) * | 2020-04-20 | 2020-08-07 | 南方电网科学研究院有限责任公司 | Data processing method and system |
| CN111708643A (en) * | 2020-06-11 | 2020-09-25 | 中国工商银行股份有限公司 | Batch operation method and device for distributed streaming media platform |
| CN112596871A (en) * | 2020-12-16 | 2021-04-02 | 中国建设银行股份有限公司 | Service processing method and device |
| CN113448692A (en) * | 2020-03-25 | 2021-09-28 | 杭州海康威视数字技术股份有限公司 | Distributed graph computing method, device, equipment and storage medium |
| CN115268800A (en) * | 2022-09-29 | 2022-11-01 | 四川汉唐云分布式存储技术有限公司 | Data processing method and data storage system based on calculation route redirection |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1791025A (en) * | 2005-12-26 | 2006-06-21 | 北京航空航天大学 | Service gridding system and method for processing operation |
| CN101169743A (en) * | 2007-11-27 | 2008-04-30 | 南京大学 | Method for implementing parallel power flow calculation based on multi-core computer in electric grid |
| CN101957780A (en) * | 2010-08-17 | 2011-01-26 | 中国电子科技集团公司第二十八研究所 | Resource state information-based grid task scheduling processor and grid task scheduling processing method |
| CN102110022A (en) * | 2011-03-22 | 2011-06-29 | 上海交通大学 | Sensor network embedded operation system based on priority scheduling |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7876763B2 (en) * | 2004-08-05 | 2011-01-25 | Cisco Technology, Inc. | Pipeline scheduler including a hierarchy of schedulers and multiple scheduling lanes |
| CN101621460B (en) * | 2008-06-30 | 2011-11-30 | 中兴通讯股份有限公司 | Packet scheduling method and device |
| CN102185761B (en) * | 2011-04-13 | 2013-08-07 | 中国人民解放军国防科学技术大学 | Two-layer dynamic scheduling method facing to ensemble prediction applications |
-
2012
- 2012-01-18 CN CN2012800001658A patent/CN102763086A/en active Pending
- 2012-01-18 WO PCT/CN2012/070551 patent/WO2013107012A1/en active Application Filing
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1791025A (en) * | 2005-12-26 | 2006-06-21 | 北京航空航天大学 | Service gridding system and method for processing operation |
| CN101169743A (en) * | 2007-11-27 | 2008-04-30 | 南京大学 | Method for implementing parallel power flow calculation based on multi-core computer in electric grid |
| CN101957780A (en) * | 2010-08-17 | 2011-01-26 | 中国电子科技集团公司第二十八研究所 | Resource state information-based grid task scheduling processor and grid task scheduling processing method |
| CN102110022A (en) * | 2011-03-22 | 2011-06-29 | 上海交通大学 | Sensor network embedded operation system based on priority scheduling |
Cited By (35)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9519338B2 (en) | 2012-12-06 | 2016-12-13 | Huawei Technologies Co., Ltd. | Task processing apparatus and method including scheduling current and next-level task processing apparatus |
| CN103064736A (en) * | 2012-12-06 | 2013-04-24 | 华为技术有限公司 | Device and method for task processing |
| CN103870334B (en) * | 2012-12-18 | 2017-05-31 | 中国移动通信集团公司 | A kind of method for allocating tasks and device of extensive vulnerability scanning |
| US10146588B2 (en) | 2014-01-14 | 2018-12-04 | Tencent Technology (Shenzhen) Company Limited | Method and apparatus for processing computational task having multiple subflows |
| WO2015106687A1 (en) * | 2014-01-14 | 2015-07-23 | Tencent Technology (Shenzhen) Company Limited | Method and apparatus for processing computational task |
| CN104778074A (en) * | 2014-01-14 | 2015-07-15 | 腾讯科技（深圳）有限公司 | Calculation task processing method and device |
| CN104778074B (en) * | 2014-01-14 | 2019-02-26 | 腾讯科技（深圳）有限公司 | A kind of calculating task processing method and processing device |
| US9886310B2 (en) | 2014-02-10 | 2018-02-06 | International Business Machines Corporation | Dynamic resource allocation in MapReduce |
| CN104102949A (en) * | 2014-06-27 | 2014-10-15 | 北京奇艺世纪科技有限公司 | Distributed workflow device and method for processing workflow by distributed workflow device |
| CN104102949B (en) * | 2014-06-27 | 2018-01-26 | 北京奇艺世纪科技有限公司 | A kind of distributed work flow device and its method for handling workflow |
| CN104035817A (en) * | 2014-07-08 | 2014-09-10 | 领佰思自动化科技(上海)有限公司 | Distributed parallel computing method and system for physical implementation of large scale integrated circuit |
| CN104123182A (en) * | 2014-07-18 | 2014-10-29 | 西安交通大学 | Map Reduce task data-center-across scheduling system and method based on master-slave framework |
| CN104123182B (en) * | 2014-07-18 | 2015-09-30 | 西安交通大学 | Based on the MapReduce task of client/server across data center scheduling system and method |
| CN107710173A (en) * | 2015-05-29 | 2018-02-16 | 高通股份有限公司 | Multithreading conversion and affairs rearrangement for MMU |
| CN106547523A (en) * | 2015-09-17 | 2017-03-29 | 北大方正集团有限公司 | Progress bar progress display packing, apparatus and system |
| CN106547523B (en) * | 2015-09-17 | 2019-08-06 | 北大方正集团有限公司 | Progress bar progress display methods, apparatus and system |
| CN105653365A (en) * | 2016-02-22 | 2016-06-08 | 青岛海尔智能家电科技有限公司 | Task processing method and device |
| CN106445681A (en) * | 2016-08-31 | 2017-02-22 | 东方网力科技股份有限公司 | Distributed task scheduling system and method |
| CN106445681B (en) * | 2016-08-31 | 2019-11-29 | 东方网力科技股份有限公司 | Distributed task dispatching system and method |
| WO2019051942A1 (en) * | 2017-09-15 | 2019-03-21 | 平安科技（深圳）有限公司 | Task allocation method, terminal, and computer readable storage medium |
| CN107818016A (en) * | 2017-11-22 | 2018-03-20 | 苏州麦迪斯顿医疗科技股份有限公司 | Server application design method, request event processing method and processing device |
| CN110569252A (en) * | 2018-05-16 | 2019-12-13 | 杭州海康威视数字技术股份有限公司 | Data processing system and method |
| CN110597613A (en) * | 2018-06-12 | 2019-12-20 | 成都鼎桥通信技术有限公司 | Task processing method, device, equipment and computer readable storage medium |
| WO2020108303A1 (en) * | 2018-11-30 | 2020-06-04 | 中兴通讯股份有限公司 | Heterogeneous computing-based task processing method and software-hardware framework system |
| CN111258744A (en) * | 2018-11-30 | 2020-06-09 | 中兴通讯股份有限公司 | Task processing method based on heterogeneous computation and software and hardware framework system |
| EP3889774A4 (en) * | 2018-11-30 | 2022-08-03 | ZTE Corporation | Heterogeneous computing-based task processing method and software-hardware framework system |
| US11681564B2 (en) | 2018-11-30 | 2023-06-20 | Zte Corporation | Heterogeneous computing-based task processing method and software and hardware framework system |
| CN109885388A (en) * | 2019-01-31 | 2019-06-14 | 上海赜睿信息科技有限公司 | A kind of data processing method and device suitable for heterogeneous system |
| CN110750371A (en) * | 2019-10-17 | 2020-02-04 | 北京创鑫旅程网络技术有限公司 | Flow execution method, device, equipment and storage medium |
| CN113448692A (en) * | 2020-03-25 | 2021-09-28 | 杭州海康威视数字技术股份有限公司 | Distributed graph computing method, device, equipment and storage medium |
| CN111506409A (en) * | 2020-04-20 | 2020-08-07 | 南方电网科学研究院有限责任公司 | Data processing method and system |
| CN111708643A (en) * | 2020-06-11 | 2020-09-25 | 中国工商银行股份有限公司 | Batch operation method and device for distributed streaming media platform |
| CN112596871A (en) * | 2020-12-16 | 2021-04-02 | 中国建设银行股份有限公司 | Service processing method and device |
| CN115268800A (en) * | 2022-09-29 | 2022-11-01 | 四川汉唐云分布式存储技术有限公司 | Data processing method and data storage system based on calculation route redirection |
| CN115268800B (en) * | 2022-09-29 | 2022-12-20 | 四川汉唐云分布式存储技术有限公司 | Data processing method and data storage system based on calculation route redirection |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2013107012A1 (en) | 2013-07-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102763086A (en) | Task processing system for distributed computation and task processing method for distributed computation | |
| CN107038069B (en) | Dynamic label matching DLMS scheduling method under Hadoop platform | |
| US9785472B2 (en) | Computing cluster performance simulation using a genetic algorithm solution | |
| US8381212B2 (en) | Dynamic allocation and partitioning of compute nodes in hierarchical job scheduling | |
| KR101893982B1 (en) | Distributed processing system, scheduler node and scheduling method of distributed processing system, and apparatus for generating program thereof | |
| CN103309946B (en) | Multimedia file processing method, Apparatus and system | |
| CN109992407B (en) | YARN cluster GPU resource scheduling method, device and medium | |
| US10505791B2 (en) | System and method to handle events using historical data in serverless systems | |
| US20110067030A1 (en) | Flow based scheduling | |
| US8903981B2 (en) | Method and system for achieving better efficiency in a client grid using node resource usage and tracking | |
| CN104239144A (en) | Multilevel distributed task processing system | |
| US11507419B2 (en) | Method,electronic device and computer program product for scheduling computer resources in a task processing environment | |
| CN103197976A (en) | Method and device for processing tasks of heterogeneous system | |
| CN107678752A (en) | A kind of task processing method and device towards isomeric group | |
| CN103377075A (en) | Task management method, device and system | |
| US20170220944A1 (en) | Best trade-off point on an elbow curve for optimal resource provisioning and performance efficiency | |
| CN102541460A (en) | Multiple disc management method and equipment | |
| CN104735095A (en) | Method and device for job scheduling of cloud computing platform | |
| CN101510163A (en) | Processing method and system for planning task | |
| KR20160099762A (en) | Cloud System for supporting auto-scaled Hadoop Distributed Parallel Processing System | |
| Delavar et al. | A synthetic heuristic algorithm for independent task scheduling in cloud systems | |
| CN105808346A (en) | Task scheduling method and device | |
| Kalayci et al. | Distributed and Adaptive Execution of Condor DAGMan Workflows. | |
| CN103617083A (en) | Storage scheduling method and system, job scheduling method and system and management node | |
| Mansouri et al. | Weighted data replication strategy for data grid considering economic approach |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20121031 |