CN102316156A - Method for distributing and processing dynamically extensible task - Google Patents

Method for distributing and processing dynamically extensible task Download PDF

Info

Publication number
CN102316156A
CN102316156A CN201110186624A CN201110186624A CN102316156A CN 102316156 A CN102316156 A CN 102316156A CN 201110186624 A CN201110186624 A CN 201110186624A CN 201110186624 A CN201110186624 A CN 201110186624A CN 102316156 A CN102316156 A CN 102316156A
Authority
CN
China
Prior art keywords
task
node
controlled node
distribution
working
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201110186624A
Other languages
Chinese (zh)
Inventor
陈诚
高尚
李剑华
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
WANDA INFORMATION CO Ltd
Original Assignee
WANDA INFORMATION CO Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by WANDA INFORMATION CO Ltd filed Critical WANDA INFORMATION CO Ltd
Priority to CN201110186624A priority Critical patent/CN102316156A/en
Publication of CN102316156A publication Critical patent/CN102316156A/en
Pending legal-status Critical Current

Links

Images

Landscapes

  • Multi Processors (AREA)

Abstract

The invention provides a method for distributing and processing a dynamically extensible task. The method is characterized by comprising the following steps of: evaluating the task; judging whether the task is required to be processed locally or processed in cooperation with a network; if the task is required to be processed in cooperation with the network, binding a group of tasks of the same type with possible working nodes according to the real-time deployment and load state of each working node in a distributed environment; mounting each task into the working nodes respectively according to the sequence of each task in the group of tasks; processing after mounting; and summarizing a processing result after processing. The invention provides the high-performance, highly available and highly reliable task distributing and processing method. Specific to complex logic processing and concurrent access, the effective distribution of the task on a single machine in a distributed environment is ensured.

Description

But a kind of task distribution processing method of dynamic expansion
Technical field
The present invention relates to a kind of distribution and processing method of computer task.
Background technology
The multithreading that in conventional use, is widely used is realized the concurrent solution module performance of software problem.In the Java field; Successively in JDK5.0 (2006) and JDK7.0 (2010), provide Java ThreadPool and Fork/Join framework to realize; Wherein the latter is especially towards the high efficiency solution of polycaryon processor, but they all can only be operated in single JVM environment at present.Constantly complicated along with what use; The cloud computing tide constantly starts; With Google is the Internet enterprise of representative, has released towards the model of the Map/Reduce of Distributed Calculation, and this model has solved the handling problem under the mass data distributed store situation; But it is data-centered, still is not suitable for the calculating of Logic Priority.
Summary of the invention
The purpose of this invention is to provide a kind of method that guarantees that task is effectively distributed and handled under unit and distributed environment.
In order to achieve the above object, but technical scheme of the present invention has provided a kind of task distribution processing method of dynamic expansion, it is characterized in that, step is:
Step 1, assessment: to the type and the characteristic of task, submitting to of task is distributed assessment,, then get into waiting list,, then get into step 2 if need network cooperating to accomplish if this task only need be handled in this locality;
Step 2, distribution: real-time deployment and load state according to each working node in the distributed environment are bound a group task of same type and possible working node, and the precedence according to each task in this group task is mounted to each task in the working node respectively again;
Step 3, processing: working node is carried out each task respectively according to precedence after receiving a group task of certain type, returns execution result after being finished;
Step 4, merger: all execution results that return are gathered merging according to task type, and merging after will carrying out after finishing of task is returned.
The invention provides a kind of high-performance, high available, highly reliable task distribution and processing method, handle and concurrent visit the effective distribution of assurance task under unit and distributed environment to complex logic.
Description of drawings
Fig. 1 is a task distribute work schematic diagram;
Fig. 2 is task status transition figure.
Embodiment
Specify the present invention below in conjunction with embodiment.
Embodiment
But the invention provides a kind of task distribution processing method of dynamic expansion, step is:
Step 1, assessment (evaluate): to the type (taskType) and the characteristic (corresponding List < Task >) of task, submitting to of task is distributed assessment, return LocalTaskList and DistributeTaskList; The LocalTaskList representative needs the task of local runtime; The task that the DistributeTaskList representative needs network cooperating to accomplish if this task only need be handled in this locality, then gets into waiting list; If need network cooperating to accomplish, then get into step 2;
Step 2, distribution (distribute): distribute are divided into two sub-phase: level distribution and vertical distribution.For horizontal distribution procedure; Main controlled node can be accomplished the relevant work node before this and comprise the registration (through teleinstruction) of module; After receiving task, the type taskType of main controlled node retrieval tasks, and will have a group task of same type and possible working node is bound according to real-time deployment and load state; And then task is delivered to corresponding working node through network; After working node receives task, then be written into corresponding computing module, in the Context of this module, implement task scheduling according to task type.This process acquiescence is to accomplish automatically, intervenes but the developer can be provided with filtration to the binding of task; In carry working node process, can start vertical distribution, the parameter that needs to handle is taskType; The developer can carry out cutting to current task, forms a plurality of parallel subtasks (childTask), comprises father's task ID in the subtask; A plurality of parallel subtasks implement to distribute once more again; A plurality of parallel childTask are mounted on the different working node, vertical distribution is recurrence successively, and needs manual intervention;
Step 3, processing (map): after working node receives a group task of certain type; Comprise subtask through vertical distribution; Carry out each task respectively according to precedence; Can introduce thread pool in the map process and improve execution efficient, return execution result FinishedTaskList after being finished;
Step 4, merger (merge): all execution results that map is returned gather merging, and the foundation of merging is taskType, import the task List that parameter is taskType and completion < Task>into; The developer can screen the result at this, revises etc., need return PrunedTaskList; In this process; If there is the subtask, then up recurrence is to the merger of root level task, and the task after will carrying out after merger finishes is returned.
Simultaneously, after task is submitted to, no matter in this locality or in distributed environment, the treatment state of tracing task that all can be real-time, and the timely termination of control task, transition between states figure is as shown in Figure 1.
Before task is finally finished dealing with, all might receive abort request, if task still is untreated, then initiatively abandon; If in the process of implementation; Can judge earlier that then can current task be ended; Engine is to stop to continue to carry out through the unusual mode of the TaskInterruptException that dishes out; Therefore need guarantee in force that writing of service logic can effectively catch this unusual back transaction rollback, otherwise this task just can not be ended.In addition, for the task of comprising the subtask, wouldn't support effectively to end service.
When task execution generation is unusual, then transfer the ERROR state to, from operation queue, shift out, and carry out the compensating operation of definition.If the subtask operation exception then can be notified host node earlier, successively to notify up to the root task, same Status Change is ERROR, and shifts out from operation queue, carries out compensating operation.
Message adopts the TCP agreement, and message format is: packet header+inclusion.Packet header is totally 4 bytes.Formal definition is: 0 to 1 byte is a control command, and the 2nd to 3 byte is inclusion length (not containing packet header).The inclusion content is the serializing bytecode of task instances.Control command comprises: 0xF1: the submission task; 0xF2: query State; 0xF3: aborting task; 0xA1: intercommunication instruction, registration new module; 0xA2: intercommunication instruction, Unload module.
In home environment; Adopt the stand-alone pattern, in distributed environment, adopt main controlled node-controlled node mode (Master-worker pattern), the working node of accomplishing assessment, distribution and handle is arranged as main controlled node end (master end); The working node of accomplishing merger is arranged as controlled node side (worker end); Master end be control side, center, and it is cooperation part that worker holds, and the deployment configuration through separately between master and worker hold can intercom mutually; The master end has been safeguarded overall work distribution chart, and the worker end can in time be notified master end with error condition.Actuating logic is deployed in the worker end, and the master end need not be disposed.Support hot deployment way; Be similar to the WebContext loading procedure, actuating logic is uploaded to the worker end with the Jar packet form, and the worker end loads automatically; And read context.xml and go into running environment, be that the message notifying master end of instruction is registered simultaneously with 0xA1.Actuating logic can according to circumstances be selected part or all of worker end, can automatic phasing pass task optionally distribute during the task distribution.

Claims (5)

1. but the task distribution processing method of a dynamic expansion is characterized in that, step is:
Step 1, assessment: to the type and the characteristic of task, submitting to of task is distributed assessment,, then get into waiting list,, then get into step 2 if need network cooperating to accomplish if this task only need be handled in this locality;
Step 2, distribution: real-time deployment and load state according to each working node in the distributed environment are bound a group task of same type and possible working node, and the precedence according to each task in this group task is mounted to each task in the working node respectively again;
Step 3, processing: working node is carried out each task respectively according to precedence after receiving a group task of certain type, returns execution result after being finished;
Step 4, merger: all execution results that return are gathered merging according to task type, and merging after will carrying out after finishing of task is returned.
2. but the task distribution processing method of a kind of dynamic expansion as claimed in claim 1; It is characterized in that: adopt the method for vertical distribution when in step 2, being mounted to task in the working node, promptly current task is carried out cutting, form a plurality of parallel subtasks; Current task is called father's task; The ID that comprises father's task in the subtask is mounted to a plurality of parallel subtasks in the different sub-working nodes, in step 4, carries out gathering when merging of execution result; If there is a subtask, then up the root level father task of recurrence to this subtask gather merging.
3. but the task distribution processing method of a kind of dynamic expansion as claimed in claim 1 is characterized in that: after task is submitted to, the state of task is implemented inspection.
4. but the task distribution processing method of a kind of dynamic expansion as claimed in claim 1 is characterized in that: before task is finished, if carry out following judgement when receiving abort request: if task still is untreated, then initiatively abandon; If in the process of implementation, judge earlier that then can current task be ended, if can be ended, then the mode through the abnormal signal of dishing out stops to continue to carry out.
5. but the task distribution processing method of a kind of dynamic expansion as claimed in claim 1; It is characterized in that: in distributed environment, adopt main controlled node-controlled node mode; The working node of accomplishing assessment, distribution and handle is arranged as the main controlled node end, the working node of accomplishing merger is arranged as controlled node side, the main controlled node end is control side, center; Controlled node side is a cooperation part; Deployment configuration through separately between main controlled node and the controlled node can intercom mutually, and the main controlled node end has been safeguarded overall work distribution chart, and controlled node side can in time be notified the main controlled node end with error condition.
CN201110186624A 2011-07-05 2011-07-05 Method for distributing and processing dynamically extensible task Pending CN102316156A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201110186624A CN102316156A (en) 2011-07-05 2011-07-05 Method for distributing and processing dynamically extensible task

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201110186624A CN102316156A (en) 2011-07-05 2011-07-05 Method for distributing and processing dynamically extensible task

Publications (1)

Publication Number Publication Date
CN102316156A true CN102316156A (en) 2012-01-11

Family

ID=45428963

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201110186624A Pending CN102316156A (en) 2011-07-05 2011-07-05 Method for distributing and processing dynamically extensible task

Country Status (1)

Country Link
CN (1) CN102316156A (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105573850A (en) * 2015-11-09 2016-05-11 广州多益网络科技有限公司 Multi-process interactive method and system and server
CN105729491A (en) * 2016-04-01 2016-07-06 纳恩博(北京)科技有限公司 Executing method, device and system for robot task
CN106874109A (en) * 2016-12-29 2017-06-20 朗新科技股份有限公司 A kind of distributed job distribution processing method and system
US9787616B2 (en) 2012-08-22 2017-10-10 Nokia Technologies Oy Method and apparatus for exchanging status updates while collaborating
CN107832150A (en) * 2017-11-07 2018-03-23 清华大学 A kind of calculating task dynamic partition strategy
CN109101308A (en) * 2018-07-20 2018-12-28 广州农村商业银行股份有限公司 A kind of transmitting of task and method for tracking and displaying and device
CN109471707A (en) * 2018-10-12 2019-03-15 传化智联股份有限公司 The dispositions method and device of scheduler task
CN110348656A (en) * 2018-04-08 2019-10-18 天津京东深拓机器人科技有限公司 Dispatching method, device, system and computer readable storage medium
CN110968406A (en) * 2018-09-30 2020-04-07 北京国双科技有限公司 Method, device, storage medium and processor for processing task

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101183368A (en) * 2007-12-06 2008-05-21 华南理工大学 Method and system for distributed calculating and enquiring magnanimity data in on-line analysis processing
US20080126859A1 (en) * 2006-08-31 2008-05-29 Guo Shang Q Methods and arrangements for distributed diagnosis in distributed systems using belief propagation
CN101727389A (en) * 2009-11-23 2010-06-09 中兴通讯股份有限公司 Automatic test system and method of distributed integrated service
CN101950300A (en) * 2010-09-20 2011-01-19 华南理工大学 Hierarchical structure, distributed search engine system and implementation method thereof

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20080126859A1 (en) * 2006-08-31 2008-05-29 Guo Shang Q Methods and arrangements for distributed diagnosis in distributed systems using belief propagation
CN101183368A (en) * 2007-12-06 2008-05-21 华南理工大学 Method and system for distributed calculating and enquiring magnanimity data in on-line analysis processing
CN101727389A (en) * 2009-11-23 2010-06-09 中兴通讯股份有限公司 Automatic test system and method of distributed integrated service
CN101950300A (en) * 2010-09-20 2011-01-19 华南理工大学 Hierarchical structure, distributed search engine system and implementation method thereof

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9787616B2 (en) 2012-08-22 2017-10-10 Nokia Technologies Oy Method and apparatus for exchanging status updates while collaborating
CN105573850A (en) * 2015-11-09 2016-05-11 广州多益网络科技有限公司 Multi-process interactive method and system and server
CN105573850B (en) * 2015-11-09 2018-12-28 广州多益网络股份有限公司 Multi-process exchange method, system and server
CN105729491A (en) * 2016-04-01 2016-07-06 纳恩博(北京)科技有限公司 Executing method, device and system for robot task
CN105729491B (en) * 2016-04-01 2018-09-04 纳恩博(北京)科技有限公司 The execution method, apparatus and system of robot task
CN106874109A (en) * 2016-12-29 2017-06-20 朗新科技股份有限公司 A kind of distributed job distribution processing method and system
CN107832150A (en) * 2017-11-07 2018-03-23 清华大学 A kind of calculating task dynamic partition strategy
CN110348656A (en) * 2018-04-08 2019-10-18 天津京东深拓机器人科技有限公司 Dispatching method, device, system and computer readable storage medium
CN110348656B (en) * 2018-04-08 2022-09-30 北京京东乾石科技有限公司 Scheduling method, device, system and computer readable storage medium
CN109101308A (en) * 2018-07-20 2018-12-28 广州农村商业银行股份有限公司 A kind of transmitting of task and method for tracking and displaying and device
CN110968406A (en) * 2018-09-30 2020-04-07 北京国双科技有限公司 Method, device, storage medium and processor for processing task
CN109471707A (en) * 2018-10-12 2019-03-15 传化智联股份有限公司 The dispositions method and device of scheduler task

Similar Documents

Publication Publication Date Title
CN102316156A (en) Method for distributing and processing dynamically extensible task
CN100578459C (en) Method and apparatus of thread scheduling
US20200310963A1 (en) Real-time replicating garbage collection
CN103399856A (en) Explosive type data caching and processing system for SCADA system and method thereof
CN101464810A (en) Service program processing method and server
CN107077390A (en) A kind of task processing method and network interface card
CN108881485A (en) The method for ensureing the high concurrent system response time under big data packet
CN103902593A (en) Data transfer method and device
US20200409841A1 (en) Multi-threaded pause-less replicating garbage collection
CN106776395B (en) A kind of method for scheduling task and device of shared cluster
CN101719153A (en) Device and method for carrying out transmission processing on massive data files
US20230266999A1 (en) Resource scheduling method, resource scheduling system, and device
CN111124648B (en) Multitasking method and device
CN103902355A (en) Quick medical image loading method
CN111208992A (en) System scheduling workflow generation method and system
CN109144749A (en) A method of it is communicated between realizing multiprocessor using processor
CN112114983B (en) Communication method, device and equipment based on shared memory
CN117435324B (en) Task scheduling method based on containerization
CN111309494A (en) Multithreading event processing assembly
CN109710679A (en) Data pick-up method and device
CN116069480B (en) Processor and computing device
CN201590836U (en) Device for transmitting and processing massive data files
CN102195361A (en) Method for acquiring and processing data of intelligent distribution terminal of multi-core single chip
CN106997304B (en) Input and output event processing method and device
Fabra et al. A framework for the flexible deployment of scientific workflows in grid environments

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C02 Deemed withdrawal of patent application after publication (patent law 2001)
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20120111