CN102316156A

CN102316156A - Method for distributing and processing dynamically extensible task

Info

Publication number: CN102316156A
Application number: CN201110186624A
Authority: CN
Inventors: 陈诚; 高尚; 李剑华
Original assignee: WANDA INFORMATION CO Ltd
Current assignee: WANDA INFORMATION CO Ltd
Priority date: 2011-07-05
Filing date: 2011-07-05
Publication date: 2012-01-11

Abstract

The invention provides a method for distributing and processing a dynamically extensible task. The method is characterized by comprising the following steps of: evaluating the task; judging whether the task is required to be processed locally or processed in cooperation with a network; if the task is required to be processed in cooperation with the network, binding a group of tasks of the same type with possible working nodes according to the real-time deployment and load state of each working node in a distributed environment; mounting each task into the working nodes respectively according to the sequence of each task in the group of tasks; processing after mounting; and summarizing a processing result after processing. The invention provides the high-performance, highly available and highly reliable task distributing and processing method. Specific to complex logic processing and concurrent access, the effective distribution of the task on a single machine in a distributed environment is ensured.

Description

But a kind of task distribution processing method of dynamic expansion

Technical field

The present invention relates to a kind of distribution and processing method of computer task.

Background technology

The multithreading that in conventional use, is widely used is realized the concurrent solution module performance of software problem.In the Java field; Successively in JDK5.0 (2006) and JDK7.0 (2010), provide Java ThreadPool and Fork/Join framework to realize; Wherein the latter is especially towards the high efficiency solution of polycaryon processor, but they all can only be operated in single JVM environment at present.Constantly complicated along with what use; The cloud computing tide constantly starts; With Google is the Internet enterprise of representative, has released towards the model of the Map/Reduce of Distributed Calculation, and this model has solved the handling problem under the mass data distributed store situation; But it is data-centered, still is not suitable for the calculating of Logic Priority.

Summary of the invention

The purpose of this invention is to provide a kind of method that guarantees that task is effectively distributed and handled under unit and distributed environment.

In order to achieve the above object, but technical scheme of the present invention has provided a kind of task distribution processing method of dynamic expansion, it is characterized in that, step is:

Step 1, assessment: to the type and the characteristic of task, submitting to of task is distributed assessment,, then get into waiting list,, then get into step 2 if need network cooperating to accomplish if this task only need be handled in this locality;

Step 2, distribution: real-time deployment and load state according to each working node in the distributed environment are bound a group task of same type and possible working node, and the precedence according to each task in this group task is mounted to each task in the working node respectively again;

Step 3, processing: working node is carried out each task respectively according to precedence after receiving a group task of certain type, returns execution result after being finished;

Step 4, merger: all execution results that return are gathered merging according to task type, and merging after will carrying out after finishing of task is returned.

The invention provides a kind of high-performance, high available, highly reliable task distribution and processing method, handle and concurrent visit the effective distribution of assurance task under unit and distributed environment to complex logic.

Description of drawings

Fig. 1 is a task distribute work schematic diagram;

Fig. 2 is task status transition figure.

Embodiment

Specify the present invention below in conjunction with embodiment.

Embodiment

But the invention provides a kind of task distribution processing method of dynamic expansion, step is:

Step 1, assessment (evaluate): to the type (taskType) and the characteristic (corresponding List < Task >) of task, submitting to of task is distributed assessment, return LocalTaskList and DistributeTaskList; The LocalTaskList representative needs the task of local runtime; The task that the DistributeTaskList representative needs network cooperating to accomplish if this task only need be handled in this locality, then gets into waiting list; If need network cooperating to accomplish, then get into step 2;

Step 2, distribution (distribute): distribute are divided into two sub-phase: level distribution and vertical distribution.For horizontal distribution procedure; Main controlled node can be accomplished the relevant work node before this and comprise the registration (through teleinstruction) of module; After receiving task, the type taskType of main controlled node retrieval tasks, and will have a group task of same type and possible working node is bound according to real-time deployment and load state; And then task is delivered to corresponding working node through network; After working node receives task, then be written into corresponding computing module, in the Context of this module, implement task scheduling according to task type.This process acquiescence is to accomplish automatically, intervenes but the developer can be provided with filtration to the binding of task; In carry working node process, can start vertical distribution, the parameter that needs to handle is taskType; The developer can carry out cutting to current task, forms a plurality of parallel subtasks (childTask), comprises father's task ID in the subtask; A plurality of parallel subtasks implement to distribute once more again; A plurality of parallel childTask are mounted on the different working node, vertical distribution is recurrence successively, and needs manual intervention;

Step 3, processing (map): after working node receives a group task of certain type; Comprise subtask through vertical distribution; Carry out each task respectively according to precedence; Can introduce thread pool in the map process and improve execution efficient, return execution result FinishedTaskList after being finished;

Step 4, merger (merge): all execution results that map is returned gather merging, and the foundation of merging is taskType, import the task List that parameter is taskType and completion < Task>into; The developer can screen the result at this, revises etc., need return PrunedTaskList; In this process; If there is the subtask, then up recurrence is to the merger of root level task, and the task after will carrying out after merger finishes is returned.

Simultaneously, after task is submitted to, no matter in this locality or in distributed environment, the treatment state of tracing task that all can be real-time, and the timely termination of control task, transition between states figure is as shown in Figure 1.

Before task is finally finished dealing with, all might receive abort request, if task still is untreated, then initiatively abandon; If in the process of implementation; Can judge earlier that then can current task be ended; Engine is to stop to continue to carry out through the unusual mode of the TaskInterruptException that dishes out; Therefore need guarantee in force that writing of service logic can effectively catch this unusual back transaction rollback, otherwise this task just can not be ended.In addition, for the task of comprising the subtask, wouldn't support effectively to end service.

When task execution generation is unusual, then transfer the ERROR state to, from operation queue, shift out, and carry out the compensating operation of definition.If the subtask operation exception then can be notified host node earlier, successively to notify up to the root task, same Status Change is ERROR, and shifts out from operation queue, carries out compensating operation.

Message adopts the TCP agreement, and message format is: packet header+inclusion.Packet header is totally 4 bytes.Formal definition is: 0 to 1 byte is a control command, and the 2nd to 3 byte is inclusion length (not containing packet header).The inclusion content is the serializing bytecode of task instances.Control command comprises: 0xF1: the submission task; 0xF2: query State; 0xF3: aborting task; 0xA1: intercommunication instruction, registration new module; 0xA2: intercommunication instruction, Unload module.

In home environment; Adopt the stand-alone pattern, in distributed environment, adopt main controlled node-controlled node mode (Master-worker pattern), the working node of accomplishing assessment, distribution and handle is arranged as main controlled node end (master end); The working node of accomplishing merger is arranged as controlled node side (worker end); Master end be control side, center, and it is cooperation part that worker holds, and the deployment configuration through separately between master and worker hold can intercom mutually; The master end has been safeguarded overall work distribution chart, and the worker end can in time be notified master end with error condition.Actuating logic is deployed in the worker end, and the master end need not be disposed.Support hot deployment way; Be similar to the WebContext loading procedure, actuating logic is uploaded to the worker end with the Jar packet form, and the worker end loads automatically; And read context.xml and go into running environment, be that the message notifying master end of instruction is registered simultaneously with 0xA1.Actuating logic can according to circumstances be selected part or all of worker end, can automatic phasing pass task optionally distribute during the task distribution.

Claims

1. but the task distribution processing method of a dynamic expansion is characterized in that, step is:

2. but the task distribution processing method of a kind of dynamic expansion as claimed in claim 1; It is characterized in that: adopt the method for vertical distribution when in step 2, being mounted to task in the working node, promptly current task is carried out cutting, form a plurality of parallel subtasks; Current task is called father's task; The ID that comprises father's task in the subtask is mounted to a plurality of parallel subtasks in the different sub-working nodes, in step 4, carries out gathering when merging of execution result; If there is a subtask, then up the root level father task of recurrence to this subtask gather merging.

3. but the task distribution processing method of a kind of dynamic expansion as claimed in claim 1 is characterized in that: after task is submitted to, the state of task is implemented inspection.

4. but the task distribution processing method of a kind of dynamic expansion as claimed in claim 1 is characterized in that: before task is finished, if carry out following judgement when receiving abort request: if task still is untreated, then initiatively abandon; If in the process of implementation, judge earlier that then can current task be ended, if can be ended, then the mode through the abnormal signal of dishing out stops to continue to carry out.

5. but the task distribution processing method of a kind of dynamic expansion as claimed in claim 1; It is characterized in that: in distributed environment, adopt main controlled node-controlled node mode; The working node of accomplishing assessment, distribution and handle is arranged as the main controlled node end, the working node of accomplishing merger is arranged as controlled node side, the main controlled node end is control side, center; Controlled node side is a cooperation part; Deployment configuration through separately between main controlled node and the controlled node can intercom mutually, and the main controlled node end has been safeguarded overall work distribution chart, and controlled node side can in time be notified the main controlled node end with error condition.