CN104714783A

CN104714783A - Task processing method and device

Info

Publication number: CN104714783A
Application number: CN201510142746.0A
Authority: CN
Inventors: 杨康; 袁瑞峰
Original assignee: Beijing Qihoo Technology Co Ltd; Qizhi Software Beijing Co Ltd
Current assignee: Beijing Qihoo Technology Co Ltd
Priority date: 2011-12-27
Filing date: 2011-12-27
Publication date: 2015-06-17
Anticipated expiration: 2031-12-27
Also published as: CN104714783B

Abstract

The application provides a task processing method and device. The task processing method comprises the following steps of: at least one component in a distributed message queue of a multi-channel production-line processing system receives task messages, wherein a connection pool is arranged in the multi-channel production-line processing system and is used for storing information of network connections among all components in the distributed message queue; according to the information of network connection, a valid network connection is selected from the connection pool; the selected valid network connection is used for sending the task messages and carries out task processing requested by the task messages. By the application, the problem of reduction of the cluster processing capability due to node failure in the existing distributed message queue system is avoided, and the cluster processing capability of the multi-channel production-line processing system is effectively guaranteed.

Description

Task processing method and device

The divisional application that patented claim of the present invention is the applying date is on Dec 27th, 2011, application number is 201110444845.6, name is called the Chinese invention patent application of " task processing method and device ".

Technical field

The application relates to networking technology area, particularly relates to a kind of task processing method and device.

Background technology

Multiple tracks pipeline processes technology is a kind of scheduling system of message communicating Network Based.In large-scale distributed environment, the normal multiple tracks pipeline processes technology that uses is issued efficiently data processing task, distributes and is processed.

At present, multiple tracks pipeline processes technology generally uses Distributed Message Queue to realize.In Distributed Message Queue, usually comprise three basic modules, that is, news release assembly, message transfer assembly and message processing components.Wherein, news release assembly, user for system provides fixing API (Application Programming Interface, application programming interfaces), person easy to use submits oneself task message to, after message is submitted, news release assembly, will by TCP (Transmission ControlProtocol, transmission control protocol) agreement or UDP (User Datagram Protocol, User Datagram Protoco (UDP)) agreement, task message is sent to message transfer assembly; The effect of message transfer assembly is interim storing message, and by the communication that releases news of task message to message processing components; When message processing components receives the notice from message transfer assembly, task message will be received by Transmission Control Protocol or udp protocol, afterwards this task message is submitted to processing unit to process, now, message is set to invisible mode by being responsible for by message transfer assembly, prevents multiple task processing unit from receiving same task message; After processing unit completes process, will give out information to message transfer assembly according to result, when processing successfully, notification message transfer assembly is deleted this message by message processing components; When processing unsuccessfully, notification message transfer assembly is set to visible state this message by message processing components again, and other processing units can be made like this to have an opportunity again to process this task.

A kind of Distributed Message Queue system is Gearman system as shown in Figure 1.Gearman is at present based on the Distributed Message Queue system that C/C++ realizes, mainly comprise three assembly: GearmanClient, Gearman Server and Gearman Worker, corresponds respectively to above-mentioned news release assembly, message transfer assembly and message processing components.Gearman system, for the flow process of Message Processing, as shown in fig. 1, comprising: Gearman Client is to Gearman Server release tasks; After GearmanServer reception task, by task scheduling, task is given Gearman Worker process; Gearman Worker processes task, and result is returned to Gearman Server; Gearman Server correspondingly processes the task that it is preserved according to the result returned, and as deleted or resetting to visible state etc., to secondary, a subtask processes into.

Visible, use this implementation of Distributed Message Queue, making task issue assembly without the need to being concerned about the position of task processing unit, only need send task message to message transfer assembly, the task that achieves is issued and is coupled with the solution of task process.On the other hand, task message carries out unified management by transfer assembly, reduces the debugging difficulty again after task process failure.

But current all kinds of Distributed Message Queue systems, all lack the effective management connected for network, connect, usually do not carry out monitoring and management to its connection status, and the network function only depending on system controls for the network set up.Like this, although reasonable the performance test results can be obtained in desirable network environment, but in actual use, particularly in large-scale distributed cluster, usually because the fault of individual node, and making all message send the node all repeatedly will attempting being connected a failure with reception link, the impact like this for performance is very huge.

Summary of the invention

Technical problems to be solved in this application are to provide a kind of task processing method and device, to solve in current Distributed Message Queue system, because of the problem that node failure causes cluster processing power to decline.

In order to solve the problem, this application discloses a kind of task processing method, comprising: one or more connection pool for storing the network connection information in described Distributed Message Queue between each assembly is set; At least one assembly in Distributed Message Queue receives task message; According to the information that described network connects, from described connection pool, an effective network is selected to connect; Use the described effective network selected to connect and send described task message, carry out the task process that described task message is asked.

Preferably, in each assembly of described Distributed Message Queue, be provided with connection pool, the information that the connection pool of each described assembly is connected for the network stored between this assembly with other assembly; The described information connected according to described network, the step selecting an effective network to connect from described connection pool comprises: the information connected according to described network, from receive described task message assembly described connection pool select an effective network to connect.

Preferably, task processing method also comprises: the described effective network selected if use connects the described task message failure of transmission, then reselect an effective network and connect the described task message of transmission, and the failed network of interruption connects and discharges its resource, this network is connected the network being set to lose efficacy and connect; Record the time point of this network connection interruption, within the time interval of setting, do not re-use this network and connect any task message of transmission.

Preferably, task processing method also comprises: when after the time interval through described setting, receive new task message, then the network attempting recovering described interruption connects.

Preferably, task processing method also comprises: the network connection failure recovering described interruption if attempt, then reset the time point of this network connection interruption, within the time interval of described setting, does not re-use this network and connects any task message of transmission; Recover the network successful connection of described interruption if attempt, then this network is connected and be set to the connection of effective network.

Preferably, in the described information connected according to described network, before the step selecting an effective network to connect from described connection pool, also comprise: the information connected according to described network, judge that whether there is effective network in described connection pool connects; If exist, then perform the described information connected according to described network, from described connection pool, select the step that an effective network connects; If do not exist, some break period connected by networks all in described connection pool is set to current point in time, and attempts recovering network connections all in described connection pool.

Preferably, task processing method also comprises: after using the described task message of described effective network connection transmission selected to be successfully completed, this network is connected and returns to described connection pool, and judge the current connection state that this network connects, according to judged result, this network is connected and is set to the connection of effective network or the connection of invalid network.

Preferably, at least one assembly in the Distributed Message Queue of described multiple tracks pipeline processes system also comprises: in each assembly of described Distributed Message Queue, arrange load-balancing algorithm before receiving the step of task message.

Preferably, the described information connected according to described network, comprise from receiving the step selecting an effective network to connect the described connection pool of the assembly of described task message: the information connected according to described network, according to described load-balancing algorithm from receive described task message assembly described connection pool select an effective network to connect.

Preferably, described Distributed Message Queue comprises news release assembly, message transfer assembly and message processing components; Before at least one assembly in the Distributed Message Queue of described multiple tracks pipeline processes system receives the step of task message, also comprise: the connection pool in each assembly of described Distributed Message Queue is encapsulated as respectively network and connects round-robin queue, and message priority queue is set in described message transfer assembly.

Preferably, the described information be connected with described network according to described load-balancing algorithm, comprise from receiving the step selecting an effective network to connect the described connection pool of the assembly of described task message: the information connected according to described network, from the described network of the assembly receiving described task message connects the head of round-robin queue, an effective network is selected to connect; Described method also comprises: after using the described task message of described effective network connection transmission selected to be successfully completed, this network is connected the afterbody returning described network and connect round-robin queue.

Preferably, the described message priority queue in described message transfer assembly is the message priority queue of first-in first-out; The step that at least one assembly in the Distributed Message Queue of described multiple tracks pipeline processes system receives task message comprises: described message transfer assembly, from the head of described message priority queue, searches the task message of first visible state backward as current pending task message.

In order to solve the problem, disclosed herein as well is a kind of Task Processing Unit, comprising: connection pool, the information that the network for storing in described Distributed Message Queue between each assembly connects; Receiver module, for receiving task message at least one assembly in Distributed Message Queue; Select module, for the information connected according to described network, from described connection pool, select an effective network to connect; Link block, sending described task message for using the described effective network of selection to connect, carrying out the task process that described task message is asked.

Preferably, in each assembly of described Distributed Message Queue, be provided with connection pool, the information that the connection pool of each described assembly is connected for the network stored between this assembly with other assembly; Described selection module, for the information connected according to described network, from receive described task message assembly described connection pool select an effective network to connect.

Preferably, Task Processing Unit also comprises: failed module, if use the described effective network selected to connect for described link block send the failure of described task message, then reselect an effective network and connect the described task message of transmission, and the failed network of interruption connects and discharges its resource, this network is connected the network being set to lose efficacy and connect; Record the time point of this network connection interruption, within the time interval of setting, do not re-use this network and connect any task message of transmission.

Preferably, Task Processing Unit also comprises: reconnect module, and for when after the time interval through described setting, described receiver module receives new task message, and the network attempting recovering described interruption connects.

Preferably, Task Processing Unit also comprises: reconnect and arrange module, if for described in reconnect module and attempt recovering the network connection failure of described interruption, then reset the time point of this network connection interruption, within the time interval of described setting, do not re-use this network and connect any task message of transmission; Recover the network successful connection of described interruption if attempt, then this network is connected and be set to the connection of effective network.

Preferably, Task Processing Unit also comprises: judge module, for the information connected according to described network in described selection module, select an effective network connection from described connection pool before, according to the information that described network connects, judge that whether there is effective network in described connection pool connects; If exist, then perform described selection module; If do not exist, some break period connected by networks all in described connection pool is set to current point in time, and attempts recovering network connections all in described connection pool.

Preferably, Task Processing Unit also comprises: successful module, after being successfully completed for using the described task message of described effective network connection transmission selected when described link block, this network is connected and returns to described connection pool, and judge the current connection state that this network connects, according to judged result, this network is connected and is set to the connection of effective network or the connection of invalid network.

Preferably, Task Processing Unit also comprises: first arranges module, before receiving task message at described receiver module, in each assembly of described Distributed Message Queue, arranges load-balancing algorithm.

Preferably, described selection module, for the information connected according to described network, according to described load-balancing algorithm from receive described task message assembly described connection pool select an effective network to connect.

Preferably, described Distributed Message Queue comprises news release assembly, message transfer assembly and message processing components; Described device also comprises: second arranges module, before receiving task message at described receiver module, connection pool in each assembly of described Distributed Message Queue is encapsulated as respectively network and connects round-robin queue, and message priority queue is set in described message transfer assembly.

Preferably, described selection module, for the information connected according to described network, from the described network of the assembly receiving described task message connects the head of round-robin queue, selects an effective network to connect; Described device also comprises: regression block, after being successfully completed, this network is connected the afterbody returning described network and connect round-robin queue for using the described task message of described effective network connection transmission selected when described link block.

Preferably, the described message priority queue in described message transfer assembly is the message priority queue of first-in first-out; Described receiver module, for described message transfer assembly from the head of described message priority queue, searches the task message of first visible state backward as current pending task message.

Compared with prior art, the application has the following advantages:

The application arranges connection pool in multiple tracks pipeline processes system, network connection in Distributed Message Queue between each assembly is managed, when there being task message to arrive, multiple tracks pipeline processes system can be that the network of this message allocative efficiency connects according to the network connection in connection pool.With prior art, when network connection is broken down, as certain node failure in network, multiple tracks pipeline processes system but can not be known, and still repeatedly attempt this network connect compare, the multiple tracks pipeline processes system of the application is by knowing network connection in time to the management of connection pool, when there is certain or some node failures in a network, abandon using the network of these failure nodes to connect, and select normal node to carry out effective network connection, this network on the one hand system can being detected in time and lost efficacy connects, ensure that on the other hand the node of inefficacy for task message transmission have influence on minimum, thus effectively prevent in current Distributed Message Queue system, because of the problem that node failure causes cluster processing power to decline, effectively ensure that the cluster processing power of multiple tracks pipeline processes system.

Accompanying drawing explanation

Fig. 1 is the communication sequential chart carrying out task process according to a kind of Gearman system of prior art;

Fig. 2 is the flow chart of steps of a kind of task processing method according to the embodiment of the present application one;

Fig. 3 is the flow chart of steps of a kind of task processing method according to the embodiment of the present application two;

Fig. 4 is the flow chart of steps of a kind of task processing method according to the embodiment of the present application three;

Fig. 5 is a kind of network in task processing method embodiment illustrated in fig. 4 schematic diagram when connecting round-robin queue's init state;

Fig. 6 is the schematic diagram of the rear state of a kind of network connection round-robin queue application connection in task processing method embodiment illustrated in fig. 4;

Fig. 7 is the structured flowchart of a kind of Task Processing Unit according to the embodiment of the present application four;

Fig. 8 is the structured flowchart of a kind of Task Processing Unit according to the embodiment of the present application five;

Fig. 9 is the communication sequential chart of the network connection management module supervising the network connection in Task Processing Unit embodiment illustrated in fig. 8;

Figure 10 is the process flow diagram of the network connection management module supervising the network connection in Task Processing Unit embodiment illustrated in fig. 8.

Embodiment

For enabling above-mentioned purpose, the feature and advantage of the application more become apparent, below in conjunction with the drawings and specific embodiments, the application is described in further detail.

Embodiment one

With reference to Fig. 2, show the flow chart of steps of a kind of task processing method according to the embodiment of the present application one.

The task processing method of the present embodiment comprises the following steps:

Step S100: the connection pool that one or more information for storing the network connection in described Distributed Message Queue between each assembly is set;

Step S102: at least one assembly in the Distributed Message Queue of multiple tracks pipeline processes system receives task message.

Task message is the message for carrying out corresponding task process, can be the message comprising specific tasks, also can be the message comprising task result.

Multiple assembly can be comprised, as news release assembly, message transfer assembly and message processing components in the Distributed Message Queue of multiple tracks pipeline processes system.When receiving task message and carrying out task process, needing the network depended between assembly to connect, if news release assembly is by being connected with the network between message transfer assembly, task being published to message transfer assembly; After message transfer assembly reception task, through scheduling, task is sent on the message processing components having network to be connected with it; After message processing components processes task, then by being connected with the network between message transfer assembly, task result being returned to message transfer assembly, carrying out subsequent treatment by message transfer assembly.

Connection pool can be arranged in each assembly of Distributed Message Queue, with store this assembly and and other assembly of this assembly and direct connection relational between network connection information, also can store the direct or indirect network connection information between other assembly in this assembly and Distributed Message Queue; Connection pool also can be arranged at the appropriate location outside independent of each assembly of Distributed Message Queue, with the network connection information between the assembly storing direct connection relational, or stores the information that the direct or indirect network between each assembly connects.

Step S104: the information connected according to network, selects an effective network to connect from connection pool.

After assembly receives task message, according to this task message task to be dealt with, from connection pool, an effective network is selected to connect.

Such as, news release assembly receives a task message, determine that this message needs to issue to message transfer assembly, from connection pool, then select the network of product that has between a news release assembly and message transfer assembly to connect, then, this task message is issued by the connection of this network to message transfer assembly.Or, news release assembly selects one to connect to the active block message processing components from news release assembly to message transfer assembly from connection pool again, connected by this network, first task message is published to message transfer assembly, pass through this connection by message transfer assembly again, task message is given task processing modules process.

Step S106: use the effective network selected to connect and send task message, carry out the task process that task message is asked.

The present embodiment by arranging connection pool in multiple tracks pipeline processes system, network connection in Distributed Message Queue between each assembly is managed, when there being task message to arrive, multiple tracks pipeline processes system can be that the network of this message allocative efficiency connects according to the network connection in connection pool.With prior art, when network connection is broken down, as certain node failure in network, multiple tracks pipeline processes system but can not be known, and still repeatedly attempt this network connect compare, the multiple tracks pipeline processes system of the application is by knowing network connection in time to the management of connection pool, when there is certain or some node failures in a network, abandon using the network of these failure nodes to connect, and select normal node to carry out effective network connection, this network on the one hand system can being detected in time and lost efficacy connects, ensure that on the other hand the node of inefficacy for task message transmission have influence on minimum, thus effectively prevent in current Distributed Message Queue system, because of the problem that node failure causes cluster processing power to decline, effectively ensure that the cluster processing power of multiple tracks pipeline processes system.

Embodiment two

With reference to Fig. 3, show the flow chart of steps of a kind of task processing method according to the embodiment of the present application two.

In the present embodiment, connection pool is arranged in each assembly of Distributed Message Queue, the information that the connection pool of each assembly is connected for the network stored between this assembly with other assembly.

In the present embodiment, Distributed Message Queue comprises news release assembly, message transfer assembly and message processing components, and setup message is issued assembly and received a task message, selects network to connect and carries out task process.It should be noted that, when the multiple assemblies in Distributed Message Queue receive task message, each assembly, to the process of task message, all can refer to the present embodiment and carries out.

Step S202: the news release assembly in the Distributed Message Queue of multiple tracks pipeline processes system receives a task message.

Step S204: news release assembly, according to the network connection information in its connection pool, judges that whether there is effective network in its connection pool connects, if exist, then performs step S206; If do not exist, then perform step S220.

Network connection information describes this assembly and the information of other component network connection state, can comprise streamline name, destination address port numbers, local port number, and the effective or disarmed state etc. that network connects.In the present embodiment, in the connection pool of news release assembly, include the information that the network between all news release assemblies with message transfer assembly is connected.

Similar, then include in the connection pool of message transfer assembly all message transfer assemblies with between news release assembly and information that message transfer assembly is connected with the network between message processing components; The information that the network between all message processing components with message transfer assembly is connected then is included in the connection pool of message processing components.

Connecting by judging whether to exist in connection pool effective network, assembly can be made to understand the current connection state of its and other assembly fast, normal with Logistics networks connection, for the normal process of task message provides effective guarantee.

Step S206: news release assembly selects a network to connect from all effective networks connect.

Step S208: news release assembly uses the network selected to connect and task message is sent to message transfer assembly.

After message transfer assembly receives this task message, carry out suitable scheduling, give message processing components and carry out corresponding task process.

Step S210: whether news release assembly judges to use the network selected to connect transmission task message successful, if success, then performs step S212; If failure, then perform step S214.

Step S212: the network connection sending task message is returned to its connection pool by news release assembly, and judge the current connection state that this network connects, according to judged result, this network is connected and is set to the connection of effective network or the connection of invalid network, terminate this task message process of transmitting.

When task message sends successfully, in time network connection resource is returned to connection pool, other task message can be made to have enough network connection resource to use; And by judging the current connection state that the network that returns connects, determine that this network connects effectively or lost efficacy, then to assembly provide that network connects timely, reliable information, with the efficient process of support mission.

Step S214: news release assembly reselects an effective network and connects transmission task message, and the network selected before interrupting connects, and discharges the resource that this network connects, and is connected by the network that the connection of this network is set to lose efficacy; Further, record the time point of this network connection interruption, within the time interval of setting, do not re-use this network and connect any task message of transmission.

The time interval, as the quantity etc. of the switch that the network bandwidth or this network are connected through, arrange flexibly, the application was not restricted this by those skilled in the art according to network condition.

Step S216: when after the time interval through setting, news release assembly receives new task message, the network attempting recovering to interrupt connects.

Step S218: judge whether the network attempting recovering interruption connects successful, if success, then connects this network and is set to the connection of effective network, return step S204; If failure, then reset the time point of this network connection interruption, within the time interval of setting, do not re-use this network and connect any task message of transmission, return step S216.

Reconnect time interval employing fixed time interval, with handled easily and realization.Therefore, the time interval of the setting in this step is consistent with the time interval in step S214.

In large-scale distributed cluster, the reason of Joint failure has a lot, as the temporary fault of network.How effectively to recover these connections of losing efficacy, become the important problem of management.In the application, introduce the setting of the time of reconnecting, namely set a time interval.When after a Joint failure, according to out-of-service time point backward, one can be selected to reconnect the time, the request before reconnecting the time, can not attempt recovering connection, to reduce the expense connecting retry.When there being request to arrive after reconnecting the time, trial being recovered this network and connects, if recover unsuccessfully, then again arrange and reconnect the time; If success, then this connection is returned and be supplied to message components and communicate.

Step S220: some break period connected by networks all in the connection pool of news release assembly is set to current point in time, and attempt recovering network connections all in this connection pool; Then, step S204 is returned.

When connections all in connection pool were all lost efficacy, by cancel all connections reconnect the time, because at this moment news release assembly cannot provide any connection for communication, news release assembly normally cannot complete the process of streamline, now should recover network as early as possible to connect, ensure the normal process of task.Preferably, some break period connected by networks all in connection pool is set to current point in time, facilitates put break period and arranges, and subsequent interval Time Calculation, is convenient to realize.

Pass through the present embodiment, effectively ensure that the transmission of task message, and then ensure that the process of task, avoid the problem because node failure causes the cluster processing power of Distributed Message Queue system to decline, ensure that the cluster processing power of multiple tracks pipeline processes system.

It should be noted that, the present embodiment is only for news release assembly, the process of task message is explained, but it should be understood by those skilled in the art that other assembly of Distributed Message Queue all can refer to the present embodiment and carried out correspondingly processing to task message.

Embodiment three

With reference to Fig. 4, show the flow chart of steps of a kind of task processing method according to the embodiment of the present application three.

The present embodiment, on basis embodiment illustrated in fig. 3, each assembly arranges load-balancing algorithm, increases load-balancing function, to carry out load balancing to network connection or task message.Further, the connection pool in each assembly of Distributed Message Queue is encapsulated as respectively network and connects round-robin queue, turn in the message in assembly simultaneously and message priority queue is set.That is, be provided with network in message transfer assembly and connect round-robin queue and message priority queue, and news release assembly and message processing components are because of can not store tasks message temporarily, so be only provided with network in these two assemblies to connect round-robin queue, and there is no message priority queue.

The present embodiment, for message transfer assembly, carries out Message Processing and network to it according to load-balancing algorithm and connects selection and be described.It should be noted that, be connected round-robin queue because news release assembly and message processing components are only provided with network, therefore, only can carry out load balancing with reference to the network connection selecting method of the message transfer assembly of the present embodiment.

Step S302: task message is passed to message transfer assembly by news release assembly.

Wherein, news release assembly, according to the load-balancing algorithm of setting, connects round-robin queue from the network of himself and selects an effective network to connect, task message is passed to message transfer assembly.

In the present embodiment, set each assembly and all use LRU (Least Recently Used) algorithm to carry out the load balancing of network connection.When selecting active block to connect, the head connecting round-robin queue being extracted an effective network and connecting, return to assembly and communicate from network.After communication completes, this connection will be placed into the afterbody that network connects round-robin queue.After schematic diagram during network connection round-robin queue's init state is connected with application, the schematic diagram of state respectively as shown in Figure 5 and Figure 6.

In this way, achieve a complete lru algorithm, and make all assemblies that this algorithm can be used to carry out the management connected, to realize the load balancing of the overall situation.Use lru algorithm, realize simple, and search active block connection rapidly and efficiently.Certainly, be not limited thereto, those skilled in the art are in actual time, and can adopt load-balancing algorithm suitable arbitrarily, the application is not restricted this.

Step S304: task message is inserted the afterbody of message priority queue by message transfer assembly successively, and this task message is set to visible state.

Step S306: message transfer assembly, according to the load-balancing algorithm of setting, selects the task message of a visible state from message priority queue.

In the present embodiment, the queue of fifo fifo mode in setup message priority query.Message transfer assembly, from message priority queue head, searches the task message of first visible state backward, as current pending task message.

Message priority queue is in order to ensure that task message can process and design by reliable, fair carrying out.This queue uses the mode of first in first out (FIFO), task message is managed, which ensure that the fairness of message in sequential, namely first submitting to of task, preferential process can be obtained, also do not affect the correct scheduling of the task of follow-up submission simultaneously.

Step S308: task message is sent to message processing components by message transfer assembly.

In this step, message transfer assembly, according to lru algorithm, connects round-robin queue from the network of self and selects an effective network to connect, task message is sent to message processing components.

Step S310: the task message sent is set to invisible mode by message transfer assembly, and the reply of the processing components that waits for the arrival of news.

Step S312: message transfer assembly, according to the reply of Message Processing process, processes this task message; Then, step S306 is returned.

If this task message success of task processing modules process, message transfer assembly is by this task message of deletion; If this task message of task processing modules process is unsuccessful, this task message will be set to visible state by message transfer assembly again.

The present embodiment emphasis is illustrated the load balancing of each assembly in Distributed Message Queue, and to the selection of network connection and the comparatively simple of processing section description, those skilled in the art in the specific implementation, the selection of active block connection and follow-up relevant treatment can be carried out with reference to the correlation technique in embodiment two, not repeat them here.

Under large-scale distributed environment, the ability of individual node is limited, is usually merely able to the part meeting a certain generic task performance requirement, even 1/tens, this just requires whole multiple tracks pipeline processes system, can make full use of the process that current all known node carry out task.Current all kinds of Distributed Message Queue systems, for the problem of load balancing, all do not provide rational solution, and most scheduling strategy just depending on system regulates.To cause part of nodes load too high like this, but system does not still make full use of the processing power of all processing nodes, the ability of great deal of nodes is wasted.Pass through the present embodiment, not only solve in Distributed Message Queue, because of the problem that node failure causes cluster processing power to decline, and solve the problem of load balancing of Distributed Message Queue, the whole Processing tasks in whole system can be assigned in each processing node equably.

Embodiment four

With reference to Fig. 7, show the structured flowchart of a kind of Task Processing Unit according to the embodiment of the present application four.

The Task Processing Unit of the present embodiment comprises: connection pool, the information that the network for storing in described Distributed Message Queue between each assembly connects; Receiver module 402, for receiving task message at least one assembly in the Distributed Message Queue of multiple tracks pipeline processes system; Select module 404, for the information connected according to network, from connection pool, select an effective network to connect; Link block 406, sending task message for using the effective network of selection to connect, carrying out the task process that task message is asked.

Preferably, in each assembly of Distributed Message Queue, be provided with connection pool, the information that the connection pool of each assembly is connected for the network stored between this assembly with other assembly; Select module 404, for the information connected according to network, from receive task message assembly connection pool select an effective network connection.

Preferably, the Task Processing Unit of the present embodiment also comprises: failed module 408, if use the effective network selected to connect for link block 406 send task message failure, then reselect an effective network and connect transmission task message, and the failed network of interruption connects and discharges its resource, this network is connected the network being set to lose efficacy and connect; Record the time point of this network connection interruption, within the time interval of setting, do not re-use this network and connect any task message of transmission.

Preferably, the Task Processing Unit of the present embodiment also comprises: reconnect module 410, for when through setting the time interval after, receiver module 402 receives new task message, attempt recover interrupt network connect.

Preferably, the Task Processing Unit of the present embodiment also comprises: reconnect and arrange module 412, if for reconnecting the network connection failure that module 410 attempts recovering to interrupt, then reset the time point of this network connection interruption, within the time interval of setting, do not re-use this network and connect any task message of transmission; Recover the network successful connection interrupted if attempt, then this network is connected and be set to the connection of effective network.

Preferably, the Task Processing Unit of the present embodiment also comprises: judge module 414, for the information connected according to network in selection module 404, select an effective network connection from connection pool before, according to the information that network connects, judge that whether there is effective network in connection pool connects; If exist, then perform and select module 404; If do not exist, some break period connected by networks all in connection pool is set to current point in time, and attempts recovering network connections all in connection pool.

Preferably, the Task Processing Unit of the present embodiment also comprises: successful module 416, after being successfully completed for using the effective network connection transmission task message selected when link block 406, this network is connected and returns to connection pool, and judge the current connection state that this network connects, according to judged result, this network is connected and is set to the connection of effective network or the connection of invalid network.

Preferably, the Task Processing Unit of the present embodiment also comprises: first arranges module 418, before receiving task message at receiver module 402, in each assembly of Distributed Message Queue, arranges load-balancing algorithm.

Preferably, select module 404, for the information connected according to network, according to load-balancing algorithm from receive task message assembly connection pool select an effective network connection.

Preferably, Distributed Message Queue comprises news release assembly, message transfer assembly and message processing components; The Task Processing Unit of the present embodiment also comprises: second arranges module 420, before receiving task message at receiver module 402, connection pool in each assembly of Distributed Message Queue is encapsulated as respectively network and connects round-robin queue, and turn in the message in assembly message priority queue is set.

Preferably, select module 404, for the information connected according to network, from the network of the assembly receiving task message connects the head of round-robin queue, select an effective network to connect; The Task Processing Unit of the present embodiment also comprises: regression block (not shown), after being successfully completed for using the effective network connection transmission task message selected when link block 406, this network is connected the afterbody returning network and connect round-robin queue.

Preferably, the message priority queue in message transfer assembly is the message priority queue of first-in first-out; Receiver module 402, for message transfer assembly from the head of message priority queue, searches the task message of first visible state backward as current pending task message.

The Task Processing Unit of the present embodiment is used for realizing corresponding task processing method in aforementioned multiple embodiment of the method, and has the beneficial effect of corresponding task processing method, does not repeat them here.

Embodiment five

With reference to Fig. 8, show the structured flowchart of a kind of Task Processing Unit according to the embodiment of the present application five.

In the present embodiment, in each assembly of Distributed Message Queue, be provided with connection pool, the information that the connection pool of each assembly is connected for the network stored between this assembly with other assembly; Further, each assembly is provided with load-balancing algorithm, to carry out load balancing to network connection or task message.

The Task Processing Unit of the present embodiment mainly comprises: network connection management module 502 and load balancing module 504.

Wherein:

Network connection management module 502 is responsible for the network connection management in all component, can be arranged in each assembly, also can be arranged on outside each assembly, major function is the state that monitor network connects, and the management (its function includes selection module, link block, failed module, successful module in embodiment four, reconnects module and reconnect the function arranging the modules such as module) connected for losing efficacy.Its monitoring and management is all the connection pool by setting up in each assembly, is connected by the all-network in current component in order and is all placed in connection pool.

For the network connection management function of network connection management module 502, when needing when there being network service to carry out, assembly connects for communication use by the corresponding interface request network, and network connection management module 502 will be selected an available connection (namely effective network connects) and return in the connection pool of assembly.When communication is successfully completed, the connection pool that assembly will return to this connection network connection management module 502 and manages; If carry out communication failure by this connection, then system can informing network connection management module 502.Afterwards, network connection management module 502 can analyze failure cause, and by this inefficacy disconnecting with releasing resource, also can record the time point of this interruption simultaneously, in a certain time interval, network connection management module 502 will not re-use the connection of this network and communicate, and prevent single point failure problem, the hydraulic performance decline caused.Network connection management module 502 pairs of networks connect the sequential chart that manages as shown in Figure 9.Visible, when the assembly of Distributed Message Queue, as news release assembly, message transfer assembly or message processing components, when needs network connects, send request the request of connection to network connection management module 502; Network connection management module 502 upon receiving a request, is selected an effective network to connect, is returned to the assembly of request from connection pool; This connection, after the active block connection using network connection management module 502 to return communicates, is returned to network connection management module 502 by assembly again; After this, network connection management module 502 judges the current connection state of this connection returned, so that the network more effectively between Management Unit connects.

Second major function of network connection management module 502 is that management inefficacy connects, and in large-scale distributed cluster, the reason of Joint failure has a lot, as the temporary fault of network.How effectively to recover these connections of losing efficacy, become the important problem of management.Here, the application introduces the setting of the time of reconnecting, when after a Joint failure, can put backward according to the out-of-service time, select one to reconnect the time, the request before reconnecting the time, network connection management module 502 can not be attempted recovering connection, to reduce the expense connecting retry.When there being request to arrive after reconnecting the time, trial is recovered this network and connects by network connection management module 502, if recover unsuccessfully, then again arranges and reconnects the time; If success, then this connection is returned and be supplied to message components and communicate.

It is as follows that network connection management module 502 arranges the example reconnecting the interface of time:

Waiting-timeout time when message components can be connected by this interface setting and the time interval reconnected.During concrete setting, different business can be arranged according to the demand of self and network state.Such as, in same network segment, can by timeout time shorten, to reduce expense when network reconnects unsuccessfully; When network fluctuation is more frequent, by reconnect_interval time shorten, can connect to recover network as early as possible, avoid when all Joint failures, reconnect a large amount of expenses that all connections cause.

In addition, in Interface design, the Process integration of recovery has been arrived the interface internal of network connection management module 502 by the application, can ensure like this, when assembly does not communicate, network connection management module 502 can not carry out unnecessary trial and connect, and reduce further the expense of recovery.It is significant to note that, when connections all in connection pool were all lost efficacy, by cancel all connections reconnect the time, because at this moment network connection management module 502 cannot provide any connection to communicate for message components, message components normally cannot complete the process of streamline, now should recover network as early as possible to connect, ensure the normal process of task.

In the present embodiment, network connection management module 502 realizes the flow process of network management as shown in Figure 10, comprising:

Step S602: network connection management module 502 receives the request that request network connects.

Step S604: network connection management module 502 checks that the next one in the connection pool of corresponding assembly connects.

Step S606: network connection management module 502 checks that whether this connection is effective, if so, then performs step S608; If not, then step S610 is performed.

Step S608: network connection management module 502 returns this connection to assembly, this connects selection course and terminates.

Step S610: network connection management module 502 judges whether the time that reconnects of this connection arrives, if so, then performs step S612; If not, then step S618 is performed.

Step S612: network connection management module 502 is attempted recovering this connection.

Step S614: whether network connection management module 502 judges to attempt connection successful, if so, then returns step S608; If not, then step S616 is performed.

Step S616: network connection management module 502 reconnects the time for this connection is arranged again, then, returns step S604.

Step S618: network connection management module 502 judges whether all connections in inspected connection pool, if so, performs step S620; If not, then step S604 is returned.

Step S620: what network connection management module 502 reset all connections in connection pool reconnects the time.

By said process, network connection management module 502 achieve component passes task message active block connect, solve existing Distributed Message Queue interior joint lost efficacy bring processing power decline problem.

But, in large-scale cluster system, i.e. enable solution node failure problem, if but a large amount of tasks concentrates in same assembly and carries out, even if then this assembly uses connection pool and network connection management module 502 pairs of networks to be connected to manage, also cannot ensure the fully available and effective of connection.For this reason, load balancing module 504 is also provided with in the Task Processing Unit of the present embodiment.

Can be known by the communication sequential chart of Fig. 1 Distributed Message Queue, the network service between the assembly of Distributed Message Queue has 3 groups of transmission directions: news release assembly → message transfer assembly; Message transfer assembly → message processing components; Message processing components → message transfer assembly.The load imbalance of any group communication, all causes serious impact by the processing power of system.

Such as: during the load imbalance of news release assembly → message transfer assembly, a large amount of message will be caused to be sent to some message transfer assembly, make these transfer assemblies too busy, response cannot be provided in time; Idle transfer assembly does not then have message to dispatch.At this moment, message transfer assembly is by handling capacity overall for restriction system.During the load imbalance of message transfer assembly → message processing components, can cause transfer assembly that a large amount of message is sent to some message processing components, make these processing components too busy; And the message that another part processing components cannot get in transfer assembly processes, process waiting status always.This phenomenon can reduce the processing power of Processing Cluster on the whole, even if carry out horizontal extension, increases the quantity of processing unit, but also cannot get the message on transfer assembly due to the node newly added, and cause the inefficacy of horizontal extension.The load imbalance of message processing components → message transfer assembly, with the similar phenomena of news release assembly → message transfer assembly.Task message on some message transfer assembly of a message processing components priority processing, when any message appears in these transfer assemblies, has a large amount of processing components to dispatch; And the task message on another part transfer assembly, then cannot be responded in time, caused the time delay of task process.

For above problem, the present embodiment is in all message components, all be provided with load balancing module 504 (its function comprise in embodiment four first, second arranges the function of the module such as module and regression block), add load-balancing algorithm, thus ensure that the task message that all news release assemblies send, can both be scheduled on all message processing components liberally.

For this reason, the present embodiment devises 2 queue systems, and one is message priority queue, and one is that network connects round-robin queue.News release assembly and message processing components, because can not store tasks message temporarily, so do not have message priority queue, only have the round-robin queue that network connects; Message transfer assembly then contains message priority queue simultaneously and is connected round-robin queue with network.

In the implementation, the network connection pool that the present embodiment will above realize, utilizes round-robin queue to encapsulate, and forms network and connects round-robin queue.When needing when there being any assembly to carry out network service, the head connecting round-robin queue from network is extracted a connection by load balancing module 504, returns to assembly and communicates.After communication completes, this connection will be placed into the afterbody that network connects round-robin queue.In this way, the present embodiment achieves a complete lru algorithm, and makes all assemblies that this algorithm can be used to carry out the management connected, to realize the load balancing of the overall situation.The schematic diagram that network connects round-robin queue's init state is connected the schematic diagram of rear state as shown in Figure 5 and Figure 6 with application.

Message priority queue is in order to ensure that message can process and design by reliable, fair carrying out.This queue uses the mode of first in first out (FIFO), manages, which ensure that the fairness of message in sequential, namely first submitting to of task, can obtain preferential process message, does not also affect the correct scheduling of the task of follow-up submission simultaneously.

When using above-mentioned message priority queue to dispatch task message, first by news release assembly, task message is passed to message transfer assembly; This message is inserted the afterbody of priority query by message transfer assembly successively, and this message is set to visible state; The load balancing module 504 of message transfer assembly, from priority query's head, searches the message of first visible state backward, sends to corresponding message processing components afterwards; The message sent is set to invisible mode by message transfer assembly, and waits for the reply of processing components; If process successfully, message transfer assembly is by this message of deletion; If process unsuccessful, this message will be set to visible state by message transfer assembly again.So far, the dispatch deal that task message turns in assembly in the message completes.

The present embodiment, for the feature under distributed environment, provides under solving large-scale cluster situation, carries out equalization task load, and when machine breaks down, the scheme of the problem that single point failure causes cluster processing power to decline.By the present embodiment, connect manage in message components for network, the network on the one hand system can being detected in time to have lost efficacy connects, ensure on the other hand the node that lost efficacy for task message transmission have influence on minimum; By configurable interface, user can be controlled to carry out for failure node the strategy that recovers according to demand; In news release assembly, message transfer assembly and message processing components, realize load-balancing algorithm respectively, the whole Processing tasks in whole system can be assigned in each processing node equably.

The task processing scheme of the application can in large-scale distributed cluster, realize the process of the multiple tracks pipeline system of task, and carried out design and study in detail for the single point failure problem in reality use, problem of load balancing, ensure that making full use of of cluster processing power.

The task processing scheme of the application supports cluster concurrent process pattern, can dynamic expansion system to the service of more than 1000+; Platform, without I/O demand, only relies on CPU, MEM, NETWORK, and data throughout is only confined to quantity and the network I/O ability of node; Improve based on open source software gearmand, solve some BUG and the performance bottleneck of original version; Processing module supports " hot plug pattern ", supports different business flow process fast construction demand.

The technical scheme of the application solves the problem of magnanimity real time data processing, solves after sample data uploads, need fast through storing, qualification, sorting, calculate the problem of the link process such as RANK.The sample data of 2 ~ 4T bytes/day every day can be supported, reach the processing power that average retardation is not more than 30s.Can be widely used on the cloud computing platform of 360, be the products such as wooden horse cloud killing engine, Active Defending System Against, net shield security system, user's commending system, provides efficient, data analysis service reliably.

Each embodiment in this instructions all adopts the mode of going forward one by one to describe, and what each embodiment stressed is the difference with other embodiments, between each embodiment identical similar part mutually see.For device embodiment, due to itself and embodiment of the method basic simlarity, so description is fairly simple, relevant part illustrates see the part of embodiment of the method.

A kind of task processing method provided the application above and device are described in detail, apply specific case herein to set forth the principle of the application and embodiment, the explanation of above embodiment is just for helping method and the core concept thereof of understanding the application; Meanwhile, for one of ordinary skill in the art, according to the thought of the application, all will change in specific embodiments and applications, in sum, this description should not be construed as the restriction to the application.

The embodiment of the invention discloses A1, a kind of task processing method, comprising:

The connection pool of one or more information for storing the network connection in described Distributed Message Queue between each assembly is set;

At least one assembly in Distributed Message Queue receives task message; According to the information that described network connects, from described connection pool, an active block is selected to connect;

Use the described active block selected to connect and send described task message, carry out the task process that described task message is asked.

A2, method according to claim A1, be provided with connection pool in each assembly of described Distributed Message Queue, the information that the connection pool of each described assembly is connected for the network stored between this assembly with other assembly;

The described information connected according to described network, the step selecting an effective network to connect from described connection pool comprises: the information connected according to described network, from receive described task message assembly described connection pool select an effective network to connect.

A3, method according to claim A1 or A2, also comprise:

The described effective network selected if use connects the described task message failure of transmission, then reselect an effective network and connect the described task message of transmission, and the failed network of interruption connects and discharges its resource, this network is connected the network being set to lose efficacy and connect; Record the time point of this network connection interruption, within the time interval of setting, do not re-use this network and connect any task message of transmission.

A4, method according to claim A3, also comprise:

When after the time interval through described setting, receive new task message, then the network attempting recovering described interruption connects.

A5, method according to claim A4, also comprise:

Recover the network connection failure of described interruption if attempt, then reset the time point of this network connection interruption, within the time interval of described setting, do not re-use this network and connect any task message of transmission; Recover the network successful connection of described interruption if attempt, then this network is connected and be set to the connection of effective network.

A6, method according to claim A1, in the described information connected according to described network, before the step selecting an effective network to connect, also comprise from described connection pool:

According to the information that described network connects, judge that whether there is effective network in described connection pool connects;

If exist, then perform the described information connected according to described network, from described connection pool, select the step that an effective network connects;

If do not exist, some break period connected by networks all in described connection pool is set to current point in time, and attempts recovering network connections all in described connection pool.

A7, method according to claim A1, also comprise:

After using the described task message of described effective network connection transmission selected to be successfully completed, this network is connected and returns to described connection pool, and judge the current connection state that this network connects, according to judged result, this network is connected and is set to the connection of effective network or the connection of invalid network.

A8, method according to claim A2, at least one assembly in the Distributed Message Queue of described multiple tracks pipeline processes system also comprises before receiving the step of task message:

In each assembly of described Distributed Message Queue, load-balancing algorithm is set.

A9, method according to claim A8, the described information connected according to described network, comprises from receiving the step selecting an effective network to connect the described connection pool of the assembly of described task message:

According to the information that described network connects, according to described load-balancing algorithm from receive described task message assembly described connection pool select an effective network to connect.

A10, method according to claim A9, described Distributed Message Queue comprises news release assembly, message transfer assembly and message processing components;

Before at least one assembly in the Distributed Message Queue of described multiple tracks pipeline processes system receives the step of task message, also comprise: the connection pool in each assembly of described Distributed Message Queue is encapsulated as respectively network and connects round-robin queue, and message priority queue is set in described message transfer assembly.

A11, method according to claim A10,

The described information be connected with described network according to described load-balancing algorithm, comprise from receiving the step selecting an effective network to connect the described connection pool of the assembly of described task message: the information connected according to described network, from the described network of the assembly receiving described task message connects the head of round-robin queue, an effective network is selected to connect;

Described method also comprises: after using the described task message of described effective network connection transmission selected to be successfully completed, this network is connected the afterbody returning described network and connect round-robin queue.

A12, method according to claim A10, the described message priority queue in described message transfer assembly is the message priority queue of first-in first-out;

The step that at least one assembly in the Distributed Message Queue of described multiple tracks pipeline processes system receives task message comprises: described message transfer assembly, from the head of described message priority queue, searches the task message of first visible state backward as current pending task message.

B13, a kind of Task Processing Unit, comprising:

Connection pool, the information that the network for storing in described Distributed Message Queue between each assembly connects;

Receiver module, for receiving task message at least one assembly in Distributed Message Queue;

Select module, for the information connected according to described network, from described connection pool, select an active block to connect;

Link block, sending described task message for using the described active block of selection to connect, carrying out the task process that described task message is asked.

B14, device according to claim B 13, be provided with connection pool in each assembly of described Distributed Message Queue, the information that the connection pool of each described assembly is connected for the network stored between this assembly with other assembly;

Described selection module, for the information connected according to described network, from receive described task message assembly described connection pool select an effective network to connect.

B15, device according to claim B 13 or B 14, also comprise:

Failure module, if use the described effective network selected to connect for described link block send the failure of described task message, then reselect an effective network and connect the described task message of transmission, and the failed network of interruption connects and discharges its resource, this network is connected the network being set to lose efficacy and connect; Record the time point of this network connection interruption, within the time interval of setting, do not re-use this network and connect any task message of transmission.

B16, device according to claim B 15, also comprise:

Reconnect module, for when after the time interval through described setting, described receiver module receives new task message, and the network attempting recovering described interruption connects.

B17, device according to claim B 16, also comprise:

Reconnect and module be set, if for described in reconnect module and attempt recovering the network connection failure of described interruption, then reset the time point of this network connection interruption, within the time interval of described setting, do not re-use this network and connect and send any task message; Recover the network successful connection of described interruption if attempt, then this network is connected and be set to the connection of effective network.

B18, device according to claim B 13, also comprise:

Judge module, for the information connected according to described network in described selection module, select an effective network connection from described connection pool before, according to the information that described network connects, judges that whether there is effective network in described connection pool connects; If exist, then perform described selection module; If do not exist, some break period connected by networks all in described connection pool is set to current point in time, and attempts recovering network connections all in described connection pool.

B19, device according to claim B 13, also comprise:

Success module, after being successfully completed for using the described task message of described effective network connection transmission selected when described link block, this network is connected and returns to described connection pool, and judge the current connection state that this network connects, according to judged result, this network is connected and is set to the connection of effective network or the connection of invalid network.

B20, device according to claim B 14, also comprise:

First arranges module, before receiving task message at described receiver module, in each assembly of described Distributed Message Queue, arranges load-balancing algorithm.

B21, device according to claim B 20, described selection module, for the information connected according to described network, according to described load-balancing algorithm from receive described task message assembly described connection pool select an effective network to connect.

B22, device according to claim B 21, described Distributed Message Queue comprises news release assembly, message transfer assembly and message processing components;

Described device also comprises: second arranges module, before receiving task message at described receiver module, connection pool in each assembly of described Distributed Message Queue is encapsulated as respectively network and connects round-robin queue, and message priority queue is set in described message transfer assembly.

B23, device according to claim B 22,

Described selection module, for the information connected according to described network, from the described network of the assembly receiving described task message connects the head of round-robin queue, selects an effective network to connect;

Described device also comprises: regression block, after being successfully completed, this network is connected the afterbody returning described network and connect round-robin queue for using the described task message of described effective network connection transmission selected when described link block.

B24, device according to claim B 22, the described message priority queue in described message transfer assembly is the message priority queue of first-in first-out;

Described receiver module, for described message transfer assembly from the head of described message priority queue, searches the task message of first visible state backward as current pending task message.

Claims

1. a task processing method, is characterized in that, comprising:

2. method according to claim 1, is characterized in that, is provided with connection pool in each assembly of described Distributed Message Queue, the information that the connection pool of each described assembly is connected for the network stored between this assembly with other assembly;

3. method according to claim 1 and 2, is characterized in that, also comprises:

4. method according to claim 3, is characterized in that, also comprises:

5. method according to claim 4, is characterized in that, also comprises:

6. method according to claim 1, is characterized in that, in the described information connected according to described network, before the step selecting an effective network to connect, also comprises from described connection pool:

7. method according to claim 1, is characterized in that, also comprises:

8. method according to claim 2, is characterized in that, at least one assembly in the Distributed Message Queue of described multiple tracks pipeline processes system also comprises before receiving the step of task message:

9. method according to claim 8, is characterized in that, the described information connected according to described network, comprises from receiving the step selecting an effective network to connect the described connection pool of the assembly of described task message:

10. a Task Processing Unit, is characterized in that, comprising: