WO2013097248A1

WO2013097248A1 - Distributed task processing method, device and system based on message queue

Info

Publication number: WO2013097248A1
Application number: PCT/CN2011/085203
Authority: WO
Inventors: 武嘉
Original assignee: 华为技术有限公司
Priority date: 2011-12-31
Filing date: 2011-12-31
Publication date: 2013-07-04
Also published as: CN103502943B; CN103502943A

Abstract

The present invention relates to the field of information technology, and in particular, to a distributed task processing method, device and system based on a message queue. In the distributed task processing method based on a message queue provided in the embodiments of the present invention, each message queue in a task message queue group stores one kind of processing task borne by a task processing node, the task processing node determines the priority level of a message queue which stores a processing task according to the priority level of a processing task borne thereby and starts to circularly scan the message queues in the task message queue group successively from the message queue with the highest priority level in an order from high priority level to low priority level and executes the scanned processing task. By way of the method provided in the embodiments of the present invention, different kinds of tasks are distributed in different message queues without ranking the processing tasks in the message queue, which cancels the system overhead due to ranking and improves the working efficiency of the system.

Description

Method, device and system for distributed task processing based on message queue

The present invention relates to the field of information technology (Information Technology, referred to as IT), and in particular, to a message queue-based distributed task processing method, device and system.

Background technique

Distributed computing is a computer science that studies how to divide a problem that requires a very large computational power into many small parts, then distributes these parts to many computers for processing, and finally combines these calculations to get the final result. the result of. These computing resources can be run on the same computer or on multiple computers connected via a network.

Distributed computing has several advantages over other algorithms: Rare resources can be shared; distributed computing can balance computing load across multiple computers; programs can be placed on the computer that is best suited to run it.

In the existing distributed computing system, in order to implement hierarchical management of tasks, a message queue (Message Queue, MQ for short), a central node, and several task processing nodes are usually maintained. The message queue is used to save the task to be executed, and the central node is used to sort the tasks in the message queue according to the priority of the task, and implement the hierarchical management function to ensure that the high priority task is located in the queue of the message queue. The processing node is used to cyclically scan the message queue, obtain the task at the head of the queue from the message queue, and execute the task.

Through the analysis of the prior art, the inventor believes that the prior art has at least the following problems: The prior art needs to sort the tasks in the message queue according to the task priority. When the number of tasks to be processed is large and the number of tasks is large, Increase the overhead of system sequencing and reduce the efficiency of the system.

Summary of the invention It is an object of embodiments of the present invention to provide a method, apparatus, and system for distributed task processing based on message queues, so that the distributed task processing system does not need to sort processing tasks in the message queue.

A message queue-based distributed task processing method is provided in the embodiment of the present invention. Each message queue in the task message queue group stores a processing task carried by the task processing node, and the method includes:

A, the task processing node determines, according to the priority level of the processing task carried by the task, the priority level of the message queue storing the processing task;

B. The task processing node sequentially scans the message queues in the task message queue group from the message queue with the highest priority level according to the priority level from high to low. If the message queue containing the processing task is scanned, Perform step C, otherwise, repeat step B;

C. The task processing node acquires the current processing task, and executes the current processing task. After the execution, the process returns to step B, where the current processing task is stored in the message queue including the processing task.

A message queue-based distributed task scheduling method provided by an embodiment of the present invention includes: acquiring a processed task after segmentation;

And according to the type of the processing task, the processing task is sent to a message queue associated with the type of the processing task in the task message queue group, waiting for the task processing node to execute.

A task processing node provided by an embodiment of the present invention includes:

a message queue priority level determining unit, configured to determine a priority level of a message queue storing the processing task according to a priority level of a processing task carried by the self-host;

a scanning unit, configured to scan, according to the priority level of the message queue determined by the message queue priority level determining unit, the task message is sequentially scanned from the message queue with the highest priority level according to the order of priority level from highest to lowest a message queue in a queue group;

a task execution unit, configured to: when the scanning unit scans a message queue that includes a processing task, Acquiring a current processing task, and executing the current processing task, where the current processing task is stored in the message queue that includes the processing task.

A scheduler provided by an embodiment of the present invention includes:

An obtaining unit, configured to obtain a processed task after the segmentation;

And a sending unit, configured to send, according to the type of the processing task acquired by the acquiring unit, the processing task to a message queue associated with the type of the processing task in the task message queue group, and wait for the task processing node to execute.

A distributed task processing system according to an embodiment of the present invention, the system includes at least one task processing node, where the task processing node is configured to determine, according to a priority level of a processing task carried by the task, a message storing the processing task. The priority level of the queue, and the message queues in the task message queue group are cyclically scanned from the message queue with the highest priority level in order of priority level, and the scanned processing tasks are executed, where Each message queue in the task message queue group stores a processing task carried by the task processing node;

The task processing node includes:

a task execution unit, configured to: when the scan unit scans a message queue that includes a processing task, acquire a current processing task, and execute the current processing task, where the current processing task is stored in the message that includes the processing task In the queue.

The method, the device and the system for the distributed task processing based on the message queue are provided in the embodiment of the present invention. Different types of tasks are allocated in different message queues. Compared with the prior art, the embodiment of the present invention does not need to be Processing tasks in the message queue are sorted in order of priority, eliminating the need for sorting The resulting system overhead increases the efficiency of the system work.

DRAWINGS

FIG. 1 is a schematic structural diagram of a distributed task processing system according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a method for distributed task processing based on message queues according to an embodiment of the present invention; FIG. 3 is a schematic diagram of a distributed task scheduling method based on message queue according to an embodiment of the present invention; A flow chart of a method for implementing distributed message processing based on message queues is provided in the example;

FIG. 5 is a flowchart of another method for implementing distributed message processing based on message queue according to an embodiment of the present invention; FIG.

FIG. 6 is a task processing node according to an embodiment of the present invention;

FIG. 7 is a scheduler according to an embodiment of the present invention;

FIG. 8 is a schematic task processing system according to an embodiment of the present invention. detailed description

The technical solutions in the embodiments of the present invention will be clearly and completely described in the following with reference to the accompanying drawings in the embodiments.

FIG. 1 is a structural diagram of a distributed task processing system according to an embodiment of the present invention. After receiving the original task, the task segmentation node segments the original task to obtain at least one processing task, and performs corresponding segmentation on the data file associated with the original task to obtain at least one fragment, so that each processing is performed. The tasks correspond to a slice that matches the processing task. After the task splitting node splits the original task, the split processing task is sent to the scheduler, and the scheduler sends the processing task to the task message queue group according to the type of the processing task. The message queue associated with the type of processing task is waiting for the task processing node to execute.

The dividing the data file refers to dividing the data file into at least one slice without processing the data file. It should be noted that the task splitting node provided by the embodiment of the present invention may be a dedicated hardware device, a task processing node in the distributed task processing system, or a scheduler.

It should be noted that, in the embodiment of the present invention, a task message queue group is taken as an example to describe the technical solution. Those skilled in the art may understand that the embodiment of the present invention may have one or more task message queue groups. The number of task message queue groups does not constitute a limitation on the embodiments of the present invention, and the following embodiments are also the same.

It should be further noted that, for the sake of clarity, only one task processing node is provided in the system, and those skilled in the art can understand that the embodiment of the present invention may include multiple task processing nodes, and the number of the task processing nodes. The present invention is not limited to the embodiments of the present invention, and the following embodiments are also the same.

It should be noted that the task processing node may be various types of servers or personal computers; the scheduler may be various types of servers or personal computers.

It should be noted that the task message queue group may be stored in a dedicated hardware device or may be stored in any storage node in the distributed task storage system.

Based on the distributed task processing system architecture shown in FIG. 1, an embodiment of the present invention provides a method for distributed task processing based on a message queue. As shown in Figure 2, the method includes:

S201: The task processing node determines, according to a priority level of the processing task carried by the task, a priority level of the message queue storing the processing task.

Each of the task message queue groups stores a processing task carried by the task processing node, for example, if the processing tasks carried by the task processing node include a video transcoding task, a video rendering task, and a person For the face recognition task, the task message queue group will include three message queues corresponding to the video transcoding task, the video rendering task, and the face recognition task;

It should be noted that, the task message queue group may be pre-configured, and the body of the task message queue group may be configured as a dedicated hardware device or any task processing node in the distributed task processing system; It should be noted that a global message queue group can also be pre-configured. Each of the message queues in the office message queue stores a processing task carried by the system, and the task processing node selects from the global message queue and the task processing node itself according to the type of the processing task carried by the task processing node. The message queue associated with the type of processing task, which constitutes the task message queue group;

The method for determining the priority of the processing task carried by the task processing node provided by the embodiment of the present invention may include any one of the following methods:

Manner 1: The task processing node determines a priority level of the processing task carried by the task processing node according to the service requirement; for example, the processing task carried by the task processing node includes a video transcoding task, a video rendering task, and a face recognition. The task, but the main processing task carried by the task processing node is a video transcoding task, followed by a video rendering task, and again a face recognition task, then the task processing node determines that the message queue with the highest priority level is a stored video transcoding task. Message queue, the second highest priority message queue is a message queue for storing video rendering tasks, and the low priority level message queue is a message queue for storing face recognition tasks;

Manner 2: The priority level of the processing task is set in advance, wherein, the entity that presets the priority level of the processing task may be a dedicated hardware device or the task processing. Node

S202: The task processing node sequentially scans the message queues in the task message queue group from the message queue with the highest priority level according to the priority level of the priority level. If the message queue containing the processing task is scanned, Step S203 is performed, otherwise, step S202 is repeated;

It should be noted that, as shown in FIG. 2, this step S202 can be further subdivided into four steps S202a, S202b, S202c, and S202d, as follows:

S202a: The task processing node scans a message queue with the highest priority level in the task message queue group;

For example, if the message queue with the highest priority level is the message queue storing the video transcoding task, the task processing node first scans the message queue of the stored video transcoding task; S202b: The task processing node determines whether the scanned message queue contains a processing task, and if so, step S203 is performed, otherwise, step S202c is performed;

S202c: The task processing node determines whether the priority level of the scanned message queue is the lowest, and if yes, returns to step S202a, otherwise, step S202d is performed;

It should be noted that, if the message queue currently scanned by the task processing node is the message queue with the lowest priority level, the task processing node rescans the message queue from the message queue with the highest priority level;

S202d: The task processing node scans the next message queue in the task message queue group according to the order of priority from high to low, and returns to step S202b;

It should be noted that, if the message queue with the highest priority level does not store the processing task, the task processing node scans the message queue of the secondary priority level according to the order of priority level from high to low;

It should be further noted that, when the task processing node first scans the message queue including the processing task, step S203 is performed;

S203: The task processing node acquires the current processing task, and executes the current processing task. After the execution, the process returns to step S202, where the current processing task is stored in the message queue that includes the processing task.

It should be noted that, in the manner that the task processing node obtains the current processing task and performs the current processing task, the method may include any one of the following methods:

Manner 1: The task processing node acquires any processing task that is not executed in the message queue that includes the processing task, and executes any processing task that is not executed; for example, if the message queue includes the processing task Include five processing tasks, randomly acquiring one processing task from the five processing tasks as a current processing task, and executing the randomly acquired current processing task;

Manner 2: the task processing node acquires a team head processing task that is not executed in the message queue that includes the processing task, and executes the unexecuted team head processing task; for example, if the processing is included The message queue of the task includes five processing tasks, and then the processing task located at the head of the message queue is selected as the current processing task, so that the same type of processing task is executed in the order of first in first out;

It should be further noted that, after the task processing node acquires the current processing task, the message queue including the current processing task deletes the current processing task.

A method for processing a distributed task based on a message queue is provided in the embodiment of the present invention. Different types of tasks are allocated in different message queues. Compared with the prior art, the embodiment of the present invention does not need to be in the message queue. The processing tasks are sorted in order of priority, which eliminates the system overhead caused by sorting and improves the efficiency of the system work. Secondly, compared with the prior art, a message queue only saves one type of processing task, and only needs to increase Message Queuing can handle other types of processing tasks, and business scalability is strong. As shown in FIG. 3, a message queue-based distributed task scheduling method according to an embodiment of the present invention includes:

S301: Obtain a processed task after the segmentation;

It should be noted that the main body executed by this embodiment is a scheduler;

It should be further noted that, after receiving the original task, the task splitting node splits the original task to obtain at least one processing task, and performs corresponding segmentation on the data file associated with the original task to obtain at least one fragment. Each processing task corresponds to a fragment matching the processing task, and after the task dividing node divides the original task, the processing task obtained after the segmentation is sent to the scheduler;

It should be further noted that the task splitting node may be the scheduler, and after the splitter splits the original task, directly acquiring the processed task after the splitting;

S302: Send, according to the type of the processing task, the processing task to a message queue associated with the type of the processing task in the task message queue group, and wait for the task processing node to execute;

Each of the task message queue groups stores a processing task carried by the task processing node, for example, if the processing task carried by the task processing node includes video transcoding. The task, the video rendering task, and the face recognition task, the task message queue group will include three message queues, corresponding to the video transcoding task, the video rendering task, and the face recognition task;

It should be noted that the task message queue group may be configured in advance, and the body of the task message queue group may be configured as a dedicated hardware device or any task processing node in the distributed task processing system;

It should be noted that a global message queue group may also be pre-configured, and each message queue in the global message queue group stores a processing task carried by the system, and the task processing node according to the type of processing task carried by the task processing node And selecting, from the global message queue, a message queue associated with the type of the processing task carried by the task processing node itself, to form the task message queue group.

A message queue-based distributed task scheduling method is provided in the embodiment of the present invention. Different types of tasks are allocated in different message queues. Compared with the prior art, the embodiment of the present invention does not need to process the message queue. Tasks are sorted in order of priority, eliminating system overhead caused by sorting and improving the efficiency of system work. Secondly, compared with the prior art, a message queue only stores one type of processing task, and only needs to add messages. The queue can handle other types of processing tasks, and the business is highly scalable. As shown in FIG. 4, a flow chart of a method for implementing distributed task processing based on message queues according to an embodiment of the present invention includes:

S401: The task splitting node receives the original task;

S402: The task segmentation node divides the original task to obtain at least one processing task, and performs corresponding segmentation on the data file associated with the original task to obtain at least one fragment, so that each processing task corresponds to one and the processing a shard that matches the task;

For example, the task segmentation node divides the original task into three processing tasks, and the task segmentation node divides the data file corresponding to the original task into segments corresponding to the three processing tasks. Each slice corresponds to a processing task;

Among them, it should be noted that each processed task after segmentation carries a complete task schedule. The information and the execution information can be executed smoothly after the task processing node obtains the processing task. Here, it should be noted that the embodiment of the present invention does not limit the specific expression of the processing task saved in the message queue, as an example. Not limited, a video transcoding task can be expressed in the following form:

{

"userid": "UserlD",

"userkey": "UserKey",

"action": "Action",

"jobid": "JobID",

"subjobid": "SubJobID",

"jobdescribe": "describe",

"SLAtime": "SLA time",

"source": ["SourceFile", "SourceFile2", ... , "SourceFileN"],

"sourcerun": ["SourceRunl", "SourceRun2", ... , "SourceRunN"], "streamtype":["typel", "type2",. . . "typeN"],

"notify": "NotifyURL",

"splitduration": "Split duration",

"splitsize": "Split size",

"splitaudio": "yes|no",

"sourceduration": "sourceduration",

"iserror": "errorcode",

"format":

[

{

"output": "Output format",

"destination": "DestFile",

"video_codec": "Video Codec",

"audio codec": "Audio Codec", "bitrate": "Video bitrate",

"framerate": "Frame rate",

"audio-bitrate": "Audio bitrate",

"audio_sample- rate": "Audio quality",

"audio_ volume": "Volume",

"size": "Size",

"crop- left": "Crop Left",

"crop- top": "Crop Top",

"crop-right": "Crop Right",

"crop_ bottom": "Crop Bottom",

"keep- aspect- ratio": "yes/no",

"set- aspect- ratio": "ASPECT- RATIO",

"add_meta": "yes/no",

"hint": "yes/no",

"rc_init_occupation": "RC Occupancy", "minrate": "Min Rate",

"maxrate": "Max Rate",

"bufsize": "RC Buffer Size",

"keyframe": "Keyframe Period (GOP)", "start": "Start From",

"duration": "Result Duration",

"fragment_ duration": "Fragment Duration' "quality_speed_balance": "balance", "metadata":

{

"title": "Title",

"copyright": "Copyright",

"author": "Author",

"description": "Description", "album": "Album"

},

"profile": "high/main/baseline",

"turbo": "yes/no",

}

]

}

It should be noted that the task splitting node provided by the embodiment of the present invention may be a dedicated hardware device, a task processing node in a distributed task processing system, or a scheduler.

It should be further noted that the embodiment of the present invention does not limit the method for dividing the original task. As an example and not by way of limitation, the original task may be divided into processing tasks according to a specific format of the data file associated with the original task, and Perform corresponding segmentation on the data file;

S403: The task splitting node sends the split processing task to the scheduler.

S404: After receiving the processing task, the scheduler pushes the processing task to a message queue associated with the type of the processing task in the task message queue group according to the type of the processing task, and waits for the task processing node to process;

It should be noted that a global message queue group may also be pre-configured, and each message queue in the global message queue group stores a processing task carried by the system, and the task processing node is configured according to A message queue associated with the type of the processing task carried by the task processing node itself is selected from the global message queue to form the task message queue group;

S405: The task processing node determines, according to a priority level of the processing task carried by the task, a priority level of the message queue storing the processing task.

It should be noted that, in the manner that the task processing node according to the embodiment of the present invention determines the priority of the processing task carried by the task processing node, the method may include any one of the following methods:

It should be further noted that the step S405 can be performed before, after or simultaneously with any of the steps S401 to S404, and there is no sequence;

S406: The task processing node sequentially scans the message queues in the task message queue group from the message queue with the highest priority level according to the priority level of the priority level. If the message queue containing the processing task is scanned, Step S407 is performed, otherwise, step S406 is repeated;

It should be noted that, as shown in FIG. 4, this step S406 can be further subdivided into four steps S406a, S406b, S406c, and S406d, as follows:

S406a: The task processing node scans a message queue with the highest priority level in the task message queue group; For example, if the message queue with the highest priority level is the message queue storing the video transcoding task, the task processing node first scans the message queue of the stored video transcoding task;

S406b: The task processing node determines whether the scanned message queue contains a processing task, and if so, step S407 is performed, otherwise, step S406c is performed;

S406c: The task processing node determines whether the priority level of the scanned message queue is the lowest, and if yes, returns to step S406a, otherwise, step S406d;

S406d: The task processing node scans the next message queue in the task message queue group according to the order of priority from high to low, and returns to step S406b;

It should be further noted that, when the task processing node first scans the message queue including the processing task, step S407 is performed;

S407: The task processing node acquires the current processing task, and executes the current processing task. After the execution, the process returns to step S406, where the current processing task is stored in the message queue that includes the processing task.

Manner 1: The task processing node acquires any processing task that is not executed in the message queue that includes the processing task, and executes any processing task that is not executed; for example, if the message queue includes the processing task Include five processing tasks, randomly acquiring one processing task from the five processing tasks as a current processing task, and executing the randomly acquired current processing task; Manner 2: the task processing node acquires an unexecuted team head processing task in the message queue that includes the processing task, and executes the unexecuted team head processing task; for example, if the message queue including the processing task is included Including five processing tasks, the processing task located at the head of the message queue is selected as the current processing task, so that the same type of processing tasks are executed in the first-in first-out order;

A method for processing a distributed task based on a message queue is provided in the embodiment of the present invention. Different types of processing tasks are allocated in different message queues. Compared with the prior art, the embodiment of the present invention does not need to be in a message queue. The processing tasks are sorted in order of priority, which eliminates the system overhead caused by sorting and improves the efficiency of the system work. Secondly, compared with the prior art, a message queue only stores one type of processing tasks, only Adding a message queue can handle other types of processing tasks, and the service scalability is strong. In addition, the priority of the processing task can be dynamically determined by the task processing node according to the business needs, ensuring that the emergency service can be obtained higher during the entire task processing process. The priority of the task processing node is more flexible, and again, the original task is segmented to obtain at least one processing task, so that the current processing task does not occupy the resources of the task processing node for a long time, and the resources of the task processing node are not guaranteed. Blocking, able to execute high-quality in time The first level of processing tasks. As shown in FIG. 5, it is a flow chart of a method for implementing distributed message processing based on a message queue according to an embodiment of the present disclosure. The embodiment is applied to a distributed video task processing system, and specifically includes:

S501: The task splitting node receives the original task. In this embodiment, the original task is an original video task.

S502: The task splitting node splits the original video task to obtain at least one processing task, and performs corresponding segmentation on the data file associated with the original video task to obtain at least one fragment, so that each processing task corresponds to one The processing task is matched with the fragmentation. In this embodiment, the processing task is a video processing task.

For example, the task splitting node divides the original video task into three video processing tasks. Each of the three video processing tasks corresponds to a video processing task; wherein, each of the video processing tasks after the segmentation carries complete task scheduling information and execution information, So that the task processing node can smoothly execute after acquiring the video processing task;

It should be noted that the embodiment of the present invention does not limit the specific expression of the video processing task stored in the message queue. As an example and not by way of limitation, a video transcoding task may be in the following form:

{

"userid": "UserlD",

"userkey": "UserKey",

"action": "Action",

"jobid": "JobID",

"subjobid": "SubJobID",

"jobdescribe": "describe",

"SLAtime": "SLA time",

"source": ["SourceFile", "SourceFile2", ... , "SourceFileN"],

"sourcerun": ["SourceRunl", "SourceRun2", ... , "SourceRunN"],

"streamtype":["typel", "type2",. . . "typeN"],

"notify": "NotifyURL",

"splitduration": "Split duration",

"splitsize": "Split size",

"splitaudio": "yes|no",

"sourceduration": "sourceduration",

"iserror": "errorcode",

"format":

[

{ "output": "Output format",

"destination": "DestFile",

"video_codec": "Video Codec",

"audio_codec": "Audio Codec",

"bitrate": "Video bitrate",

"framerate": "Frame rate",

"audio-bitrate": "Audio bitrate",

"audio_sample- rate": "Audio quality",

"audio_ volume": "Volume",

"size": "Size",

"crop- left": "Crop Left",

"crop- top": "Crop Top",

"crop-right": "Crop Right",

"crop_ bottom": "Crop Bottom",

"keep- aspect- ratio": "yes/no",

"set- aspect- ratio": "ASPECT- RATIO",

"add_meta": "yes/no",

"hint": "yes/no",

"rc_init_occupation": "RC Occupancy", "minrate": "Min Rate",

"maxrate": "Max Rate",

"bufsize": "RC Buffer Size",

"keyframe": "Keyframe Period (GOP)", "start": "Start From",

"duration": "Result Duration",

{ "title": "Title",

"copyright": "Copyright",

"author": "Author",

"description": "Description",

"album": "Album"

},

"profile": "high/main/baseline",

"turbo": "yes/no",

}

]

}

It should be noted that, the manner in which the task splitting node splits the original task specifically includes: the task splitting node splits the original video task according to a preset splitting strategy to obtain at least one video processing task, and is in an I frame. Separating the video data file associated with the original video task to obtain at least one fragment, wherein the segmentation strategy needs to ensure that the video data file is segmented at the position of the I frame;

It should be noted that the I frame is the first frame in the Group Of Pictures (GOP), and can be used as a reference point for random access;

It should be further noted that the task splitting node provided by the embodiment of the present invention may be a dedicated hardware device, a task processing node in a distributed task processing system, or a scheduler.

S503: The task segmentation node sends the segmented video processing task to the scheduler.

S504: After receiving the video processing task, the scheduler pushes the video processing task to a message queue associated with the type of the video processing task in the task message queue group according to the type of the video processing task, and waits for task processing. Node processing

Each of the task message queues stores a video processing task carried by the task processing node, for example, if the video processing task carried by the task processing node includes The frequency transcoding task, the video rendering task, and the face recognition task, the task message queue group will include three message queues corresponding to the video transcoding task, the video rendering task, and the face recognition task;

It should be noted that a global message queue group may also be pre-configured, and each message queue in the global message queue group stores a video processing task carried by the system, and the task processing node processes the video according to its own bearer. a task type, selecting, from the global message queue, a message queue associated with a type of a video processing task carried by the task processing node itself, to form the task message queue group;

S505: The task processing node determines, according to a priority level of the video processing task carried by the task processing node, a priority level of the message queue for storing the video processing task.

It should be noted that the manner in which the task processing node determines the priority of the video processing task carried by the task processing node provided by the embodiment of the present invention may include any one of the following methods:

Manner 1: The task processing node determines a priority level of the video processing task carried by the task processing node according to the service requirement; for example, the video processing task carried by the task processing node includes a video transcoding task, a video rendering task, and a person The face recognition task, but the main video processing task carried by the task processing node is a video transcoding task, followed by a video rendering task, and again a face recognition task, then the task processing node determines that the message queue with the highest priority level is a stored video. The message queue of the transcoding task, the message queue of the second highest priority level is a message queue for storing the video rendering task, and the message queue of the low priority level is a message queue for storing the face recognition task;

Manner 2: The priority level of the video processing task is set in advance, wherein, the entity that presets the priority level of the video processing task may be a dedicated hardware device, or may be Task processing node;

It should be further noted that this step S505 can be any of steps S501 to S504. Before, after or at the same time, there is no order;

S506: The task processing node sequentially scans the message queue in the task message queue group from the message queue with the highest priority level according to the priority level from high to low, and scans the message queue that includes the video processing task. , step S507 is performed, otherwise, step S506 is repeated;

It should be noted that, as shown in FIG. 5, this step S506 can be further subdivided into S506a,

S506b, S506c, S506d four steps, as follows:

S506a: The task processing node scans a message queue with the highest priority level in the task message queue group;

For example, if the message queue with the highest priority level is the message queue storing the video transcoding task, the task processing node first scans the message queue of the stored video transcoding task;

S506b: The task processing node determines whether the scanned message queue contains a video processing task, and if so, step S507 is performed, otherwise, step S506c is performed;

S506c: The task processing node determines whether the priority level of the scanned message queue is the lowest, and if yes, returns to step S506a, otherwise, step S506d;

S506d: The task processing node scans the next message queue in the task message queue group according to the order of priority from high to low, and returns to step S506b;

It should be noted that, if the message queue with the highest priority level does not store the video processing task, the task processing node scans the message queue of the secondary priority level according to the order of priority level from high to low;

It should be further noted that, when the task processing node first scans the message queue including the video processing task, step S507 is performed;

S507: The task processing node acquires a current processing task, and executes the current processing task. After the execution, the process returns to step S506. In the embodiment of the present invention, the current processing task is a current video processing task, and the current video task is stored in the message queue including the video processing task. The manner in which the task processing node obtains the current video processing task and performs the current video processing task may include any one of the following methods:

Manner 1: The task processing node acquires any video processing task that is not executed in the message queue that includes the video processing task, and executes any video processing task that is not executed; for example, if the processing task is included The message queue includes five video processing tasks, and randomly acquires one video processing task from the five video processing tasks as a current video processing task, and executes the randomly acquired current video processing task;

Manner 2: The task processing node acquires a team head video processing task that is not executed in the message queue that includes the video processing task, and executes the unexecuted team head video processing task; for example, if the video processing is included The message queue of the task includes five video processing tasks, and then the video processing task located at the head of the message queue is selected as the current video processing task, so that the same type of video processing task is executed in the first-in first-out order;

It should be further noted that, after the task processing node acquires the current video processing task, the message queue including the current video processing task deletes the current video processing task.

A method for processing a distributed task based on a message queue is provided in the embodiment of the present invention. Different types of video processing tasks are allocated in different message queues. Compared with the prior art, the embodiment of the present invention does not need to use a message queue. The video processing tasks are sorted in priority order, which eliminates the system overhead caused by sorting and improves the efficiency of the system work. Secondly, compared with the prior art, a message queue only stores one type of video processing task. Only need to increase the message queue to handle other types of video processing tasks, and the service scalability is strong. In addition, the priority of the video processing task can be dynamically determined by the task processing node according to the business needs, ensuring emergency in the whole task processing process. Video services can get higher priority, making the task processing node more flexible in processing video tasks. The initial video task is segmented to obtain at least one video processing task, so that the current video processing task does not occupy the resources of the task processing node for a long time, ensures that the resources of the task processing node are not blocked, and can perform high-priority video processing tasks in time. As shown in FIG. 6, a task processing node according to an embodiment of the present invention includes: a message queue priority level determining unit 601, configured to determine, according to a priority level of a processing task carried by itself, to store the processing task. The priority level of the message queue;

The scanning unit 602 is configured to scan according to the priority level of the message queue determined by the message queue priority level determining unit 601, in order of priority level from highest to lowest, starting from the message queue with the highest priority level. a message queue in the task message queue group;

The task execution unit 603 is configured to: when the scanning unit 602 scans a message queue that includes a processing task, acquire a current processing task, and execute the current processing task, where the current processing task is stored in the included processing task In the message queue.

It should be noted that, the task processing node provided by the embodiment of the present invention further includes: a processing task priority level determining unit, configured to determine, according to the service requirement, a priority level of the processing task carried by the task processing node, or pre-set The priority level of the processing task is described, so that the message queue priority level determining unit 601 determines the priority level of the message queue in the task message queue group.

It should be noted that the task processing node provided by the embodiment of the present invention further includes:

a segmentation unit, configured to segment the original task to obtain at least one processing task, and perform corresponding segmentation on the data file corresponding to the original task to obtain at least one fragment, so that each processing task corresponds to the processing task a matching fragment, so that the scheduler sends the processing task to a message queue associated with the type of the processing task in the task message queue group according to the type of the processing task, so that the scanning unit 602 scans the segmentation Processing tasks.

It should be noted that the original task is an original video task, the processing task is a video processing task, and the task segmentation unit is specifically configured to: use a preset segmentation policy to the original video task. Performing segmentation to obtain at least one video processing task, and segmenting the video data file associated with the original video task at the position of the I frame to obtain at least one slice, such that each video processing task corresponds to one of the video processing tasks. Matching fragments;

The segmentation policy ensures that the video data file is segmented at the location of the I frame.

It should be further noted that, when the scanning unit scans the message queue that includes the processing task, the task execution unit is specifically configured to acquire any processing task that is not executed in the message queue that includes the processing task, and execute Any of the processing tasks not performed; or,

Specifically, the team leader processing task that is not executed in the message queue that includes the processing task is acquired, and the unexecuted team head processing task is executed.

In the task processing node provided by the embodiment of the present invention, the message queue priority level determining unit determines the priority of the message queue for storing the processing task, and does not need to sort the processing tasks according to the priority, and eliminates The system overhead caused by the sorting improves the efficiency of the system work. Secondly, the priority of the processing task can be dynamically determined by the processing task priority level determining unit according to the business needs, ensuring that the urgent business can be performed during the entire task processing process. Obtaining a higher priority makes the task processing node more flexible in processing the video task. In addition, the segmentation unit splits the original task to obtain at least one video processing task, so that the current processing task does not occupy the resources of the task processing node for a long time, thereby ensuring The resources of the task processing node are not blocked, and high-priority processing tasks can be executed in time. As shown in FIG. 7, a scheduler according to an embodiment of the present invention includes:

The obtaining unit 701 is configured to obtain the processed task after the segmentation;

The sending unit 702 is configured to send, according to the type of the processing task acquired by the acquiring unit 701, the processing task to a message queue associated with the type of the processing task in the task message queue group, and wait for the task processing node to execute.

The scheduler provided by the embodiment of the present invention further includes:

a dividing unit, configured to split the original task to obtain at least one processing task, and to the original The data file corresponding to the initial task is segmented to obtain at least one fragment, so that each processing task corresponds to a fragment matching the processing task, so that the obtaining unit 701 acquires the processing after the segmentation unit is divided. task.

It should be noted that, the original task is an original video task, and the processing task is a video processing task, and the dividing unit is specifically configured to segment the original video task according to a preset segmentation strategy to obtain at least one video processing task, and Segmenting the video data file associated with the original video task at a position of the I frame to obtain at least one slice, such that each video processing task corresponds to a slice matching the video processing task;

According to the embodiment of the present invention, the sending unit sends the processing task to the message queue associated with the type of the processing task in the task message queue group. Compared with the prior art, the video in the message queue is not needed. The processing tasks are sorted in order of priority, which eliminates the system overhead caused by sorting and improves the efficiency of the system work. Secondly, compared with the prior art, a message queue only stores one type of video processing task, only Adding a message queue can handle other types of video processing tasks, and the service is highly scalable. In addition, the splitting unit splits the original task to obtain at least one video processing task, so that the current processing task does not occupy the resources of the task processing node for a long time, ensuring The resources of the task processing node are not blocked, and high-priority processing tasks can be executed in time. The embodiment of the present invention further provides a distributed task processing system, including at least one task processing node, where the task processing node is configured to determine, according to a priority level of a processing task carried by the task, a message queue for storing the processing task. Priority level, and in order from highest to lowest priority level, the message queue in the task message queue group is cyclically scanned from the message queue with the highest priority level, and the scanned processing task is executed, where the task is performed. Each message queue in the message queue group stores a processing task carried by the task processing node;

The task processing node includes:

a message queue priority level determining unit for prioritizing processing tasks according to its own No, determining a priority level of a message queue storing the processing task;

A distributed task processing system is provided in the embodiment of the present invention. Different types of tasks are allocated in different message queues. Compared with the prior art, the embodiment of the present invention does not need to prioritize processing tasks in the message queue. Sorting in order eliminates the overhead caused by sorting and improves the efficiency of the system. Secondly, compared with the prior art, a message queue only stores one type of processing task, and only needs to increase the message queue to process Other types of processing tasks, business scalability. The embodiment of the present invention further provides a distributed task processing system, where the system includes a scheduler 801, a task message queue group storage node 802, and a task processing node 803, as shown in FIG. Obtaining the processed processing task, and sending the processing task to the message queue associated with the type of the processing task in the task message queue group stored by the task message queue group storage node 802 according to the type of the processing task Waiting for the task processing node to execute;

The task message queue group storage node 802 is configured to store the task message queue group; the task processing node 803 is configured to determine, according to a priority level of a processing task carried by the host, a message queue for storing the processing task. a priority level, and sequentially scanning the message queues in the task message queue group stored by the task message queue group storage node 802 in order from the highest priority message queue in the order of priority level, and executing The scanned processing task, wherein each message queue in the task message queue group stores a processing task carried by the task processing node 803.

It should be noted that the task processing node 803 includes: a message queue priority level determining unit, configured to determine a priority level of a message queue storing the processing task according to a priority level of a processing task carried by the self-host;

The scheduler 801 includes:

a sending unit, configured to send, according to the type of the processing task acquired by the acquiring unit, the processing task to a message associated with the type of the processing task in the task message queue group stored by the task message queue group storage node 802 Queue, waiting for the task processing node to execute.

It should be noted that the task message queue group storage node 802 and the task processing node 803 may be the same node.

It should be noted that, in the embodiment of the present invention, the number of task processing nodes is not limited, and there may be multiple task processing nodes.

A method, a device, and a system for processing a distributed task based on a message queue are provided in the embodiments of the present invention. Different types of video processing tasks are allocated in different message queues. Compared with the prior art, the embodiment of the present invention does not. The video processing tasks in the message queue need to be sorted in order of priority, which eliminates the system overhead caused by the sorting and improves the efficiency of the system work. Secondly, compared with the prior art, one message queue only stores one type. The video processing task can process other types of video processing tasks by adding a message queue, and the service scalability is strong. In addition, the priority of the video processing task can be dynamically determined by the task processing node according to the business needs, ensuring the processing of the entire task. The urgent video service in the process can obtain higher priority, making the task processing node more flexible in processing video tasks. Then, the original video task is segmented to obtain at least one video processing task, so that the current video processing task does not occupy the resources of the task processing node for a long time, and the resources of the task processing node are not blocked, and the high priority video can be executed in time. Handling tasks.

In summary, the above description is only a preferred embodiment of the technical solution of the present invention, and is not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and scope of the present invention are intended to be included within the scope of the present invention.

Claims

Claim

A message queue-based distributed task processing method, characterized in that each message queue in the task message queue group stores a processing task carried by the task processing node, and the method includes:

B. The task processing node sequentially scans the message queue in the task message queue group from the message queue with the highest priority level according to the priority level from high to low. If the message queue containing the processing task is scanned, Perform step C, otherwise, repeat step B;

2. The method according to claim 1, further comprising:

The task processing node determines a priority level of the processing task carried by the task according to the service requirement, or sets a priority level of the processing task in advance.

The method according to claim 1 or 2, further comprising:

After receiving the processing task, the scheduler sends the processing task to a message queue associated with the type of the processing task according to the type of the processing task.

The method according to claim 3, wherein, before receiving the processing task, the scheduler further includes:

Dividing the original task to obtain at least one processing task, and performing corresponding segmentation on the data file associated with the original task to obtain at least one fragment, so that each processing task corresponds to a fragment matching the processing task. .

The method according to claim 4, wherein the original task is an original video task, and the processing task is a video processing task;

Bay' J, the segmentation of the original task to obtain at least one processing task, and with the original The associated data file is segmented to obtain at least one fragment, so that each processing task corresponds to a fragment matching the processing task, including:

And dividing the original video task according to a preset segmentation strategy to obtain at least one video processing task, and segmenting the video data file associated with the original video task at the position of the I frame to obtain at least one slice, so that each The video processing tasks all correspond to a slice that matches the video processing task;

The method according to any one of claims 1 to 5, wherein the task processing node acquires the current processing task and executes the current task, including:

The task processing node acquires any processing task that is not executed in the message queue that includes the processing task, and executes any processing task that is not executed; or

The task processing node acquires a team head processing task that is not executed in the message queue that includes the processing task, and executes the unexecuted team head processing task.

A message queue-based distributed task scheduling method, the method comprising: obtaining a divided processing task;

The method according to claim 7, wherein before the obtaining the processed task after the segmentation, the method further includes:

The method according to claim 8, wherein the original task is an original video task, and the processing task is a video processing task;

10. A task processing node, comprising:

The task processing node according to claim 10, further comprising: a processing task priority level determining unit, configured to determine a priority level of the processing task carried by the task processing node according to the service requirement, or preset And processing the priority level of the task, so that the message queue priority level determining unit determines a priority level of the message queue in the task message queue group.

The task processing node according to claim 10 or 11, further comprising: a segmentation unit, configured to segment the original task to obtain at least one processing task, and corresponding to the data file corresponding to the original task Segmentation results in at least one slice such that each processing task corresponds to a slice that matches the processing task, so that the scheduler is based on the class of the processing task The type sends the processing task to a message queue associated with the type of the processing task in the task message queue group, such that the scanning unit scans the divided processing tasks.

The task processing node according to claim 12, wherein the task segmentation unit is specifically configured to segment the original video task according to a preset segmentation strategy to obtain at least one video processing task, and at the position of the I frame. Segmenting the video data file associated with the original video task to obtain at least one slice, such that each video processing task corresponds to a slice matching the video processing task;

The task processing node according to any one of claims 10 to 13, wherein when the scanning unit scans a message queue including a processing task, the task execution unit is specifically configured to acquire the inclusion processing. Any processing task that is not executed in the message queue of the task, and executes any of the processing tasks that are not performed; or

15. A scheduler, comprising:

The scheduler according to claim 15, further comprising:

a segmentation unit, configured to segment the original task to obtain at least one processing task, and perform corresponding segmentation on the data file corresponding to the original task to obtain at least one fragment, so that each processing task corresponds to the processing task Matching the fragment, so that the acquiring unit acquires the processing task after the dividing unit is divided.

The scheduler according to claim 16, wherein the dividing unit is specifically used And dividing the original video task according to the preset segmentation strategy to obtain at least one video processing task, and segmenting the video data file associated with the original video task at the position of the I frame to obtain at least one fragment, so that each video The processing tasks all correspond to a slice matching the video processing task;

A distributed task processing system, wherein the system includes at least one task processing node, and the task processing node is configured to determine, according to a priority level of a processing task carried by the task, a message storing the processing task. The priority level of the queue, and the message queues in the task message queue group are cyclically scanned from the message queue with the highest priority level in order of priority level, and the scanned processing tasks are executed, where Each message queue in the task message queue group stores a processing task carried by the task processing node;

The task processing node includes:

19. The system of claim 18, further comprising:

a task message queue group storage node, configured to store the task message queue group;

a scheduler, configured to acquire the processed processing task, and send the processing task to a task message queue group stored in the task message queue group storage node and a type of the processing task according to the type of the processing task An associated message queue, waiting for the task processing node to execute; The scheduler includes:

a sending unit, configured to send, according to the type of the processing task acquired by the acquiring unit, the processing task to a message queue associated with the type of the processing task in the task message queue group stored by the task message queue group storage node Wait for the task processing node to execute.