CN108304255A - Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing - Google Patents
Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing Download PDFInfo
- Publication number
- CN108304255A CN108304255A CN201711486482.6A CN201711486482A CN108304255A CN 108304255 A CN108304255 A CN 108304255A CN 201711486482 A CN201711486482 A CN 201711486482A CN 108304255 A CN108304255 A CN 108304255A
- Authority
- CN
- China
- Prior art keywords
- task
- node
- execution
- scheduling
- goal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0796—Safety measures, i.e. ensuring safe condition in the event of error, e.g. for controlling element
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/202—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where processing functionality is redundant
- G06F11/2023—Failover techniques
- G06F11/203—Failover techniques using migration
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Software Systems (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
A kind of distributed task dispatching method and device of offer of the embodiment of the present invention, electronic equipment and readable storage medium storing program for executing.This method includes：Host node receives multiple tasks execution node and pulls request with the task that predeterminated frequency is sent, host node pulls request according to received task and assigns tasks to goal task execution node, it is a task execution node in multiple tasks execution node that goal task, which executes node, host node receives goal task and executes the task action result that node is sent after executing distributed task, and goal task is executed to the task execution node that task queue in the execution of node is transferred to other survivals when perceiving goal task execution node and breaking down.To can avoid Single Point of Faliure, the stability and availability of system are improved.
Description
Technical field
The present embodiments relate to distributed scheduling technical field more particularly to a kind of distributed task dispatching method and dresses
It sets, electronic equipment and readable storage medium storing program for executing.
Background technology
In operation system, it can generally go periodically to handle some batch jobs (task) using background task, to carry
It rises user experience and increases the scalability of system, therefore distributed task dispatching is widely used.
Quartz is commonly used in the prior art as distributed task dispatching frame, and there are two types of occupation modes by Quartz：It is a kind of
It is that task is integrated into operation system, executes periodically or periodically the good task of some configured in advance, task details, triggering
The information storages such as time are in local memory, and only there are one nodes for synchronization in the task of execution, to realize distributed task scheduling
Scheduling, which there are Single Point of Faliure, do not support failure to shift the shortcomings of.Another kind is the clustering deployment of Quartz, multiple
Task details are stored in database or other distributed memory systems, by all Quartz task process (also referred to as node)
Shared, system availability is not high under which, is communicated by shared data bank between all Quartz nodes, is encountering network
When failure, each Quartz nodes think that other Quartz nodes are off-line state, to entire to take over as host node
System, therefore will appear " fissure " phenomenon.
Invention content
A kind of distributed task dispatching method and device of offer of the embodiment of the present invention, electronic equipment and readable storage medium storing program for executing,
Single Point of Faliure is avoided the occurrence of, the stability and availability of system are improved.
In a first aspect, the embodiment of the present invention provides a kind of distributed task dispatching method, including：
Host node receives multiple tasks execution node and pulls request with the task that predeterminated frequency is sent, and the host node is
Zookeeper clusters are registered in multiple tasks scheduling node and the multiple task execution node on Zookeeper clusters
Later, the task scheduling node selected from the multiple task scheduling node according to preset rules；
The host node pulls request according to received task and assigns tasks to goal task execution node, the target
Task execution node is a task execution node in the multiple task execution node；
The host node receives the goal task and executes the task execution that node is sent after executing distributed task
As a result, the task in perceiving the goal task and executing the execution that the goal task is executed to node when node breaks down
Queue is transferred to the task execution node of other survivals.
Optionally, the preset rules are：Using the minimum task scheduling node of sequential number as host node；It is led currently
When node delay machine, using the minimum task scheduling node of sequential number in remaining task scheduling node as new host node.
Optionally, the method further includes：
For the host node when the task action result is failure, being determined whether according to the task configuration information can be into
Row task retries；
The host node retries if it is determined that task can be carried out, then according in the task configuration information can number of retries into
Row task retries；
The host node if it is determined that be not available for task retry or residue can number of retries be 0 when, determine that current task is held
Row failure.
Optionally, the method further includes：
Host node calling task scheduling system basic service module when determining that current task executes failure is reported
It is alert.
Second aspect, the embodiment of the present invention provide a kind of distributed task dispatching device, the distributed task dispatching dress
Zookeeper clusters are set to be registered on Zookeeper clusters in multiple tasks scheduling node and multiple tasks execution node
Later, the task scheduling node selected from the multiple task scheduling node according to preset rules, described distributed
Business dispatching device include：
Receiving module pulls request for receiving the multiple task execution node with the task that predeterminated frequency is sent；
Task allocating module assigns tasks to goal task execution node for pulling request according to received task,
It is a task execution node in the multiple task execution node that the goal task, which executes node,；
The receiving module is additionally operable to：It receives the goal task and executes what node was sent after executing distributed task
Task action result；
Processing module, for executing the goal task when perceiving the goal task and executing node failure
Task queue is transferred to the task execution node of other survivals in the execution of node.
Optionally, the preset rules are：
Using the minimum task scheduling node of sequential number as host node；
In current primary node delay machine, using the minimum task scheduling node of sequential number in remaining task scheduling node as
New host node.
Optionally, described device further includes：
Task retries module, for when the task action result is failure, being determined according to the task configuration information
Task whether can be carried out to retry；
Retried if it is determined that task can be carried out, then according in the task configuration information can number of retries carry out task weight
Examination；
If it is determined that be not available for task retry or residue can number of retries be 0 when, determine that current task executes failure.
Optionally, described device further includes：
Alarm module, for the calling task scheduling system basic service module progress when determining that current task executes failure
Alarm.
The third aspect, the embodiment of the present invention provide a kind of distributed task dispatching system, including：
Zookeeper clusters, multiple tasks scheduling node and multiple tasks execute node, and each task scheduling node is noted
On volume to the Zookeeper clusters, each task execution node is registered on the Zookeeper clusters；
The Zookeeper clusters are used to select a main section from the multiple task scheduling node according to preset rules
Point；
The multiple task execution node, which is used to send task to the host node with predeterminated frequency, pulls request；
The host node, which is used to pull request according to received task, assigns tasks to goal task execution node, described
It is a task execution node in the multiple task execution node that goal task, which executes node,；
The goal task executes node for executing distributed task, and reports task execution knot to the host node
Fruit；
The host node is additionally operable to receive the task action result, and node appearance is executed perceiving the goal task
The goal task is executed to the task execution node that task queue in the execution of node is transferred to other survivals when failure.
Optionally, the preset rules are：After the completion of the multiple task scheduling Node registry, sequential is numbered minimum
Task scheduling node as host node；
In current primary node delay machine, using the minimum task scheduling node of sequential number in remaining task scheduling node as
New host node.
Optionally, the host node is additionally operable to：When the task action result is failure, confidence is matched according to the task
Breath determines whether that can carry out task retries；
Retried if it is determined that task can be carried out, then according in the task configuration information can number of retries carry out task weight
Examination；
If it is determined that be not available for task retry or residue can number of retries be 0 when, determine that current task executes failure.
Optionally, further include：Back Administration Module, for according to the operation requests that receive carry out task configuration management or
Show current task action result；
First database, for the configurations information of store tasks scheduling, the configurations information of the task scheduling
Configuration information including task configuration information and the multiple task execution node；
Second database is used for the execution queue of store tasks burst information details and task, the execution team of the task
Row include that can perform task queue in task queue, the execution and retry queue；
Task scheduling system infrastructure service module, for providing unification for the Back Administration Module and the host node
Reading and writing data service.
Optionally, further include：Alarm module, the alarm module is registered on the Zookeeper clusters, for monitoring
The off-line state of the multiple task scheduling node and the multiple task execution node is determining task scheduling node off-line
When number is more than first threshold or when task execution node off-line number is more than second threshold, the task scheduling system basis is called
Service module is alarmed；
The task scheduling system infrastructure service module is additionally operable to provide reading and writing data service for the alarm module；
The first database is additionally operable to storage alarm configuration information, and the alarm configuration information includes alert receipt people
Location, the first threshold and the second threshold.
Optionally, the host node is additionally operable to：The task scheduling system is called when determining that current task executes failure
Infrastructure service module is alarmed.
Optionally, the task scheduling system infrastructure service module includes：
Task dispensing unit, the configurations information for providing the task scheduling read and write interface；
Task alarm unit reads and writees interface and alert process for provide the alarm configuration information；
Task queue read-write cell, the execution queue for providing the task read and write interface；
Task action result reporting unit reads and writees interface for provide task action result.
Fourth aspect, the embodiment of the present invention provide a kind of electronic equipment, including：
Memory, for storing program instruction；
Processor, for calling and executing the program instruction in the memory, to realize the distribution described in first aspect
Formula method for scheduling task.
5th aspect, the embodiment of the present invention provide a kind of readable storage medium storing program for executing, which is characterized in that the readable storage medium storing program for executing
In be stored with computer program, when distributed task dispatching device at least one processor execute the computer program when,
Distributed task dispatching device executes the distributed task dispatching method described in first aspect.
Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing provided in an embodiment of the present invention, often
A task scheduling node is registered on Zookeeper clusters, and each task execution node is registered to Zookeeper clusters
On, ensure that all task scheduling nodes and task execution node are temporarily orderly, Zookeeper clusters are according to preset rules from more
A host node is selected in a task scheduling node, avoids the occurrence of " fissure " phenomenon, and host node is perceiving goal task execution
Goal task is executed to the task execution node that task queue in the execution of node is transferred to other survivals when node breaks down,
Single Point of Faliure is can avoid, and using the high availability of Zookeeper clusters, to improve the stability and availability of system.
Description of the drawings
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is this hair
Some embodiments of bright embodiment, for those of ordinary skill in the art, without having to pay creative labor,
Other drawings may also be obtained based on these drawings.
Fig. 1 is a kind of flow chart of distributed task dispatching embodiment of the method provided in an embodiment of the present invention；
Fig. 2 is a kind of flow chart of distributed task dispatching embodiment of the method provided in an embodiment of the present invention；
Fig. 3 is a kind of interaction diagrams of distributed task dispatching embodiment of the method provided in an embodiment of the present invention；
Fig. 4 is a kind of structural schematic diagram of distributed task dispatching device embodiment provided in an embodiment of the present invention；
Fig. 5 is a kind of structural schematic diagram of distributed task dispatching device embodiment provided in an embodiment of the present invention；
Fig. 6 is a kind of configuration diagram of distributed task dispatching system embodiment provided in an embodiment of the present invention；
Fig. 7 is a kind of configuration diagram of distributed task dispatching system embodiment provided in an embodiment of the present invention；
Fig. 8 is a kind of configuration diagram of distributed task dispatching system embodiment provided in an embodiment of the present invention；
Fig. 9 is the structural schematic diagram of a kind of electronic equipment embodiment provided in an embodiment of the present invention.
Specific implementation mode
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the embodiment of the present invention, instead of all the embodiments.Based on the embodiment in the embodiment of the present invention, this field
The every other embodiment that those of ordinary skill is obtained without making creative work, belongs to implementation of the present invention
The range of example protection.
In the prior art, common Quartz is as distributed task dispatching frame, in occupation mode that there are two types of Quartz, one
Kind of mode there are Single Point of Faliure, do not support failure to shift the shortcomings of.Another way will appear " fissure " phenomenon, to avoid the occurrence of
Single Point of Faliure and " fissure " phenomenon, improve the stability and availability of system, and the embodiment of the present invention provides a kind of distributed task scheduling
Dispatching method and device, electronic equipment and readable storage medium storing program for executing, by using Zookeeper clusters, multiple tasks scheduling node
Node is executed with multiple tasks to be registered on Zookeeper clusters, is realized the distributed deployment of task execution node, is avoided list
Point failure, realizes the distributed deployment of task scheduling node, while so that control centre is more stable using Zookeeper clusters,
Zookeeper clusters select a host node according to preset rules from multiple tasks scheduling node, and it is existing to avoid the occurrence of " fissure "
As improving the stability and availability of system.Moreover, in the prior art, when task execution failure or node off-line, lacking one
A perfect monitoring alarm mechanism.Further, the embodiment of the present invention can realize that monitoring alarm, the realization of monitoring alarm to appoint
Business, which executes, breaks down and can be found and be handled in time, greatly reduces the loss unsuccessfully brought by task execution.It ties below
Close the technical solution of attached drawing the present invention will be described in detail embodiment.
First, the part term in the embodiment of the present invention is explained below, in order to those skilled in the art
Understand.
1、Zookeeper：It it is one distributed, the distributed application program coordination service of open source code can be point
Cloth application provides the software of Consistency service.There is provided function includes：Configuring maintenance, domain name service, distributed synchronization, group service
Deng.
2, distributed task dispatching：Task, that is, operation is timing, delay or the application program completion periodically started
This subjob executes sheet when being controlled its startup opportunity by control centre and started using the application program on which platform server
The process of subjob is dispatched.
3, fissure：It is the different nodes in the same cluster, causes to deposit in cluster for the state acquisition of cluster is inconsistent
Multiple host nodes the problem of.For one of the most common the reason is that when network failure, each node all thinks that other nodes are offline
State, thus as host node come take over whole system.
4、Netty：It is a high performance network service development frame, it provides asynchronous, event driven network and answers
With program frame and tool, quickly to develop the network server and client-side program of high-performance, high reliability.
Fig. 1 is a kind of flow chart of distributed task dispatching embodiment of the method provided in an embodiment of the present invention, such as Fig. 1 institutes
Show, the method for the present embodiment may include：
S101, host node receive multiple tasks execution node and pull request with the task that predeterminated frequency is sent, and host node is
Zookeeper clusters are executed in multiple tasks scheduling node and multiple tasks after node is registered on Zookeeper clusters,
The task scheduling node selected from multiple tasks scheduling node according to preset rules.
Specifically, multiple tasks scheduling node is the center of task scheduling, is responsible for all tasks of scheduling and judgement is appointed
Business executes when node breaks down and executes failover operation.In the present embodiment, task scheduling node is disposed with trunking mode, example
3 or 3 or more are such as disposed, each task scheduling node is registered on Zookeeper clusters, and become has in sequential temporarily
The node of sequence.It is also to be disposed with trunking mode, such as deployment 3 or 3 or more, multiple tasks execute that multiple tasks, which execute node,
Node is managed in the form of organizing, and the scheduler task of the same group of same host node of receiving, the task between group and group is held
Row node is mutually isolated.Each task execution node, which is also registered on Zookeeper clusters, becomes an interim orderly node,
Relationship equality between each task execution node.
Optionally, preset rules are specifically as follows：After the completion of multiple tasks scheduling node is registered, sequential is numbered minimum
Task scheduling node as host node, in current primary node delay machine, sequential in remaining task scheduling node is numbered minimum
Task scheduling node as new host node, take over all work of host node.Utilize the height of Zookeeper clusters
Availability and the interim orderly characteristic of all task scheduling nodes, the height that entire distributed task dispatching system may be implemented can
With determining host node according to preset rules, avoid the occurrence of " fissure " phenomenon, improve the stability and availability of system.
Wherein, multiple tasks execute node and pull request to host node transmission task with predeterminated frequency.Such as predeterminated frequency
It is 1 second, multiple tasks execute node and pull request to send task to host node every 1 second frequency, executable with acquisition
Task.
S102, host node pull request according to received task and assign tasks to goal task execution node, and target is appointed
It is a task execution node in multiple tasks execution node that business, which executes node,.
Specifically, host node pulls task after receiving the task that multiple tasks execution node is sent and pulling request
Request is sequentially placed into queue, is pulled according to the task in queue when there is task and is requested assignment to a task execution node,
Ensure that a task is assigned to only a task execution node and executes.
S103, host node receive goal task and execute the task execution knot that node is sent after executing distributed task
Task queue in the execution of goal task execution node is transferred to by fruit when perceiving goal task and executing node failure
The task execution node of other survivals.
Specifically, there are two types of situations for task action result：Task execution success or task execution failure.Goal task executes
It for example can be that generation network failure or other systems are asked in goal task execution node execution task process that node, which breaks down,
Topic leads to system failure, and goal task, which executes node, can be in off-line state, and host node can be perceived by Zookeeper clusters
This deferred information, if being more than offline preset time (such as 1 minute), host node will appoint in the execution of goal task execution node
Business queue is transferred to the task execution node of other survivals.
Further, can also include：Host node is determined when task action result is failure according to task configuration information
Task whether can be carried out to retry, host node retries if it is determined that task can be carried out, then according to retrying time in task configuration information
Number carry out tasks retry, host node if it is determined that be not available for task retry or residue can number of retries be 0 when, determine current task
Execute failure.Task is carried out when determining mission failure according to number of retries to retry, it is successfully general to which task execution can be improved
Rate.
In the present embodiment, using Netty as network communication programming framework, communication sequence agreement uses protobuf,
The efficient communication between task execution node and task scheduling node is improved with this and improves throughput of system.
Distributed task dispatching method provided in this embodiment, each task scheduling node are registered to Zookeeper collection
On group, each task execution node is registered on Zookeeper clusters, ensures all task scheduling nodes and task execution
Node is temporarily orderly, and Zookeeper clusters select a host node according to preset rules from multiple tasks scheduling node, avoid
There is " fissure " phenomenon, goal task is executed holding for node by host node when perceiving goal task and executing node failure
Task queue is transferred to the task execution node of other survivals in row, can avoid Single Point of Faliure, and utilize Zookeeper clusters
High availability, to improve the stability and availability of system.
Fig. 2 is a kind of flow chart of distributed task dispatching embodiment of the method provided in an embodiment of the present invention, such as Fig. 2 institutes
Show, on the basis of the method for the present embodiment is shown in Fig. 1, further, can also include：
S104, host node the calling task scheduling system basic service module when determining that current task executes failure are reported
It is alert.
Task scheduling system infrastructure service module is alarmed, and is specifically as follows：Task scheduling system infrastructure service module
The alert receipt people address stored according to first database sends warning message in a manner of mail or short message, alert receipt people
Location can be email address or telephone number.
Distributed task dispatching method provided in this embodiment passes through the calling task when determining that current task executes failure
Scheduling system basic service module is alarmed, and further may make that task execution failure can be found and be located in time
Reason, greatly reduce the loss unsuccessfully brought by task execution, it can be achieved that cluster it is automatically-monitored.
A specific embodiment is used below, Fig. 1 and distributed task dispatching method shown in Fig. 2 is carried out detailed
Explanation.
Fig. 3 is a kind of interaction diagrams of distributed task dispatching embodiment of the method provided in an embodiment of the present invention, such as Fig. 3
Shown, the distributed task dispatching system in the present embodiment includes Zookeeper clusters (abbreviation ZK in figure), a task scheduling
Node group, including 3 task scheduling nodes, a task execution node group, including 3 task execution nodes, 3 task schedulings
Node and 3 task execution nodes are registered on Zookeeper clusters, and Zookeeper clusters are according to preset rules from multiple
A host node is selected in task scheduling node, specifically, sequential is numbered minimum task scheduling node by Zookeeper clusters
As host node, in current primary node delay machine, by the task scheduling node that sequential number in remaining task scheduling node is minimum
As new host node, task scheduling node 1 as shown in Figure 6 is host node.By taking task execution node 1 as an example, the present embodiment
Method may include：
Whether S201, task execution node 1 have executable task with predeterminated frequency poll, and sending task to host node pulls
Request.
S202, host node processing task pull request, pull request according to received task and assign tasks to task and hold
Row node 1.
S203, task execution node 1 execute task, and report task execution to host node when confirmation has task return
As a result.
S204, host node handle task action result.
Specifically, host node determines whether to be appointed when task action result is failure according to task configuration information
Business retries, however, it is determined that the task that can carry out retries, then according in task configuration information can number of retries (be, for example, 3 times) appointed
Business retries, and updates task action result to task action result database, and task action result database includes task result
Table, information table and execute queue, host node if it is determined that be not available for task retry or residue can number of retries be 0 when, determination work as
Preceding task execution failure.
Host node calling task scheduling system basic service module when determining that current task executes failure is alarmed, with
The mode of mail or short message sends warning message.
S205, host node execute node 1 by ZK perception tasks and break down, and sub-process is shifted into failure.
Troubleshooting sub-process specifically includes：
Host node judges whether task execution node 1 has task to be carrying out, and is shifted if it is not, being then not necessarily to failure, if so,
Task queue in the execution of task execution node 1 is transferred to the task execution node of other survivals.
Alarm module by Zookeeper clusters perceive each task scheduling node and each task execution node from
Linear state, if a certain node off-line and Offtime is more than 1 minute, predicate node is really offline.Determining task scheduling section
When the offline number of point is more than first threshold or when task execution node off-line number is more than second threshold, calling task dispatches system
Infrastructure service module is alarmed.
Fig. 4 is a kind of structural schematic diagram of distributed task dispatching device embodiment provided in an embodiment of the present invention, this reality
It applies in example, distributed task dispatching device is that Zookeeper clusters execute node in multiple tasks scheduling node and multiple tasks
After being registered on Zookeeper clusters, a task tune being selected from multiple tasks scheduling node according to preset rules
Node is spent, as shown in figure 4, the distributed task dispatching device of the present embodiment may include：Receiving module 11, task allocating module
12 and processing module 13, wherein
Receiving module 11 pulls request for receiving multiple tasks execution node with the task that predeterminated frequency is sent.Task point
It is used to pull request according to received task with module 12 and assigns tasks to goal task execution node, goal task executes section
Point is a task execution node in multiple tasks execution node.Receiving module 11 is additionally operable to：It receives goal task and executes section
The task action result that point is sent after executing distributed task.Processing module 13 is used to execute section perceiving goal task
Goal task is executed to the task execution node that task queue in the execution of node is transferred to other survivals when point breaks down.
Optionally, preset rules are：
Using the minimum task scheduling node of sequential number as host node；
In current primary node delay machine, using the minimum task scheduling node of sequential number in remaining task scheduling node as
New host node.
The device of the present embodiment can be used for executing the technical solution of embodiment of the method shown in Fig. 1, realization principle and skill
Art effect is similar, and details are not described herein again.
Fig. 5 is a kind of structural schematic diagram of distributed task dispatching device embodiment provided in an embodiment of the present invention, this reality
On the basis of the device device shown in Fig. 4 for applying example, further, can also include：
Task retries module 14, which retries module 14 for when task action result is failure, matching according to task
Confidence breath determines whether that can carry out task retries；It is retried if it is determined that task can be carried out, then according to weighing in task configuration information
Examination number carries out task and retries；If it is determined that be not available for task retry or residue can number of retries be 0 when, determine that current task is held
Row failure.
Optionally, can also include alarm module 15, the alarm module 15 is for when determining that current task executes failure
Calling task scheduling system basic service module is alarmed.
The device of the present embodiment can be used for executing the technical solution of embodiment of the method shown in Fig. 1 or Fig. 2, realize former
Reason is similar with technique effect, and details are not described herein again.
Fig. 6 is a kind of configuration diagram of distributed task dispatching system embodiment provided in an embodiment of the present invention, such as Fig. 6
Shown, the distributed task dispatching system of the present embodiment may include：Zookeeper clusters 21, multiple tasks scheduling node 22
Node 23 is executed with multiple tasks, each task scheduling node 22 is registered on Zookeeper clusters 21, each task execution
Node 23 is registered to 21 on Zookeeper clusters.
Specifically, multiple tasks scheduling node 22 is the center of task scheduling, is responsible for all tasks of scheduling and judgement
Task execution node executes failover operation when breaking down.In the present embodiment, task scheduling node 22 is with trunking mode portion
Administration, such as deployment 3 or 3 or more, each task scheduling node 22 is registered on Zookeeper clusters 21, become when
Interim orderly node in sequence, Zookeeper clusters 21 are registered on Zookeeper clusters it in multiple tasks scheduling node
Afterwards, it selects a task scheduling node as host node from multiple tasks scheduling node according to preset rules, optionally, presets
Rule is specifically as follows：After the completion of multiple tasks scheduling node 22 is registered, by the task scheduling node 22 that sequential number is minimum
As host node, in current primary node delay machine, by the task scheduling section that sequential number in remaining task scheduling node 22 is minimum
Point 22 takes over all work of host node as new host node.High availability using Zookeeper clusters and institute
The interim orderly characteristic of some task scheduling nodes, may be implemented the High Availabitity of entire distributed task dispatching system, according to pre-
If rule determines host node, " fissure " phenomenon is avoided the occurrence of, the stability and availability of system are improved.
Multiple tasks execute node 23 and are used to pull request to host node transmission task with predeterminated frequency.In the present embodiment,
It is also to be disposed with trunking mode, such as deployment 3 or 3 or more, multiple tasks execute node 23 that multiple tasks, which execute node 23,
It is managed in the form of group, the scheduler task of the same group of same host node of receiving, the task execution section between group and group
Point 23 is mutually isolated.Each task execution node 23, which is also registered on Zookeeper clusters, becomes an interim orderly node,
Relationship equality between each task execution node 23, such as predeterminated frequency is 1 second, multiple tasks execute node 23 with every 1 second
Frequency to host node send task pull request, with obtain can perform task.
Host node, which is used to pull request according to received task, assigns tasks to goal task execution node 23, and target is appointed
It is a task execution node in multiple tasks execution node 23 that business, which executes node, and host node is receiving multiple tasks execution
After the task that node 23 is sent pulls request, task is pulled into request and is sequentially placed into queue, when there is task according in queue
Task pull and request assignment to a task execution node 23, ensure that a task is assigned to only a task execution node 23
It executes.Goal task executes node 23 for executing distributed task, and reports task action result to host node.
Host node is additionally operable to receive task action result, is executed goal task when node 23 breaks down in goal task
Execute the task execution node 23 that task queue in the execution of node 23 is transferred to other survivals.Specifically, goal task executes
Node 23 break down for example can be goal task execute node 23 execute task process in occur network failure or other be
System problem leads to system failure, and goal task, which executes node 23, can be in off-line state, and host node can pass through Zookeeper clusters
This deferred information is perceived, if being more than preset time (such as 1 minute) offline, goal task is executed node 23 by host node
Task queue is transferred to the task execution node 23 of other survivals in execution.
In the present embodiment, host node is additionally operable to when it is failure that goal task, which executes the task action result that node reports,
Determine whether that can carry out task retries according to task configuration information, however, it is determined that the task that can carry out retries, then matches confidence according to task
In breath can number of retries retried, however, it is determined that the task of being not available for retry or residue can number of retries be 0 when, it is determined that when
Preceding task execution failure.When determining mission failure according to number of retries carry out task retry, to can be improved task execution at
The probability of work(.
In the present embodiment, using Netty as network communication programming framework, communication sequence agreement uses protobuf,
The efficient communication between task execution node and task scheduling node is improved with this and improves throughput of system.
Distributed task dispatching system provided in this embodiment, including Zookeeper clusters, multiple tasks scheduling node and
Multiple tasks execute node, and each task scheduling node is registered on Zookeeper clusters, and each task execution node is noted
On volume to Zookeeper clusters, ensure that all task scheduling nodes and task execution node are temporarily orderly, Zookeeper collection
Group selects a host node according to preset rules from multiple tasks scheduling node, avoids the occurrence of " fissure " phenomenon, host node exists
Perceive goal task execute node break down when by goal task execute node execution in task queue be transferred to it is other
The task execution node of survival can avoid Single Point of Faliure, and using the high availability of Zookeeper clusters, to improve system
Stability and availability.
Fig. 7 is a kind of configuration diagram of distributed task dispatching system embodiment provided in an embodiment of the present invention, such as Fig. 7
It is shown, on the basis of the distributed task dispatching system system shown in Fig. 6 of the present embodiment, can also include：Back-stage management mould
Block 24, task scheduling system infrastructure service module 26, first database 27 and the second database 28, wherein
Back Administration Module 24 is used to carry out task configuration management according to the operation requests received or shows current appoint
Business implementing result, operation requests are specially the operation requests of administrator, and administrator passes through Back Administration Module configuration task.
First database 27 is used for the configurations information of store tasks scheduling, and the configurations information of task scheduling includes
Task configuration information and multiple tasks execute the configuration information of node.Task configuration information can specifically include task identification
(ID), task type, task status, whether rely on last task action result, task priority, the task belonging to task and hold
Row node group ID, cron expression formula, task triggered time, iterative task repetition interval, iterative task number of retries, upper one
Subtask triggered time, user's spreading parameter etc..Multiple tasks execute node if one group, then multiple tasks execute matching for node
Confidence breath is the configuration information of task execution node group.The configurations information of task scheduling can also include that user task closes
Be table, i.e. the affiliated relationship of task.First database 27 can be relevant database.
Second database 28 is used for the execution queue of store tasks burst information details and task, wherein task fragment is believed
Ceasing details can include：Task ID, the task execution node group ID belonging to task, task maximum number of retries, is appointed at task type
Whether business priority relies on last task action result, execution status of task, the task triggered time, user's spreading parameter, appoints
The fields such as business creation time, task modification time.The execution queue of task includes task queue in executable task queue, execution
With retry queue.Wherein, the second database 28 can be divided into two database difference store tasks burst information details and task
Execution queue, the databases of store tasks burst information details can be K-V storage engines, and key is task ID+task execution
ID, globally unique, value is task details, and the database of the execution queue of store tasks can be deposited for the data of k-list types
Store up engine.
Task scheduling system infrastructure service module 26 is used to provide unified data for Back Administration Module 24 and host node
Read-write service.
In the present embodiment, to realize the function of distributed task dispatching system shown in fig. 6, shows and realize cloth task
A kind of example for the configurations for including needed for scheduling feature.
Further, so that task execution failure can be found and be handled in time, greatly reduce because of task
The loss unsuccessfully brought is executed, Fig. 8 is a kind of framework of distributed task dispatching system embodiment provided in an embodiment of the present invention
Schematic diagram can also wrap as shown in figure 8, on the basis of the distributed task dispatching system system shown in Fig. 2 of the present embodiment
It includes：
Alarm module 25, alarm module 25 are registered on Zookeeper clusters, for monitoring multiple tasks scheduling node 22
The off-line state that node 23 is executed with multiple tasks when determining that 22 offline number of task scheduling node is more than first threshold or is appointed
When business executes 23 offline number of node more than second threshold, calling task scheduling system basic service module 26 is alarmed；
Task scheduling system infrastructure service module is additionally operable to provide reading and writing data service for alarm module；
First database 27 is additionally operable to storage alarm configuration information, and alarm configuration information includes alert receipt people address, the
One threshold value and second threshold.
Specifically, task scheduling system infrastructure service module 26 is alarmed, and is specifically as follows：Task scheduling system basis
The alert receipt people address that service module 26 is stored according to first database 27 sends warning message in a manner of mail or short message,
Alert receipt people address can be email address or telephone number.
Optionally, host node is additionally operable to：When determining that current task executes failure, calling task dispatches system basic service
Module 26 is alarmed, i.e., task scheduling system infrastructure service module 26 sends warning message in a manner of mail or short message.
Optionally, as shown in figure 3, task scheduling system infrastructure service module 26 may include：
Task dispensing unit 261, the configurations information for providing task scheduling read and write interface.
Task alarm unit 262 reads and writees interface and alert process, alarm for provide alarm configuration information
Processing is specially to send warning message in a manner of mail or short message.
Task queue read-write cell 263, the execution queue for providing task read and write interface.
Task action result reporting unit 264 reads and writees interface for provide task action result.
Distributed task dispatching system provided in this embodiment can further be made by the way that alarm module monitoring alarm is arranged
Obtaining task execution failure can be found and be handled in time, and the loss unsuccessfully brought by task execution is greatly reduced, can
So that administrator know the abnormal conditions of system operation in time, it can be achieved that cluster it is automatically-monitored.
Fig. 9 is the structural schematic diagram of a kind of electronic equipment embodiment provided in an embodiment of the present invention, as shown in figure 9, this reality
The electronic equipment for applying example may include：Memory 31 and processor 32,
Memory 31, for storing program instruction, which can be flash (flash memory).
Processor 32, for calling and executing the program instruction in memory, to realize that cloth shown in fig. 1 or fig. 2 is appointed
Each step in business scheduling.The specific associated description that may refer in previous methods embodiment.
Optionally, memory 31 can also be that memory 31 is integrated with processor 32 either independent.
The embodiment of the present invention also provides a kind of readable storage medium storing program for executing, and computer program is stored in readable storage medium storing program for executing, when
When at least one processor of distributed task dispatching device executes the computer program, in the execution of distributed task dispatching device
State the key word matching method in embodiment of the method.
One of ordinary skill in the art will appreciate that：Realize that all or part of step of above-mentioned each method embodiment can lead to
The relevant hardware of program instruction is crossed to complete.Program above-mentioned can be stored in a computer read/write memory medium.The journey
When being executed, execution includes the steps that above-mentioned each method embodiment to sequence；And storage medium above-mentioned includes：ROM, RAM, magnetic disc or
The various media that can store program code such as person's CD.
Finally it should be noted that：The above various embodiments is only to illustrate the technical solution of the embodiment of the present invention, rather than to it
Limitation；Although the embodiment of the present invention is described in detail with reference to foregoing embodiments, those skilled in the art
It should be understood that：Its still can with technical scheme described in the above embodiments is modified, either to which part or
All technical features carries out equivalent replacement；And these modifications or replacements, it does not separate the essence of the corresponding technical solution this hair
The range of bright each embodiment technical solution of embodiment.
Claims (10)
1. a kind of distributed task dispatching method, which is characterized in that including：
Host node receives multiple tasks execution node and pulls request with the task that predeterminated frequency is sent, and the host node is
Zookeeper clusters are registered in multiple tasks scheduling node and the multiple task execution node on Zookeeper clusters
Later, the task scheduling node selected from the multiple task scheduling node according to preset rules；
The host node pulls request according to received task and assigns tasks to goal task execution node, the goal task
It is a task execution node in the multiple task execution node to execute node；
The host node receives the goal task and executes the task action result that node is sent after executing distributed task,
The task queue in perceiving the goal task and executing the execution that the goal task is executed to node when node breaks down
It is transferred to the task execution node of other survivals.
2. according to the method described in claim 1, it is characterized in that, the preset rules are：
Using the minimum task scheduling node of sequential number as host node；
In current primary node delay machine, using sequential number in remaining task scheduling node minimum task scheduling node as newly
Host node.
3. according to the method described in claim 1, it is characterized in that, the method further includes：
The host node determines whether to be appointed when the task action result is failure according to the task configuration information
Business retries；
The host node retries if it is determined that task can be carried out, then according in the task configuration information can number of retries appointed
Business retries；
The host node if it is determined that be not available for task retry or residue can number of retries be 0 when, determine that current task executes mistake
It loses.
4. according to claims 1 to 3 any one of them method, which is characterized in that the method further includes：
Host node calling task scheduling system basic service module when determining that current task executes failure is alarmed.
5. a kind of distributed task dispatching device, which is characterized in that the distributed task dispatching device is Zookeeper clusters
After multiple tasks scheduling node and multiple tasks execution node are registered on Zookeeper clusters, according to preset rules
The task scheduling node selected from the multiple task scheduling node, the distributed task dispatching device include：
Receiving module pulls request for receiving the multiple task execution node with the task that predeterminated frequency is sent；
Task allocating module assigns tasks to goal task execution node for pulling request according to received task, described
It is a task execution node in the multiple task execution node that goal task, which executes node,；
The receiving module is additionally operable to：It receives the goal task and executes the task that node is sent after executing distributed task
Implementing result；
Processing module, for the goal task to be executed node when perceiving the goal task and executing node failure
Execution in task queue be transferred to the task execution nodes of other survivals.
6. device according to claim 5, which is characterized in that the preset rules are：
Using the minimum task scheduling node of sequential number as host node；
In current primary node delay machine, using sequential number in remaining task scheduling node minimum task scheduling node as newly
Host node.
7. device according to claim 5, which is characterized in that described device further includes：
Task retries module, for when the task action result is failure, being determined whether according to the task configuration information
Task can be carried out to retry；
Retried if it is determined that task can be carried out, then according in the task configuration information can number of retries carry out task retry；
If it is determined that be not available for task retry or residue can number of retries be 0 when, determine that current task executes failure.
8. according to claim 5~7 any one of them device, which is characterized in that described device further includes：
Alarm module, for determine current task execute failure when calling task scheduling system basic service module reported
It is alert.
9. a kind of electronic equipment, which is characterized in that including：
Memory, for storing program instruction；
Processor, for calling and executing the program instruction in the memory, to realize described in any one of Claims 1 to 4
Distributed task dispatching method.
10. a kind of readable storage medium storing program for executing, which is characterized in that be stored with computer program in the readable storage medium storing program for executing, work as distribution
When at least one processor of formula task scheduling apparatus executes the computer program, distributed task dispatching device perform claim
It is required that 1~4 any one of them distributed task dispatching method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711486482.6A CN108304255A (en) | 2017-12-29 | 2017-12-29 | Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711486482.6A CN108304255A (en) | 2017-12-29 | 2017-12-29 | Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108304255A true CN108304255A (en) | 2018-07-20 |
Family
ID=62868142
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711486482.6A Pending CN108304255A (en) | 2017-12-29 | 2017-12-29 | Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108304255A (en) |
Cited By (47)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109379448A (en) * | 2018-12-27 | 2019-02-22 | 深圳市网心科技有限公司 | A kind of file distribution dispositions method, device, electronic equipment and storage medium |
| CN109523187A (en) * | 2018-11-27 | 2019-03-26 | 北京字节跳动网络技术有限公司 | Method for scheduling task, device and equipment |
| CN109558231A (en) * | 2018-11-26 | 2019-04-02 | 广州虎牙信息科技有限公司 | Distributed task dispatching method and computer equipment |
| CN109582451A (en) * | 2018-11-21 | 2019-04-05 | 金色熊猫有限公司 | Method for scheduling task, system, equipment and readable medium |
| CN109586951A (en) * | 2018-11-07 | 2019-04-05 | 北京达佳互联信息技术有限公司 | Implement method, apparatus, electronic equipment and the readable storage medium storing program for executing of failure transfer |
| CN109617988A (en) * | 2018-12-28 | 2019-04-12 | 平安科技（深圳）有限公司 | Request retries method and Related product |
| CN109656706A (en) * | 2018-12-25 | 2019-04-19 | 江苏满运软件科技有限公司 | Distributed task dispatching method, system, equipment and medium |
| CN109710263A (en) * | 2018-12-18 | 2019-05-03 | 北京字节跳动网络技术有限公司 | Compilation Method, device, storage medium and the electronic equipment of code |
| CN109783717A (en) * | 2018-12-04 | 2019-05-21 | 北京奇艺世纪科技有限公司 | Query task processing method, system, server cluster and device, computer readable storage medium |
| CN109814998A (en) * | 2019-01-22 | 2019-05-28 | 中国联合网络通信集团有限公司 | A kind of method and device of multi-process task schedule |
| CN109857558A (en) * | 2019-01-18 | 2019-06-07 | 苏宁易购集团股份有限公司 | A kind of data flow processing method and system |
| CN110231995A (en) * | 2019-05-20 | 2019-09-13 | 平安科技（深圳）有限公司 | A kind of method for scheduling task based on Actor model, device and storage medium |
| CN110290180A (en) * | 2019-05-31 | 2019-09-27 | 北京大米科技有限公司 | Distributed task dispatching method, apparatus, computer equipment and storage medium |
| CN110308979A (en) * | 2019-06-27 | 2019-10-08 | 苏宁消费金融有限公司 | The asynchronous process frame and its implementation of task based access control scheduling |
| CN110377410A (en) * | 2019-07-16 | 2019-10-25 | 中信百信银行股份有限公司 | Method for scheduling task, system, electronic equipment and computer readable storage medium |
| CN110569115A (en) * | 2019-08-15 | 2019-12-13 | 贝壳技术有限公司 | Multipoint deployment process management method and process contention method |
| CN110597608A (en) * | 2019-09-12 | 2019-12-20 | 阿里巴巴集团控股有限公司 | Task processing method and device, distributed system and storage medium |
| CN110661844A (en) * | 2019-08-16 | 2020-01-07 | 北京旷视科技有限公司 | Automatic release scheduling system, method and storage medium |
| CN110727505A (en) * | 2019-12-17 | 2020-01-24 | 杭州连银科技有限公司 | Distributed task scheduling and service monitoring system capable of being hot-loaded |
| CN110798339A (en) * | 2019-10-09 | 2020-02-14 | 国电南瑞科技股份有限公司 | Task disaster tolerance method based on distributed task scheduling framework |
| CN110868564A (en) * | 2018-08-28 | 2020-03-06 | 杭州海康威视系统技术有限公司 | Storage task allocation method, cloud storage system and electronic equipment |
| CN110968410A (en) * | 2019-11-29 | 2020-04-07 | 深圳前海环融联易信息科技服务有限公司 | Task failure retry method and device, computer equipment and storage medium |
| CN111147541A (en) * | 2019-11-18 | 2020-05-12 | 广州文远知行科技有限公司 | Node processing method, device and equipment based on parameter server and storage medium |
| CN111324435A (en) * | 2020-02-06 | 2020-06-23 | 北京奇艺世纪科技有限公司 | Distributed task scheduling and registering method, device and distributed task scheduling system |
| CN111352709A (en) * | 2018-12-20 | 2020-06-30 | 顺丰科技有限公司 | Task scheduling method and device in distributed system |
| CN111427694A (en) * | 2020-03-26 | 2020-07-17 | 北京金山云网络技术有限公司 | Task execution method, device, system and server |
| CN111522649A (en) * | 2019-02-01 | 2020-08-11 | 北京京东尚科信息技术有限公司 | Distributed task allocation method, device and system |
| CN111597056A (en) * | 2020-05-18 | 2020-08-28 | 北京思特奇信息技术股份有限公司 | Distributed scheduling method, system, storage medium and device |
| CN111708627A (en) * | 2020-06-22 | 2020-09-25 | 中国平安财产保险股份有限公司 | Task scheduling method and device based on distributed scheduling framework |
| CN111818159A (en) * | 2020-07-08 | 2020-10-23 | 腾讯科技（深圳）有限公司 | Data processing node management method, device, equipment and storage medium |
| CN111913791A (en) * | 2020-07-02 | 2020-11-10 | 北京和瑞精准医学检验实验室有限公司 | Task scheduling method, device, equipment and computer readable storage medium |
| CN111930830A (en) * | 2020-06-22 | 2020-11-13 | 心有灵犀科技股份有限公司 | Distributed transaction data processing method and system based on shared database |
| CN111949475A (en) * | 2020-07-15 | 2020-11-17 | 北京思特奇信息技术股份有限公司 | Method and system for achieving distributed task scheduling based on zookeeper shell |
| WO2020248626A1 (en) * | 2019-06-10 | 2020-12-17 | 北京达佳互联信息技术有限公司 | Task execution method, apparatus, device and system, and storage medium |
| CN112395140A (en) * | 2020-11-17 | 2021-02-23 | 平安科技（深圳）有限公司 | Decentralized task scheduling method, device, equipment and medium |
| CN112416562A (en) * | 2020-12-11 | 2021-02-26 | 深圳市思迪信息技术股份有限公司 | Method and device for distributed task scheduling engine |
| CN112437148A (en) * | 2020-11-20 | 2021-03-02 | 北京奇艺世纪科技有限公司 | Service request processing method and device, service server and system |
| CN112527489A (en) * | 2020-12-22 | 2021-03-19 | 税友软件集团股份有限公司 | Task scheduling method, device and equipment and computer readable storage medium |
| CN112954008A (en) * | 2021-01-26 | 2021-06-11 | 网宿科技股份有限公司 | Distributed task processing method and device, electronic equipment and storage medium |
| CN112965807A (en) * | 2021-03-30 | 2021-06-15 | 建信金融科技有限责任公司 | Task scheduling method, device, equipment and medium |
| CN113051063A (en) * | 2021-06-02 | 2021-06-29 | 北京轻松筹信息技术有限公司 | Task scheduling method and device for distributed tasks and electronic equipment |
| CN113377518A (en) * | 2021-07-05 | 2021-09-10 | 四川新网银行股份有限公司 | Distributed task timing scheduling method and device |
| WO2021179522A1 (en) * | 2020-03-13 | 2021-09-16 | 平安国际智慧城市科技股份有限公司 | Computing resource allocation system, method, and apparatus, and computer device |
| CN113760485A (en) * | 2020-07-16 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Scheduling method, device and equipment of timing task and storage medium |
| CN113867911A (en) * | 2021-09-26 | 2021-12-31 | 聚好看科技股份有限公司 | Task scheduling method and device and micro-service system |
| CN114124959A (en) * | 2021-12-06 | 2022-03-01 | 天地伟业技术有限公司 | Data processing device of cloud streaming media service and cloud streaming media cluster |
| WO2024109003A1 (en) * | 2022-11-22 | 2024-05-30 | 华为云计算技术有限公司 | Task processing method and apparatus, and node |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060198386A1 (en) * | 2005-03-01 | 2006-09-07 | Tong Liu | System and method for distributed information handling system cluster active-active master node |
| CN103763378A (en) * | 2014-01-24 | 2014-04-30 | 中国联合网络通信集团有限公司 | Task processing method and system and nodes based on distributive type calculation system |
| CN104461752A (en) * | 2014-11-21 | 2015-03-25 | 浙江宇视科技有限公司 | Two-level fault-tolerant multimedia distributed task processing method |
| CN105447097A (en) * | 2015-11-10 | 2016-03-30 | 北京北信源软件股份有限公司 | Data acquisition method and system |
| CN106126346A (en) * | 2016-07-05 | 2016-11-16 | 东北大学 | A kind of large-scale distributed data collecting system and method |
| CN106502769A (en) * | 2016-09-30 | 2017-03-15 | 华为技术有限公司 | Distributed transaction processing method, apparatus and system |
-
2017
- 2017-12-29 CN CN201711486482.6A patent/CN108304255A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20060198386A1 (en) * | 2005-03-01 | 2006-09-07 | Tong Liu | System and method for distributed information handling system cluster active-active master node |
| CN103763378A (en) * | 2014-01-24 | 2014-04-30 | 中国联合网络通信集团有限公司 | Task processing method and system and nodes based on distributive type calculation system |
| CN104461752A (en) * | 2014-11-21 | 2015-03-25 | 浙江宇视科技有限公司 | Two-level fault-tolerant multimedia distributed task processing method |
| CN105447097A (en) * | 2015-11-10 | 2016-03-30 | 北京北信源软件股份有限公司 | Data acquisition method and system |
| CN106126346A (en) * | 2016-07-05 | 2016-11-16 | 东北大学 | A kind of large-scale distributed data collecting system and method |
| CN106502769A (en) * | 2016-09-30 | 2017-03-15 | 华为技术有限公司 | Distributed transaction processing method, apparatus and system |
Non-Patent Citations (2)
| Title |
|---|
| 童维勤，黄林鹏: "《数据密集型计算和模型》", 31 January 2015, 上海科学技术出版社 * |
| 赵业清: "《钢铁生产复杂物流网络系统仿真 面向多Agent的系统工程方法》", 31 August 2015, 新华出版社 * |
Cited By (69)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110868564B (en) * | 2018-08-28 | 2021-06-04 | 杭州海康威视系统技术有限公司 | Storage task allocation method, cloud storage system and electronic equipment |
| CN110868564A (en) * | 2018-08-28 | 2020-03-06 | 杭州海康威视系统技术有限公司 | Storage task allocation method, cloud storage system and electronic equipment |
| CN109586951A (en) * | 2018-11-07 | 2019-04-05 | 北京达佳互联信息技术有限公司 | Implement method, apparatus, electronic equipment and the readable storage medium storing program for executing of failure transfer |
| CN109586951B (en) * | 2018-11-07 | 2020-04-17 | 北京达佳互联信息技术有限公司 | Method, device, electronic equipment and readable storage medium for implementing fault transfer |
| CN109582451A (en) * | 2018-11-21 | 2019-04-05 | 金色熊猫有限公司 | Method for scheduling task, system, equipment and readable medium |
| CN109558231A (en) * | 2018-11-26 | 2019-04-02 | 广州虎牙信息科技有限公司 | Distributed task dispatching method and computer equipment |
| CN109523187A (en) * | 2018-11-27 | 2019-03-26 | 北京字节跳动网络技术有限公司 | Method for scheduling task, device and equipment |
| CN109783717B (en) * | 2018-12-04 | 2022-02-01 | 北京奇艺世纪科技有限公司 | Query task processing method, system, server cluster, device and computer readable storage medium |
| CN109783717A (en) * | 2018-12-04 | 2019-05-21 | 北京奇艺世纪科技有限公司 | Query task processing method, system, server cluster and device, computer readable storage medium |
| CN109710263A (en) * | 2018-12-18 | 2019-05-03 | 北京字节跳动网络技术有限公司 | Compilation Method, device, storage medium and the electronic equipment of code |
| CN111352709A (en) * | 2018-12-20 | 2020-06-30 | 顺丰科技有限公司 | Task scheduling method and device in distributed system |
| CN109656706A (en) * | 2018-12-25 | 2019-04-19 | 江苏满运软件科技有限公司 | Distributed task dispatching method, system, equipment and medium |
| CN109379448A (en) * | 2018-12-27 | 2019-02-22 | 深圳市网心科技有限公司 | A kind of file distribution dispositions method, device, electronic equipment and storage medium |
| CN109379448B (en) * | 2018-12-27 | 2022-06-10 | 深圳市网心科技有限公司 | File distributed deployment method and device, electronic equipment and storage medium |
| CN109617988A (en) * | 2018-12-28 | 2019-04-12 | 平安科技（深圳）有限公司 | Request retries method and Related product |
| CN109857558A (en) * | 2019-01-18 | 2019-06-07 | 苏宁易购集团股份有限公司 | A kind of data flow processing method and system |
| CN109814998A (en) * | 2019-01-22 | 2019-05-28 | 中国联合网络通信集团有限公司 | A kind of method and device of multi-process task schedule |
| CN111522649A (en) * | 2019-02-01 | 2020-08-11 | 北京京东尚科信息技术有限公司 | Distributed task allocation method, device and system |
| CN111522649B (en) * | 2019-02-01 | 2022-04-12 | 北京京东尚科信息技术有限公司 | Distributed task allocation method, device and system |
| CN110231995A (en) * | 2019-05-20 | 2019-09-13 | 平安科技（深圳）有限公司 | A kind of method for scheduling task based on Actor model, device and storage medium |
| CN110231995B (en) * | 2019-05-20 | 2023-08-08 | 平安科技（深圳）有限公司 | Task scheduling method, device and storage medium based on Actor model |
| CN110290180A (en) * | 2019-05-31 | 2019-09-27 | 北京大米科技有限公司 | Distributed task dispatching method, apparatus, computer equipment and storage medium |
| CN110290180B (en) * | 2019-05-31 | 2022-03-29 | 北京大米科技有限公司 | Distributed task scheduling method and device, computer equipment and storage medium |
| US11556380B2 (en) | 2019-06-10 | 2023-01-17 | Beijing Dajia Internet Information Technology Co., Ltd. | Task execution method, apparatus, device and system, and storage medium |
| WO2020248626A1 (en) * | 2019-06-10 | 2020-12-17 | 北京达佳互联信息技术有限公司 | Task execution method, apparatus, device and system, and storage medium |
| CN110308979A (en) * | 2019-06-27 | 2019-10-08 | 苏宁消费金融有限公司 | The asynchronous process frame and its implementation of task based access control scheduling |
| CN110377410A (en) * | 2019-07-16 | 2019-10-25 | 中信百信银行股份有限公司 | Method for scheduling task, system, electronic equipment and computer readable storage medium |
| CN110569115B (en) * | 2019-08-15 | 2023-08-04 | 贝壳技术有限公司 | Multi-point deployment process management method and process competing method |
| CN110569115A (en) * | 2019-08-15 | 2019-12-13 | 贝壳技术有限公司 | Multipoint deployment process management method and process contention method |
| CN110661844A (en) * | 2019-08-16 | 2020-01-07 | 北京旷视科技有限公司 | Automatic release scheduling system, method and storage medium |
| CN110597608A (en) * | 2019-09-12 | 2019-12-20 | 阿里巴巴集团控股有限公司 | Task processing method and device, distributed system and storage medium |
| CN110597608B (en) * | 2019-09-12 | 2023-08-22 | 创新先进技术有限公司 | Task processing method and device, distributed system and storage medium |
| CN110798339A (en) * | 2019-10-09 | 2020-02-14 | 国电南瑞科技股份有限公司 | Task disaster tolerance method based on distributed task scheduling framework |
| CN111147541A (en) * | 2019-11-18 | 2020-05-12 | 广州文远知行科技有限公司 | Node processing method, device and equipment based on parameter server and storage medium |
| CN110968410B (en) * | 2019-11-29 | 2023-08-08 | 深圳前海环融联易信息科技服务有限公司 | Task failure retry method and device, computer equipment and storage medium |
| CN110968410A (en) * | 2019-11-29 | 2020-04-07 | 深圳前海环融联易信息科技服务有限公司 | Task failure retry method and device, computer equipment and storage medium |
| CN110727505A (en) * | 2019-12-17 | 2020-01-24 | 杭州连银科技有限公司 | Distributed task scheduling and service monitoring system capable of being hot-loaded |
| CN110727505B (en) * | 2019-12-17 | 2020-04-10 | 杭州连银科技有限公司 | Distributed task scheduling and service monitoring system capable of being hot-loaded |
| CN111324435A (en) * | 2020-02-06 | 2020-06-23 | 北京奇艺世纪科技有限公司 | Distributed task scheduling and registering method, device and distributed task scheduling system |
| WO2021179522A1 (en) * | 2020-03-13 | 2021-09-16 | 平安国际智慧城市科技股份有限公司 | Computing resource allocation system, method, and apparatus, and computer device |
| CN111427694A (en) * | 2020-03-26 | 2020-07-17 | 北京金山云网络技术有限公司 | Task execution method, device, system and server |
| WO2021190087A1 (en) * | 2020-03-26 | 2021-09-30 | 北京金山云网络技术有限公司 | Task execution method, device and system, and server |
| CN111597056B (en) * | 2020-05-18 | 2023-06-30 | 北京思特奇信息技术股份有限公司 | Distributed scheduling method, system, storage medium and equipment |
| CN111597056A (en) * | 2020-05-18 | 2020-08-28 | 北京思特奇信息技术股份有限公司 | Distributed scheduling method, system, storage medium and device |
| CN111930830A (en) * | 2020-06-22 | 2020-11-13 | 心有灵犀科技股份有限公司 | Distributed transaction data processing method and system based on shared database |
| CN111708627A (en) * | 2020-06-22 | 2020-09-25 | 中国平安财产保险股份有限公司 | Task scheduling method and device based on distributed scheduling framework |
| CN111708627B (en) * | 2020-06-22 | 2023-06-20 | 中国平安财产保险股份有限公司 | Task scheduling method and device based on distributed scheduling framework |
| CN111913791B (en) * | 2020-07-02 | 2023-10-13 | 北京和瑞精湛医学检验实验室有限公司 | Task scheduling method, device, equipment and computer readable storage medium |
| CN111913791A (en) * | 2020-07-02 | 2020-11-10 | 北京和瑞精准医学检验实验室有限公司 | Task scheduling method, device, equipment and computer readable storage medium |
| CN111818159B (en) * | 2020-07-08 | 2024-04-05 | 腾讯科技（深圳）有限公司 | Management method, device, equipment and storage medium of data processing node |
| CN111818159A (en) * | 2020-07-08 | 2020-10-23 | 腾讯科技（深圳）有限公司 | Data processing node management method, device, equipment and storage medium |
| CN111949475A (en) * | 2020-07-15 | 2020-11-17 | 北京思特奇信息技术股份有限公司 | Method and system for achieving distributed task scheduling based on zookeeper shell |
| CN113760485A (en) * | 2020-07-16 | 2021-12-07 | 北京沃东天骏信息技术有限公司 | Scheduling method, device and equipment of timing task and storage medium |
| CN112395140B (en) * | 2020-11-17 | 2023-01-17 | 平安科技（深圳）有限公司 | A decentralized task scheduling method apparatus, device and medium |
| CN112395140A (en) * | 2020-11-17 | 2021-02-23 | 平安科技（深圳）有限公司 | Decentralized task scheduling method, device, equipment and medium |
| CN112437148B (en) * | 2020-11-20 | 2023-03-07 | 北京奇艺世纪科技有限公司 | Service request processing method and device, service server and system |
| CN112437148A (en) * | 2020-11-20 | 2021-03-02 | 北京奇艺世纪科技有限公司 | Service request processing method and device, service server and system |
| CN112416562B (en) * | 2020-12-11 | 2024-06-04 | 深圳市思迪信息技术股份有限公司 | Method and device for distributed task scheduling engine |
| CN112416562A (en) * | 2020-12-11 | 2021-02-26 | 深圳市思迪信息技术股份有限公司 | Method and device for distributed task scheduling engine |
| CN112527489A (en) * | 2020-12-22 | 2021-03-19 | 税友软件集团股份有限公司 | Task scheduling method, device and equipment and computer readable storage medium |
| CN112954008B (en) * | 2021-01-26 | 2022-11-04 | 网宿科技股份有限公司 | Distributed task processing method and device, electronic equipment and storage medium |
| CN112954008A (en) * | 2021-01-26 | 2021-06-11 | 网宿科技股份有限公司 | Distributed task processing method and device, electronic equipment and storage medium |
| CN112965807A (en) * | 2021-03-30 | 2021-06-15 | 建信金融科技有限责任公司 | Task scheduling method, device, equipment and medium |
| CN113051063A (en) * | 2021-06-02 | 2021-06-29 | 北京轻松筹信息技术有限公司 | Task scheduling method and device for distributed tasks and electronic equipment |
| CN113051063B (en) * | 2021-06-02 | 2021-09-17 | 北京轻松筹信息技术有限公司 | Task scheduling method and device for distributed tasks and electronic equipment |
| CN113377518A (en) * | 2021-07-05 | 2021-09-10 | 四川新网银行股份有限公司 | Distributed task timing scheduling method and device |
| CN113867911A (en) * | 2021-09-26 | 2021-12-31 | 聚好看科技股份有限公司 | Task scheduling method and device and micro-service system |
| CN114124959A (en) * | 2021-12-06 | 2022-03-01 | 天地伟业技术有限公司 | Data processing device of cloud streaming media service and cloud streaming media cluster |
| WO2024109003A1 (en) * | 2022-11-22 | 2024-05-30 | 华为云计算技术有限公司 | Task processing method and apparatus, and node |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108304255A (en) | Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing | |
| CN105897946A (en) | Obtaining method and system of access address | |
| CN111552556B (en) | GPU cluster service management system and method | |
| US9641685B2 (en) | Systems and methods for an omni-channel routing broker | |
| CN113886089B (en) | Task processing method, device, system, equipment and medium | |
| US11221943B2 (en) | Creating an intelligent testing queue for improved quality assurance testing of microservices | |
| CN113778623A (en) | Resource processing method and device, electronic equipment and storage medium | |
| CN113094053B (en) | Delivery method and device of product and computer storage medium | |
| US20230029198A1 (en) | Scheduling complex jobs in a distributed network | |
| CN107066339A (en) | Distributed job manager and distributed job management method | |
| CN111427670A (en) | Task scheduling method and system | |
| CN110740047B (en) | Network slice management arrangement system | |
| CN110971439A (en) | Policy decision method and device, system, storage medium, policy decision unit and cluster | |
| CN110138577A (en) | Cluster creation method, first server and readable storage medium storing program for executing | |
| CN110521173A (en) | Virtual Networking System, VIM, virtual network controls method and recording medium | |
| CN106293911A (en) | Dispatching System, method | |
| CN113076186B (en) | Task processing method, device, electronic equipment and storage medium | |
| CN109842500A (en) | A kind of dispatching method and system, working node and monitoring node | |
| CN114168297A (en) | Method, device, equipment and medium for scheduling collection tasks | |
| CN111522630B (en) | Method and system for executing planned tasks based on batch dispatching center | |
| CN110532105A (en) | A kind of control method, system and the device of message queue consumer process | |
| CN108243205A (en) | A kind of method, equipment and system for being used to control cloud platform resource allocation | |
| CN111324459A (en) | Calendar-based resource scheduling method and device, electronic equipment and storage medium | |
| CN109510862A (en) | Hough transformation method, apparatus and system | |
| CN112328403B (en) | SparkContext configuration method, sparkContext configuration device and server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180720 |
|
| RJ01 | Rejection of invention patent application after publication |