CN111176812A

CN111176812A - Clustered timing task scheduling system

Info

Publication number: CN111176812A
Application number: CN201911381894.2A
Authority: CN
Inventors: 崔云龙
Original assignee: Unicloud Nanjing Digital Technology Co Ltd
Current assignee: Unicloud Nanjing Digital Technology Co Ltd
Priority date: 2019-12-27
Filing date: 2019-12-27
Publication date: 2020-05-19

Abstract

The invention discloses a clustered timing task scheduling system which comprises a local task container, a Quartz timing server, a zookeeper cluster, a mysql database and a RabbitMQ message notification module, wherein the local task container holds a timing task held by the Quartz timing server, the zookeeper cluster takes the local task container from the mysql database and divides the local task container to nodes of different Quartz timing servers, and the Quartz timing server sends the task to the RabbitMQ message notification module to be timed by service monitoring triggering service. The systematization can simultaneously execute tasks by multiple nodes, improve the performance and avoid resource waste.

Description

Clustered timing task scheduling system

Technical Field

The invention relates to a clustered timing task scheduling system, and belongs to the technical field of computers.

Background

Whether in internet application or enterprise-level application, a large number of batch processing tasks are filled, and some task scheduling systems are often needed to solve problems, so tools capable of realizing timing tasks such as Timer, scheduled executive, spring timing framework and the like appear. Although these timing task frameworks are simple to implement, the scenarios they can support are very limited. With the gradual evolution of the micro-service architecture, the single architecture of the system gradually evolves into a distributed micro-service architecture, and many original task scheduling platforms cannot meet the requirements of a business system. Therefore, some task scheduling platforms based on distribution, such as an open-source timing scheduler quartz, appear, which support rich and diverse scheduling methods, can meet various conventional and special requirements, have flexible application modes, support various combination modes of tasks and scheduling to support various storage modes of scheduling data, and support clustered deployment. The cluster function of quartz brings high availability and expandability to task scheduling through the functions of fault transfer and load balancing, and solves the problem of single-point fault of a single task.

But there is no communication between the nodes in the quartz cluster, the quartz application is aware of the other application through the database tables. In brief, the distributed scheduling policy of quartz is an asynchronous policy of database-bound resources, each scheduler complies with an operation rule based on a database lock so as to ensure the uniqueness of operation, and meanwhile, the asynchronous operation of a plurality of nodes ensures the reliability of service. However, this strategy has its own limitations: the cluster characteristic has a good effect on tasks with high CPU utilization rate, but for a large number of short tasks, each node can preempt a database lock, so that a large number of threads wait for resources, and the situation becomes more and more serious along with the increase of the nodes. In addition, the distribution of quartz only solves the problem of high availability, does not solve the problem of task fragmentation, and still has the limit of single machine processing.

Disclosure of Invention

In order to overcome the defects of the prior art, the invention provides a clustered timing task scheduling system which can simultaneously execute tasks by multiple nodes, improve the performance and avoid the waste of resources. .

In order to achieve the purpose, the clustered timing task scheduling system comprises a local task container, Quartz timing servers, a zookeeper cluster, a mysql database and a RabbitMQ message notification module, wherein timing tasks held by the Quartz timing servers are held in the local task container, the zookeeper cluster is used for taking the local task container from the mysql database and dividing the local task container to nodes of different Quartz timing servers, and the Quartz timing servers are used for sending the tasks to the RabbitMQ message notification module and are timed by service monitoring triggering services.

Further, each Quartz timing server interacts with the zookeeper cluster, a master node in the zookeeper cluster performs task division, and division results are distributed to the Quartz timing server nodes in the zookeeper cluster.

Further, when a Quartz timing server node in the zookeeper cluster is down, the tasks held by the Quartz timing server node which is down are redistributed to the Quartz timing server nodes which are normally operated.

Further, the node of each Quartz timing server holds a unique timing task, and the Quartz timing server triggers the timing task received by the Quartz timing server.

Further, the node creation structure of the zookeeper cluster includes a server node and a task node.

Further, the child node under the server node is a service registration node in the Quartz timing server cluster.

Further, the child nodes under the task node are nodes of all timing tasks.

Furthermore, each task node has a unique child node, and the child node of the task node is a machine node which uniquely holds the task for the Quartz timing server in the Quartz timing server cluster.

The clustered timing task scheduling system provided by the invention removes strong dependence on the database when quartz self-distributed deployment is carried out, and avoids the problem that the original clustered architecture frequently competes for the database lock. And a plurality of quartz nodes execute tasks simultaneously through zookeeper, so that the system scheduling performance is improved, the task fragmentation is realized, the horizontal expansion of the quartz task scheduling is realized, and the performance is greatly improved.

Drawings

The present invention will be further described and illustrated with reference to the following drawings.

FIG. 1 is a block diagram of a clustered timed task scheduling system in accordance with a preferred embodiment of the present invention;

fig. 2 is a node creation structure diagram in a clustered timing task scheduling system according to a preferred embodiment of the present invention.

Detailed Description

The technical solution of the present invention will be more clearly and completely explained by the description of the preferred embodiments of the present invention with reference to the accompanying drawings.

As shown in fig. 1, a clustered timing task scheduling system according to a preferred embodiment of the present invention includes a local task container, Quartz timing servers, a zookeeper cluster, a mysql database, and a RabbitMQ message notification module, where a node of each Quartz timing server holds a unique timing task, and the Quartz timing servers trigger timing tasks received by the Quartz timing servers.

The local task container holds the timing task held by the Quartz timing server, and the task of the local container is pushed to the container of Quartz to complete the triggering of the timing task. And the zookeeper cluster takes the local task container from the mysql database and divides the local task container to nodes of different Quartz timing servers, and the Quartz timing servers send the tasks to a RabbitMQ message notification module to be timed by service monitoring triggering service.

Each Quartz timing server interacts with the zookeeper cluster, a master node in the zookeeper cluster performs task division, and division results are distributed to the Quartz timing server nodes in the zookeeper cluster.

When a Quartz timing server node in the zookeeper cluster is down, the tasks held by the Quartz timing server node which is down are redistributed to the Quartz timing server nodes which normally run, and the coordinated result is delivered to a local task container.

As shown in fig. 2, the node creation structure of the zookeeper cluster includes a server node and a task node.

And the child node under the server node is a service registration node in the Quartz timing server cluster.

And the child nodes under the task node are nodes of all timing tasks. And each task node is provided with a unique child node, and the child node of the task node is a machine node which uniquely holds the task for the Quartz timing server in the Quartz timing server cluster.

The above detailed description merely describes preferred embodiments of the present invention and does not limit the scope of the invention. Without departing from the spirit and scope of the present invention, it should be understood that various changes, substitutions and alterations can be made herein by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims and their equivalents. The scope of the invention is defined by the claims.

Claims

1. A clustered timing task scheduling system is characterized by comprising a local task container, a Quartz timing server, a zookeeper cluster, a mysql database and a RabbitMQ message notification module, wherein the local task container holds a timing task held by the Quartz timing server, the zookeeper cluster takes the local task container from the mysql database and divides the local task container to nodes of different Quartz timing servers, and the Quartz timing server sends a task to the RabbitMQ message notification module to be timed by service monitoring triggering service.

2. The clustered timed task scheduling system of claim 1 wherein each of said Quartz timing servers interacts with a zookeeper cluster, is task partitioned by a master node in said zookeeper cluster, and distributes the results of the partitioning to the Quartz timing server nodes in the zookeeper cluster.

3. The clustered timed task scheduling system of claim 2 wherein when a Quartz timing server node in said zookeeper cluster is down, tasks held by said Quartz timing server node that is down will be redistributed to normally operating Quartz timing server nodes.

4. A clustered timing task scheduling system as claimed in claim 3 wherein each of said Quartz timing server nodes holds a unique timing task, said Quartz timing server triggering its own received timing task.

5. The clustered timed task scheduling system of claim 1 wherein the node creation structure of the zookeeper cluster includes a server node and a task node.

6. The clustered timing task scheduling system of claim 5, wherein a child node under the server node is a service registration node in a Quartz timing server cluster.

7. The clustered timing task scheduling system of claim 5, wherein the child nodes under the task node are nodes of all timing tasks.

8. The clustered timing task scheduling system of claim 7, wherein each of said task nodes has a unique child node under it, said child node of task node being a machine node that is uniquely owned by a Quartz timing server in a Quartz timing server cluster.