CN111158930A - Redis-based high-concurrency time-delay task system and processing method - Google Patents

Abstract

The Redis-based high-concurrency time-delay task system comprises a daemon component, a Redis component, an event process, a message queue component and a work component, wherein the Redis component is provided with a plurality of keys, the tasks are hash-routed into the corresponding keys through uuid, the event process polls the keys, the event process writes the tasks into the message queue component, and the work component cancels the messages from the message queue component and executes the services. The system can realize persistence by relying on persistence of Redis, support high concurrency and high availability by relying on a Redis cluster, and is low in development cost and high in real-time performance. The invention also provides a processing method of the high-concurrency time-delay task system based on Redis.

Description

Redis-based high-concurrency time-delay task system and processing method

Technical Field

The invention relates to a high-concurrency time-delay task system based on Redis, and belongs to the technical field of the Internet.

Background

With the rapid development of the internet, service scenes in a network system become more and more complex, and delay task scenes also become very common. Such as the time-lapse task scenario common in the following lives: pushing a message to the user 2 hours before the activity is finished; pushing a message to the user 2 hours before the coupon is expired; unpaid orders are automatically cancelled 24 hours after order placement, and the like.

For such a kind of delayed task scenario, there are currently a number of solutions. The most common scheme is to sweep the table. The table scanning is to start a background process, scan the whole data table of the database at intervals, judge whether each task meets the triggering condition, and execute the corresponding service if the condition is met. Because scanning the full table places a great deal of stress on the database, scanning the slave library is typically chosen. The biggest advantage of table scanning is that the table scanning is simple to realize, and the data is stored in the DB, so that the data of the task is not needed to be lost, and the failed task can be re-entered in the next scanning. In addition, some message queues may support delayed messages, such as Kafka, which are messages that are sent before a specified time before being sent to the message originator. Another scheme for realizing delay queue is to use ring queue to make time slice, to maintain a linked list in each grid of ring queue, to point a current pointer to a certain grid of ring queue at each moment, to point the current pointer to the next grid of ring queue every time the timer times out, then to process the task in the linked list stored in the grid.

Several of the above latency task solutions have some problems: the biggest problem of table scanning is that delay exists, triggering cannot be carried out at a specified time, and the requirement cannot be met for a scene with high timeliness; the bottleneck of realizing the delay task by the message queue is that the delay task cannot be realized if time needs to be updated again, because the message is sent out and cannot be recovered; the time slice polling scheme is costly to develop without off-the-shelf wheels.

Disclosure of Invention

In order to overcome the defects of the prior art, the invention provides a high-concurrency time-delay task system based on Redis, which can realize persistence by depending on the persistence of the Redis, support high concurrency and high availability by depending on a Redis cluster, and has the advantages of low development cost and high real-time property.

In order to achieve the purpose, the Redis-based high-concurrency delay task system comprises a daemon component, a Redis component, an event process, a message queue component and a work component, wherein the Redis component is provided with a plurality of keys, the tasks are hash-routed into the corresponding keys through uuid, the event process polls the keys, the event process writes the tasks into the message queue component, and the work component cancels the messages from the message queue component and executes the services.

Further, the key stores a source and a value, the source is the execution time of the task, and the value is the data of the task to be executed.

Further, the event processes correspond to keys one to one.

Further, the system also comprises a daemon component, wherein the daemon component monitors a task and a Redis component in real time, and when Redis writing fails, asynchronous compensation is carried out.

Furthermore, the event process is deployed in multiple machines, a zookeeper owner is arranged, and a leader host of the zookeeper owner can fetch the message from the Redis.

Further, after the zookeeper selects the master leader host machine to be down, the zookeeper selects the master leader host machine to reselect a new leader host machine.

Further, the number of keys is expandable.

The invention also provides a processing method of the high-concurrency time-delay task based on Redis, which comprises the following steps:

s1: taking the execution time of the task as score, taking the task data to be executed as value, and storing the value in zset;

s2: the element with the minimum score of zset is inquired by a process in a timing mode, and the operation is realized by using a ZRANGEBYSCORE command;

s3: if the minimum score is less than or equal to the current timestamp, the task is taken out for execution, otherwise, the task is dormant for a period of time and then queried.

The high-concurrency time-delay task system based on Redis can realize persistence by depending on the persistence of the Redis, support high concurrency and high availability by depending on a Redis cluster, and is low in development cost and high in instantaneity.

Drawings

The present invention will be further described and illustrated with reference to the following drawings.

Fig. 1 is a block diagram of a high-concurrency latency task system based on Redis according to a preferred embodiment of the present invention.

Detailed Description

The technical solution of the present invention will be more clearly and completely explained by the description of the preferred embodiments of the present invention with reference to the accompanying drawings.

As shown in fig. 1, a system for high-concurrency and delayed tasks based on Redis according to a preferred embodiment of the present invention includes a daemon component, a Redis component, an event process, a message queue component, and a work component. The Redis component is provided with a plurality of keys, and the number of the keys is expandable. The key stores a source and a value, the source is the execution time of the task, and the value is the data of the task to be executed. And the task is subjected to hash routing to the corresponding key through the uuid, and the event processes correspond to the keys one by one. And polling the key by the event process, writing the task into the message queue component by the event process, and canceling the message from the message queue component and executing the service by the work component.

The system also comprises a daemon component, wherein the daemon component monitors tasks and a Redis component in real time, and when Redis writing fails, asynchronous compensation is carried out.

The event process is deployed in a multi-machine mode, a zookeeper owner is arranged, and a leader host of the zookeeper owner can fetch the message from the Redis. After the zookeeper selects the master leader host machine to be down, the zookeeper selects the master to reselect a new leader host machine.

The invention also provides a processing method of the high-concurrency time-delay task based on Redis, which comprises the following steps:

s1: taking the execution time of the task as score, taking the task data to be executed as value, and storing the value in zset;

s2: the element with the minimum score of zset is inquired by a process in a timing mode, and the operation is realized by using a ZRANGEBYSCORE command;

s3: if the minimum score is less than or equal to the current timestamp, the task is taken out for execution, otherwise, the task is dormant for a period of time and then queried.

Specifically, the method comprises the following steps: the delay task is realized based on Redis through a data structure ZSET of the delay task. The ZSET will store one score and one value, which can be ordered according to score, while the SET is unordered.

The delay task is realized by the following steps:

s1: taking the execution time of the task as score, taking the task data to be executed as value, and storing the value in zset;

s2: the element with the lowest score of zset is inquired by a process in a timing mode, and the operation is realized by using a ZRANGEBYSCORE key-inf + inf limit 01 with scores command;

s3: if the minimum score is less than or equal to the current timestamp, the task is taken out for execution, otherwise, the task is dormant for a period of time and then queried.

The ZSET of Redis is realized by a skip list, the complexity is O (logN), and N is the number of elements stored in the ZSET. Implementation with Redis may rely on persistence of Redis itself to implement persistence, the clustering of Redis to support high concurrency and high availability. Therefore, the development cost is low, and real-time operation can be realized.

Wherein:

1. since the time complexity of the ZSET is o (logn), if there are multiple tasks at the same time point, one key cannot be distributed, which causes congestion. The inquiry speed is prevented from being slowed down after the data amount stored by one key is increased. Therefore, the method is designed to be stored in a plurality of keys, hash routing is carried out on the keys through uuid, and if the task amount is increased, the number of Redis keys can be rapidly expanded to resist the increased number.

2. The event process is to establish a process or thread number which is the same as a plurality of keys, each process has a number, and each process corresponds to one key and continuously polls the corresponding key.

3. The event process only queries out the task, but does not process the traffic, and writes the task into the message queue. In addition, work takes messages from the message queue and then executes services. In this way, work can be deployed in a distributed mode, and events only need to be distributed, so that concurrency can be very high, and even if a large number of tasks exist at the same time, the tasks can be completed within a very short delay.

4. In order to avoid event process stand-alone deployment and incapability of taking messages after machine downtime, data stored in Redis can be backlogged. We deploy event processes from multiple machines, and use zookeeper to opt for the host, only the process on the leader host gets the message from Redis. After the host of the leader crashes, the zookeeper automatically selects a new leader.

5. In actual business, DB write data is also relied upon. The delayed task generation is to modify the DB first and then write data into Redis, so that a scene of successful update of the DB exists, and then failure write of the Redis exists, at this time, the probability of failure write of the Redis is reduced by retry, and if the retry is not successful, a message is sent to the daemon process for asynchronous compensation.

In addition, the time of the delayed task may be altered if the execution time has not been reached. To fulfill this requirement, a ZSET is additionally maintained for each subscriber, in which all the delayed tasks of the current subscriber are stored. For ease of description, this ZSET will be referred to as ZSET-USER. If the user needs to modify his delay task, if there is no way to find this task from the ZSET of the overall delay task, but if it can, it can only traverse this ZSET, obviously this method is too slow and resource consuming. The method adopted by the invention is to take out the delay task of the USER from the ZST-USER, modify score, and finally re-ZADD to the delay tasks ZST and ZST-USER, wherein ZADD can cover the original task, and score is updated. Thereby achieving the effect of modifying the time of the delayed task.

The high-concurrency time-delay task system based on Redis can realize persistence by depending on persistence of the Redis, support high concurrency and high availability by depending on a Redis cluster, and is low in development cost and high in instantaneity. And the architecture also supports the independent modification of the delay time of the delay task, and provides a better scheme selection for the delay task scene.

The above detailed description merely describes preferred embodiments of the present invention and does not limit the scope of the invention. Without departing from the spirit and scope of the present invention, it should be understood that various changes, substitutions and alterations can be made herein by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims and their equivalents. The scope of the invention is defined by the claims.

Claims (8)

1. A high-concurrency delay task system based on Redis is characterized by comprising a daemon component, a Redis component, an event process, a message queue component and a work component, wherein the Redis component is provided with a plurality of keys, the tasks are hash-routed into the corresponding keys through uuid, the event process polls the keys, the event process writes the tasks into the message queue component, and the work component cancels messages from the message queue component and executes services.

2. The Redis-based high-concurrency delay task system as claimed in claim 1, wherein the key stores a source and a value, the source is an execution time of the task, and the value is task data to be executed.

3. The Redis-based high-concurrency delay task system as recited in claim 1, wherein the event processes correspond to keys one to one.

4. The Redis-based high-concurrency and latency task system according to claim 1, further comprising a daemon component that monitors tasks and Redis components in real time, and when Redis writing fails, performs asynchronous compensation.

5. The Redis-based high-concurrency delayed task system according to claim 1, wherein the event process is deployed in multiple machines, and is provided with a zookeeper owner, and a leader host of the zookeeper owner can fetch messages from the Redis.

6. The Redis-based high-concurrency delayed task system, as claimed in claim 5, wherein after said zookeeper elects master leader host downtime, said zookeeper elects master reselects new leader host.

7. A Redis-based high-concurrency, latency task system, as recited in claim 1, wherein the number of keys is expandable.

8. A processing method of high-concurrency time-delay tasks based on Redis is characterized by comprising the following steps:

s1: taking the execution time of the task as score, taking the task data to be executed as value, and storing the value in zset;

s2: the element with the minimum score of zset is inquired by a process in a timing mode, and the operation is realized by using a ZRANGEBYSCORE command;

s3: if the minimum score is less than or equal to the current timestamp, the task is taken out for execution, otherwise, the task is dormant for a period of time and then queried.

Images