CN115480894A - Method, device and equipment for realizing distributed delay task - Google Patents

Method, device and equipment for realizing distributed delay task Download PDF

Info

Publication number
CN115480894A
CN115480894A CN202211109479.3A CN202211109479A CN115480894A CN 115480894 A CN115480894 A CN 115480894A CN 202211109479 A CN202211109479 A CN 202211109479A CN 115480894 A CN115480894 A CN 115480894A
Authority
CN
China
Prior art keywords
task
executor
target
processed
database redis
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211109479.3A
Other languages
Chinese (zh)
Inventor
臧朱龙
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tianyi Safety Technology Co Ltd
Original Assignee
Tianyi Safety Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tianyi Safety Technology Co Ltd filed Critical Tianyi Safety Technology Co Ltd
Priority to CN202211109479.3A priority Critical patent/CN115480894A/en
Publication of CN115480894A publication Critical patent/CN115480894A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/48Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806Task transfer initiation or dispatching
    • G06F9/4843Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2455Query execution
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46Multiprogramming arrangements
    • G06F9/52Program synchronisation; Mutual exclusion, e.g. by means of semaphores
    • G06F9/526Mutual exclusion algorithms

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Computational Linguistics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application provides a method, a device and equipment for realizing a distributed delay task, wherein the method comprises the following steps: acquiring a task to be processed, acquiring a distributed lock by taking a task ID as a Key, and submitting the task to a database Redis; and periodically traversing the task executor queue, when traversing to the current task executor, accessing the database Redis by taking the task type corresponding to the current task executor as a Key, inquiring the most front target task in the task execution expiration time of the tasks to be processed corresponding to the task type in the queue to determine whether the task is expired, otherwise traversing the next current task executor, if so, acquiring a distributed lock by using the ID of the target task, executing the target task through the working thread of the current task executor, traversing the next current task executor, and if not, traversing the next current task executor. The method and the system solve the consistency problem and the idempotent problem with lower complexity and effectively manage different types of tasks.

Description

Method, device and equipment for realizing distributed delay task
Technical Field
The present invention relates to the field of message queue execution technologies, and in particular, to a method, an apparatus, and a device for implementing a distributed delay task.
Background
At present, distributed delay tasks are mainly realized by using a message middleware open source message queue system RabbitMQ and a platform database Redis based on open source rapid development software. The distributed delay task realized by the RabbitMQ depends on the operation and maintenance of the RabbitMQ, so the complexity and the cost are high. The distributed delay task is realized based on the database Redis, the tasks are automatically sequenced according to the execution time of the tasks by utilizing the data structure characteristic of the database Redis Zset, then the Zset set is polled and inquired at regular time, the tasks are taken out and executed if the tasks meeting the conditions are found, and the tasks are deleted in the Zset after the execution is finished.
However, in the prior art, the distributed delay task is realized based on the database Redis by using the data structure characteristics of the database Redis Zset, and the situation that when the production ends of the tasks are deployed in a cluster mode, the same task is generated and written into the database Redis at a plurality of production ends is not considered, and if the task is not controlled, the repeated generation of the task is caused, namely the idempotent problem. Meanwhile, when the consumption ends of the tasks are deployed in a cluster mode, the prior art ensures that one task is consumed by only one consumption end by introducing the intermediate component, and the problem of consistency of repeated execution of the tasks to be consumed by a plurality of consumption ends is solved. However, the introduction of the intermediate component not only increases the complexity and the cost of development and maintenance, but also does not effectively manage and mobilize tasks of different types.
Disclosure of Invention
The application provides a method, a device and equipment for realizing distributed delay tasks, which solve the problem of idempotency of tasks submitted by a plurality of task production ends and the problem of consistency of tasks consumed by a plurality of task consumption ends by utilizing a database Redis distributed lock, and simultaneously traverse task executors corresponding to different types of tasks periodically at the consumption ends to effectively manage and mobilize the tasks of different types.
In a first aspect, the present application provides a method for implementing a distributed latency task, which serves as a production end, and includes:
acquiring a task to be processed, and acquiring a distributed lock from a database Redis by using the ID of the task;
if the distributed lock is not acquired, automatically giving up the task to be processed;
if the distributed lock is obtained, generating a task structure body corresponding to the task to be processed according to the task information of the task to be processed and a predefined task structure body;
and storing the generated task structure body in a Zset data structure, and submitting the task structure body to a task queue of a database Redis, wherein the Zset data structure comprises a task type serving as a Key Key, a task structure serving as a value and a task execution expiration time serving as score.
In one or more embodiments, the task structure includes at least one of the following items of information:
task ID, task type, task execution expiration time, extension data and data required by task execution;
the extension data contains request context information of the task.
In a second aspect, the present application provides a method for implementing a distributed latency task, which serves as a consuming side, and includes:
periodically traversing a task executor queue, and accessing a database Redis by taking a task type corresponding to a current task executor as a Key each time the current task executor is traversed;
querying the score of at least one task to be processed corresponding to the task type in the task queue, and determining a target task with the task execution expiration time most ahead in the at least one task to be processed;
traversing the next current task executor when the target task is determined to be not due, otherwise, acquiring a distributed lock of a database Redis by using the ID of the target task;
if the distributed lock is not acquired, automatically giving up execution of the target task, and traversing the next current task executor;
and if the distributed lock is acquired, acquiring the value of the target task from a database Redis, executing the target task through the working thread of the current task executor, and traversing the next current task executor.
In one or more embodiments, the task executor is generated as follows:
labeling task types corresponding to different task executors in an annotation form;
aiming at task executors of different task types, defining processing logic for processing data in a task structure body;
and structuring the processing logic and the corresponding annotations defined for each task type into task executors corresponding to the task types.
In one or more embodiments, structuring the processing logic and corresponding annotations defined for each task type into task executors for the corresponding task type includes:
structuring the processing logic defined for each task type and the corresponding annotations into a task executor comprising the task type and a reflection calling parameter;
and the reflection calling parameter is used for executing the target task by using the value of the target task acquired from the database Redis according to the processing logic for processing the data in the task structure.
In one or more embodiments, the task executor further comprises:
a task confirmation interface for reading a user instruction;
and reading a user instruction from the task confirmation interface through a working thread of the current task executor, deleting the target task after the target task is executed according to the user instruction, or resubmitting the target task and resetting the task execution expiration time of the target task.
In one or more embodiments, confirming whether the target task is due comprises:
and if the score of the target task is later than the current system clock, the target task is expired, otherwise, the target task is not expired.
In one or more embodiments, obtaining a value of the target task from a database Redis, and executing the target task through a worker thread of a current task executor includes:
acquiring the value of the target task from a database Redis, and restoring the request context of the target task in the working thread of the current task executor by using the extended data in the value of the target task;
and after the request context is restored, executing the required data by using the task in the value of the target task acquired from the database Redis, and executing the target task.
In one or more embodiments, further comprising:
and after the target task is executed and completed through the working thread of the current task executor, the target task is automatically deleted by the database Redis.
In a third aspect, the present application provides a device for implementing a distributed latency task, as a production end, including:
the distributed lock acquisition module is used for acquiring a task to be processed and acquiring a distributed lock from a database Redis by using the ID of the task;
the task abandoning module is used for automatically abandoning and submitting the to-be-processed task if the distributed lock is not acquired;
the task structure body definition module is used for generating a task structure body corresponding to the task to be processed according to the task information of the task to be processed and a predefined task structure body if the distributed lock is obtained;
and the task submitting module is used for storing the generated task structure in a Zset data structure and submitting the task structure to a task queue of a database Redis, wherein the Zset data structure comprises a task type serving as a Key, a task structure serving as a value and a task execution expiration time serving as score.
In a fourth aspect, the present application provides an apparatus for implementing a distributed latency task, which serves as a consuming side, and includes:
the task executor traversing module is used for periodically traversing the task executor queue, and accessing a database Redis by taking the task type corresponding to the current task executor as a Key each time the current task executor is traversed;
the task searching due time module is used for searching the score of at least one task to be processed corresponding to the task type in the task queue and determining a target task with the task execution due time most ahead in the at least one task to be processed;
the distributed lock acquisition module is used for traversing the next current task executor when the target task is determined not to expire, and otherwise, acquiring a distributed lock of a database Redis by using the ID of the target task;
the task abandoning module is used for automatically abandoning and executing the target task and traversing the next current task executor if the distributed lock is not acquired;
and the task execution module is used for acquiring the value of the target task from a database Redis if the distributed lock is acquired, executing the target task through a working thread corresponding to the current task executor, and traversing the next current task executor.
In a fifth aspect, the present application provides an apparatus for implementing distributed latency tasks, comprising at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the above-described method of implementing distributed latency tasks.
In a sixth aspect, an embodiment of the present application provides a computer storage medium, where a computer program is stored, where the computer program is used to enable a computer to execute the foregoing method for implementing a distributed latency task.
The method, the device and the equipment for realizing the distributed delay task have the following beneficial effects:
the distributed lock of the database Redis is utilized to solve the idempotent problem and the consistency problem of a plurality of task consumption end consumption tasks under a plurality of task production end scenes with lower cost and lower calculation complexity, meanwhile, a method for identifying the consumption tasks by user-defined annotations is adopted, the methods corresponding to different types of tasks are structured into task executors, a task executor queue is periodically traversed, the task executors are asynchronously executed, and the tasks of different types are effectively managed and executed.
Additional features and advantages of the present application will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by the practice of the present application. The objectives and other advantages of the application may be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the description below are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a schematic diagram illustrating implementation of a distributed delay task through a RabbitMQ message queue according to an embodiment of the present application;
fig. 2 is a schematic diagram illustrating a method for implementing a distributed delay task applied to a production end according to an embodiment of the present application;
fig. 3 is a schematic diagram illustrating that a plurality of production terminals compete to submit the to-be-processed task according to an embodiment of the present application;
fig. 4 is a schematic diagram illustrating a method for implementing a distributed delay task applied to a consuming side according to an embodiment of the present application;
fig. 5 is a schematic diagram of an apparatus applied to a production end to implement a distributed delay task according to an embodiment of the present application;
fig. 6 is a schematic diagram of an apparatus for implementing a distributed delay task by a consumer according to an embodiment of the present application;
fig. 7 is a diagram of a device for implementing a distributed latency task according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application without making any creative effort belong to the protection scope of the present application.
In development, some requirements regarding latency tasks are often encountered. For example:
30 minutes of unpaid of the generated order, then automatically cancel;
60 seconds after the order is generated, the user is short-circuited.
In the related technology, a distributed delay task is realized by using a message middleware open source message queue system RabbitMQ and a platform database Redis based on open source rapid development software.
It is a common way to use RabbitMQ as delay queue, and actually the RabbitMQ itself does not directly support providing delay queue function, but indirectly implements distributed delay task through two attributes of TTL and DXL of RabbitMQ message queue, as shown in fig. 1.
TTL refers to the survival time of the message, and the RabbitMQ can set the message expiration time, so that when the message in the buffer queue exists in the queue, the message becomes Dead Letter once the TTL expiration time is exceeded. DLX refers to a Dead-Letter switch, i.e. a Dead-Letter queue bound on the Dead-Letter switch, and once a Dead-Letter occurs in the queue, the message is rerouted to another Exchange to be consumed again.
However, the implementation of distributed delay tasks by using the RabbitMQ depends on the operation and maintenance of the RabbitMQ, so that the complexity and the cost are high.
The distributed delay task is realized based on the database Redis by utilizing the natural ordering capability of an ordered set Zset of the database Redis, and each member of the ordered set is associated with a scoring attribute score which is used for ordering the members in the set from lowest to highest. The tasks are taken as members of an ordered set Zset, and the scoring attribute score of each member is the execution expiration time of any of the tasks. And polling and inquiring the Zset at regular time, taking out and executing the task if the task meeting the conditions is found, and deleting the task in the Zset after the execution is finished.
At present, a technology for realizing a distributed delay task by utilizing Zset data structure characteristics based on a database Redis does not consider that when a production end of the task is deployed in a cluster mode, the situation that the same task is generated and written into the database Redis at a plurality of production ends exists, and if the situation is not controlled, the situation of repeated generation of the task is caused, namely the idempotent problem.
Meanwhile, when the consumption ends of the tasks are deployed in a cluster mode, after the same task reaches the execution time and before the execution is not completed, a plurality of consumption ends pull the task from the database Redis, so that the task is repeatedly executed, and the consistency problem exists.
In the prior art, one task is guaranteed to be consumed by only one consumption end by introducing the intermediate component of single-process deployment, but the intermediate component is introduced, so that not only is the complexity improved, but also the development and maintenance costs are increased, and different types of tasks are not effectively managed and mobilized.
In view of the above problems, the method, the device and the equipment for realizing the distributed delay task in the embodiments of the present application utilize a distributed lock of a database Redis to solve the idempotent problem in a scene of a plurality of task production ends and the consistency problem of a plurality of task consumption ends with lower cost and lower computational complexity, and simultaneously adopt a method of custom annotation to identify consumption tasks, structure methods corresponding to different types of tasks into task executors, periodically traverse task executor queues, asynchronously execute the task executors, and effectively manage and execute the different types of tasks.
The embodiment of the present application provides a method for implementing a distributed delay task, which is applied to a production end, and as shown in fig. 2, the method includes:
step S201, acquiring a task to be processed, and acquiring a distributed lock from a database Redis by using the ID of the task;
after the task to be processed is generated, the task to be processed has a unique ID at a plurality of production ends, and the ID may have a certain service attribute, for example, a task for an order, and the ID may be set as an order number. The ID of one task to be processed is consistent under a plurality of production ends. And after the production end obtains the ID of the task to be processed, the unique distributed lock corresponding to the ID of the task to be processed is obtained from a database Redis by taking the ID as a Key.
Step S202, if the distributed lock is not acquired, the submission of the task to be processed is automatically abandoned;
if the production end does not acquire the distributed lock corresponding to the ID of the to-be-processed task, which indicates that the distributed lock corresponding to the ID of the to-be-processed task has been acquired by other production ends in advance, then the to-be-processed task is automatically abandoned and submitted, and a schematic diagram of a plurality of production ends competing for submitting the to-be-processed task is shown in fig. 3.
Step S203, if the distributed lock is obtained, generating a task structure corresponding to the task to be processed according to the task information of the task to be processed and a predefined task structure;
in one or more embodiments, the task structure includes at least one of the following items of information:
task ID, task type, task execution expiration time, extension data and data required by task execution;
the task type indicates which type of task the task to be processed belongs to, one type of task indicates that the method for processing the type of task is consistent, the task execution expiration time indicates when the task to be processed is delayed to be executed, the extended data contains request context information of the task, and the data required by the task execution includes key data required by the task to be specifically executed.
Step S204, storing the generated task structure in a Zset data structure, and submitting the task structure to a task queue of a database Redis, wherein the Zset data structure comprises a task type serving as a Key Key, a task structure serving as a value and a task execution expiration time serving as score.
The database Redis is a database for storing data by Key value pairs Key-value, and the value is fetched according to the Key value, and each Key corresponds to a unique value. The data structure of the ordered set ZSet of the database Redis used in the embodiments of the present application contains the following attributes:
key: a task type;
value: a task structure;
score attribute score: task execution expiration time.
And the production terminal stores a task structure generated by the task to be processed into an ordered set Zset of a database Redis in a Zset data structure to form a task queue.
According to the method and the device, the unique task ID is defined at the production end of the task and is used as the Key of the distributed lock, and the task can be submitted only by the production end acquiring the distributed lock, so that the idempotent is ensured.
The embodiment of the present application provides a method for implementing a distributed delay task, which is applied to a consuming side, and as shown in fig. 4, the method includes:
step S401, a task executor queue is periodically traversed, and when a current task executor is traversed, a task type corresponding to the current task executor is used as a Key to access a database Redis;
in one or more embodiments, the task executor is generated as follows:
labeling task types corresponding to different task executors in an annotation form;
aiming at task executors of different task types, defining processing logic for processing data in a task structure body;
and structuring the processing logic and the corresponding annotations defined for each task type into task executors corresponding to the task types.
The processing logic for processing the data in the task structure, namely the method for consuming the task, is a method for marking the consumption task by setting annotations corresponding to different task types, so that each task executor corresponds to one task type, and the task executors for processing the different task types form a task executor queue.
By adopting the method of marking the consumption tasks by the user-defined annotation, the methods corresponding to different types of tasks are structured into the task executors, and one task processor is defined for each type of task, so that the effective management of different types of tasks is completed.
In one or more embodiments, structuring the processing logic and corresponding annotations defined for each task type into task executors for the corresponding task type includes:
structuring the processing logic defined aiming at each task type and the corresponding annotation into a task executor comprising the task type and a reflection calling parameter;
and the reflection calling parameter is used for executing the target task according to the processing logic for processing the data in the task structure body by using the value of the target task acquired from the database Redis by taking the target task ID as a Key Key.
In one or more embodiments, the task executor further comprises:
a task confirmation interface for reading user instructions;
and reading a user instruction from the task confirmation interface through a working thread of the current task executor, and deleting the target task after the target task is executed according to the user instruction, or resubmitting the target task and resetting the task execution expiration time of the target task.
Whether the task confirmation interface is set in the task executor depends on whether the task confirmation interface needs to be provided for a user when the task executor is set, and the task is deleted or resubmitted according to a user instruction and the task execution expiration time of the target task is reset.
The method and the device complete one-time operation of registering the consumer side during first deployment. After the application program scans that the service where the consumption method corresponding to a certain type of task is started, all the methods marked with the custom annotation are scanned and found, the method information is structured into a task executor which is stored in a task executor queue, and the registration process is completed.
The method for marking task consumption by user-defined annotation is adopted, and then after the method is scanned by an application program, the application program is structured into a task executor according to the related information of the method, and the task executor is stored in a task executor queue. And opening the periodic timing task to traverse the task executor queue and asynchronously execute the task executor. This allows for efficient management and execution of different types of tasks.
Step S402, inquiring the score of at least one task to be processed corresponding to the task type in a task queue, and determining a target task with the task execution due time most ahead in the at least one task to be processed;
and taking the task type corresponding to the current task executor as a Key, and inquiring the task with the minimum score attribute score of the task queue corresponding to the task type from a database Redis, namely taking the task with the most advanced due time and the closest current system clock as a target task.
Step S403, traversing the next current task executor when the target task is determined not to expire, otherwise, acquiring a distributed lock of a database Redis by using the ID of the target task;
in one or more embodiments, confirming whether the target task is due comprises:
and if the score of the target task is later than the current system clock, the target task is expired, otherwise, the target task is not expired.
The target task being due means that the task execution time of the target task is later than the current system clock.
Step S404, if the distributed lock is not obtained, automatically giving up executing the target task and traversing the next current task executor;
and after the target task expires, the consumption end acquires the distributed lock from the database Redis by taking the ID of the target task as a Key, and the acquisition of the distributed lock indicates that the current task consumption end acquires the resource executed by the target task, so that the next task can be executed. And if the distributed lock is not acquired, automatically giving up the execution task, traversing the next current task executor, and entering the next working thread.
And the consumption end of the task can consume and execute the task only by acquiring the unique ID of the task as the Key of the distributed lock, thereby ensuring the consistency.
And S405, if the distributed lock is acquired, acquiring the value of the target task from a database Redis, executing the target task through the working thread of the current task executor, and traversing the next current task executor.
In one or more embodiments, obtaining a value of the target task from a database Redis, and executing the target task through a worker thread of a current task executor includes:
acquiring the value of the target task from a database Redis, and restoring the request context of the target task in the working thread of the current task executor by using the extended data in the value of the target task;
and after the request context is restored, executing the required data by using the task in the value of the target task acquired from the database Redis, and executing the target task.
Asynchronous consumption tasks have the problem of transmission of request context, and user data is inevitably lost in the process that a thread Chi Zhongla acquires a current working thread to execute a task, so that when a target task is executed, request context information submitted to the task at that time is acquired from extension data in corresponding task data in the value of the target task. Then, the context is set to the request context of the current working thread, so that the request context which submits the task at the moment is restored in the working thread, the user data is recovered, and the user can transmit the request context without feeling.
The application includes the request context into the extension data of the task data at the time of task submission. When a task starts to execute at a consumption end, the request context is taken out and set to the request context of a work thread executed by the task, and the request context information is restored.
After the context is restored, a consuming method of the task is called through a Java reflection calling principle, and the data required by the task execution in the value of the target task is utilized to perform real task processing.
In one or more embodiments, further comprising:
and after the target task is executed and completed through the working thread of the current task executor, the target task is automatically deleted by the database Redis.
And after the task processing is finished, if the task is automatically deleted, deleting the target task from the database Redis task queue, and if the task is not automatically deleted, determining whether the interface deletes the task according to the task. The task confirmation interface has two methods, one is to delete the target task, and the other is to add the target task into the task queue again and reset the task execution expiration time of the target task.
The application provides a task confirmation interface (Acker) to solve the problem of how to consume the task again after the task processing fails, and the task data is manually deleted through the interface, and after the task fails to be executed, the task is added into a task queue again and the execution time of the task is reset.
And all the task consumption ends periodically traverse the task executor queues, after confirming that the task with the minimum score in the tasks corresponding to the current task executor expires, one consumption end acquires a unique distributed lock from a database Redis by using the task ID and then executes the task, other consumption ends automatically traverse the next task executor, and all the consumption ends asynchronously execute the task executor to realize effective management and execution of the tasks of different types.
Based on the same inventive concept, an embodiment of the present application further provides a device for implementing a distributed delay task, as a production end, as shown in fig. 5, including:
a distributed lock acquisition module 501, configured to acquire a task to be processed, and acquire a distributed lock from a database Redis by using an ID of the task;
a task abandoning module 502, configured to, if the distributed lock is not acquired, automatically abandon and submit the to-be-processed task;
a task structure defining module 503, configured to generate a task structure corresponding to the to-be-processed task according to the task information of the to-be-processed task and a predefined task structure if the distributed lock is obtained;
and a task submitting module 504, configured to store the generated task structure in a ZSet data structure, and submit the ZSet data structure to a task queue of a database Redis, where the ZSet data structure includes a task type as a Key, a task structure as a value, and a task execution expiration time as score.
Based on the same inventive concept, an embodiment of the present application further provides a device for implementing a distributed latency task, as a consuming side, as shown in fig. 6, including:
the task executor traversing module 601 is used for periodically traversing a task executor queue, and when a current task executor is traversed, a task type corresponding to the current task executor is used as a Key to access a database Redis;
a task due search module 602, configured to query the score of at least one to-be-processed task corresponding to the task type in the task queue, and determine a target task whose task execution due time is the most advanced in the at least one to-be-processed task;
a distributed lock acquisition module 603, configured to traverse a next current task executor when determining that the target task is not due, and otherwise, acquire a distributed lock of the database Redis by using the ID of the target task;
a task abandoning module 604, configured to automatically abandon execution of the target task and traverse a next current task executor if the distributed lock is not acquired;
and the task execution module 605 is configured to, if the distributed lock is acquired, acquire the value of the target task from a database Redis, execute the target task through a worker thread corresponding to the current task executor, and traverse the next current task executor.
Based on the same inventive concept, the present application provides an apparatus 700 for performing distributed latency tasks, as shown in fig. 7, comprising at least one processor 702; and a memory 701 communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the above-described method of implementing distributed latency tasks.
The memory 701 is used to store programs. In particular, the program may include program code comprising computer operating instructions. The memory 701 may be a volatile memory (volatile memory), such as a random-access memory (RAM); a non-volatile memory (non-volatile memory) such as a flash memory (flash memory), a hard disk (HDD) or a solid-state drive (SSD); combinations of any of the above, or any of the above, may also be used.
The processor 702 may be a Central Processing Unit (CPU), a Network Processor (NP), or a combination of a CPU and an NP. But also a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof. Based on the same inventive concept, embodiments of the present application provide a computer storage medium, where a computer program is stored, and the computer program is used to enable a computer to execute the above method for implementing a distributed latency task.
Based on the same inventive concept, embodiments of the present application provide a computer program medium, where the computer program is stored in the computer storage medium, and the computer program is used to enable a computer to execute the above method for implementing a distributed latency task.
The storage medium may be a non-transitory computer readable storage medium, for example, ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.
The embodiments of the present disclosure further provide a computer program product, which, when running on an electronic device, enables the electronic device to execute any one of the methods for implementing a distributed latency task or any one of the methods for implementing a distributed latency task, which may be related to the method of the present disclosure.
In the several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the modules is merely a logical division, and in actual implementation, there may be other divisions, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed coupling or direct coupling or communication connection between each other may be through some interfaces, indirect coupling or communication connection between devices or modules, and may be in an electrical, mechanical or other form.
The modules described as separate parts may or may not be physically separate, and parts displayed as modules may or may not be physical modules, may be located in one place, or may be distributed on a plurality of network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, functional modules in the embodiments of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented in the form of a software functional module and sold or used as a stand-alone product, may be stored in a computer readable storage medium.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, it may be implemented in whole or in part in the form of a computer program product.
The computer program product includes one or more computer instructions. The procedures or functions described in accordance with the embodiments of the application are all or partially generated when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device, such as a server, a data center, etc., that is integrated with one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), among others.
The technical solutions provided by the present application are introduced in detail, and the present application applies specific examples to explain the principles and embodiments of the present application, and the descriptions of the above examples are only used to help understand the method and the core ideas of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It should be appreciated by those skilled in the art that the embodiments of the present application may be provided as a method, system, or computer program product, as may be embodied by computer program instructions. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may employ a computer usable storage medium having computer usable program code embodied in one or more media (implementing each flow and/or block in the flowchart and/or block diagram, and combinations of flows and/or blocks in the flowchart and/or block diagram).
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.

Claims (13)

1. A method for realizing distributed delay tasks is applied to a production end and is characterized by comprising the following steps:
acquiring a task to be processed, and acquiring a distributed lock from a database Redis by using the ID of the task;
if the distributed lock is not acquired, automatically giving up submission of the task to be processed;
if the distributed lock is obtained, generating a task structure body corresponding to the task to be processed according to the task information of the task to be processed and a predefined task structure body;
and storing the generated task structure body in a Zset data structure, and submitting the task structure body to a task queue of a database Redis, wherein the Zset data structure comprises a task type serving as a Key Key, a task structure serving as a value and a task execution expiration time serving as score.
2. The method of claim 1, wherein the task structure includes at least one of the following items of information:
task ID, task type, task execution expiration time, extension data and data required by task execution;
the extension data includes request context information of the task.
3. A method for realizing distributed delay task is applied to a consumption end, and is characterized by comprising the following steps:
periodically traversing a task executor queue, and accessing a database Redis by taking a task type corresponding to a current task executor as a Key each time the current task executor is traversed;
querying the score of at least one task to be processed corresponding to the task type in the task queue, and determining a target task with the task execution expiration time most ahead in the at least one task to be processed;
traversing the next current task executor when the target task is determined not to be due, otherwise, acquiring a distributed lock of a database Redis by using the ID of the target task;
if the distributed lock is not acquired, automatically giving up executing the target task and traversing the next current task executor;
and if the distributed lock is acquired, acquiring the value of the target task from a database Redis, executing the target task through the working thread of the current task executor, and traversing the next current task executor.
4. The method of claim 3, wherein the task executor is generated by:
labeling task types corresponding to different task executors in an annotation form;
aiming at task executors of different task types, defining processing logic for processing data in a task structure body;
the processing logic and corresponding annotations defined for each task type are structured into task executors corresponding to the task type.
5. The method of claim 4, wherein structuring the processing logic and corresponding annotations defined for each task type into task executors for the corresponding task type comprises:
structuring the processing logic defined for each task type and the corresponding annotations into a task executor comprising the task type and a reflection calling parameter;
and the reflection calling parameter is used for executing the target task by utilizing the value of the target task acquired from the database Redis according to the processing logic for processing the data in the task structure.
6. The method of claim 5, wherein the task executor further comprises:
a task confirmation interface for reading user instructions;
and reading a user instruction from the task confirmation interface through a working thread of the current task executor, and deleting the target task after the target task is executed according to the user instruction, or resubmitting the target task and resetting the task execution expiration time of the target task.
7. The method of claim 3, wherein confirming whether the target task is due comprises:
and if the score of the target task is later than the current system clock, the target task is expired, otherwise, the target task is not expired.
8. The method of claim 3, wherein obtaining the value of the target task from a database Redis, the target task being executed by a worker thread of a current task executor, comprises:
acquiring the value of the target task from a database Redis, and restoring the request context of the target task in the working thread of the current task executor by using the extended data in the value of the target task;
and after the request context is restored, executing the required data by using the task in the value of the target task acquired from the database Redis, and executing the target task.
9. The method of claim 3, further comprising:
and after the target task is executed and completed through the working thread of the current task executor, the target task is automatically deleted by the database Redis.
10. An apparatus for implementing a distributed latency task, as a production side, comprising:
the distributed lock acquisition module is used for acquiring a task to be processed and acquiring a distributed lock from a database Redis by using the ID of the task;
the task abandoning module is used for automatically abandoning and submitting the to-be-processed task if the distributed lock is not acquired;
the task structure body definition module is used for generating a task structure body corresponding to the task to be processed according to the task information of the task to be processed and a predefined task structure body if the distributed lock is obtained;
and the task submitting module is used for storing the generated task structure in a Zset data structure and submitting the task structure to a task queue of a database Redis, wherein the Zset data structure comprises a task type serving as a Key, a task structure serving as a value and a task execution expiration time serving as score.
11. An apparatus for implementing distributed latency tasks, as a consuming side, comprising:
the task executor traversing module is used for periodically traversing the task executor queue, and accessing a database Redis by taking the task type corresponding to the current task executor as a Key each time the current task executor is traversed;
the task searching due time module is used for searching the score of at least one task to be processed corresponding to the task type in the task queue and determining a target task with the task execution due time most ahead in the at least one task to be processed;
the distributed lock acquisition module is used for traversing the next current task executor when the target task is determined not to expire, and otherwise, acquiring a distributed lock of a database Redis by using the ID of the target task;
the task abandoning module is used for automatically abandoning the execution of the target task and traversing the next current task executor if the distributed lock is not acquired;
and the task execution module is used for acquiring the value of the target task from a database Redis if the distributed lock is acquired, executing the target task through a working thread corresponding to the current task executor, and traversing the next current task executor.
12. An electronic device implementing distributed latency tasks, comprising at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.
13. A computer program medium, characterized in that the computer storage medium stores a computer program for causing a computer to perform the method of any one of claims 1-9.
CN202211109479.3A 2022-09-13 2022-09-13 Method, device and equipment for realizing distributed delay task Pending CN115480894A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211109479.3A CN115480894A (en) 2022-09-13 2022-09-13 Method, device and equipment for realizing distributed delay task

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211109479.3A CN115480894A (en) 2022-09-13 2022-09-13 Method, device and equipment for realizing distributed delay task

Publications (1)

Publication Number Publication Date
CN115480894A true CN115480894A (en) 2022-12-16

Family

ID=84424127

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211109479.3A Pending CN115480894A (en) 2022-09-13 2022-09-13 Method, device and equipment for realizing distributed delay task

Country Status (1)

Country Link
CN (1) CN115480894A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115878097A (en) * 2023-03-03 2023-03-31 智慧眼科技股份有限公司 Method and system for creating timing task
CN116501533A (en) * 2023-06-26 2023-07-28 神州医疗科技股份有限公司 Error processing method and system for asynchronous queue task

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115878097A (en) * 2023-03-03 2023-03-31 智慧眼科技股份有限公司 Method and system for creating timing task
CN116501533A (en) * 2023-06-26 2023-07-28 神州医疗科技股份有限公司 Error processing method and system for asynchronous queue task
CN116501533B (en) * 2023-06-26 2023-08-25 神州医疗科技股份有限公司 Error processing method and system for asynchronous queue task

Similar Documents

Publication Publication Date Title
CN115480894A (en) Method, device and equipment for realizing distributed delay task
CN109391646B (en) Message middleware message acquisition method, device and system
CN113760509B (en) Timeout management method and device and electronic equipment
WO2022187005A1 (en) Replication of parent record having linked child records that were previously replicated asynchronously across data storage regions
CN110647570B (en) Data processing method and device and electronic equipment
CN111355765B (en) Network request processing and sending method and device
CN112015595B (en) Master-slave database switching method, computing device and storage medium
CN110955460A (en) Service process starting method and device, electronic equipment and storage medium
CN108121514B (en) Meta information updating method and device, computing equipment and computer storage medium
US20200218708A1 (en) Methods, Systems, Databases and Network Nodes of Data Communication Networks for Handling Data Posts
US11604810B2 (en) Asynchronous replication of linked parent and child records across data storage regions
US20220283706A1 (en) Media storage for online meetings in edge network storage
CN106354830B (en) Method and device for data synchronization between database cluster nodes
CN115510036A (en) Data migration method, device, equipment and storage medium
CN113127231B (en) Order processing and order inquiring method, device, system and electronic equipment
CN114691781A (en) Data synchronization method, system, device, equipment and medium
CN111061719B (en) Data collection method, device, equipment and storage medium
CN109739883B (en) Method and device for improving data query performance and electronic equipment
CN113986942A (en) Message queue management method and device based on man-machine conversation
US10503722B2 (en) Log management apparatus and log management method
CN110609841B (en) Method, device, readable medium and electronic equipment for asynchronously processing messages
CN114003842A (en) Form processing method and device, electronic equipment and computer readable storage medium
CN114500443B (en) Message pushing method, device, system, electronic equipment and storage medium
CN116361388A (en) Data processing method, device, equipment and storage medium
CN117931805A (en) Data processing method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination