CN111382177A - Service data task processing method, device and system - Google Patents
Service data task processing method, device and system Download PDFInfo
- Publication number
- CN111382177A CN111382177A CN202010156747.1A CN202010156747A CN111382177A CN 111382177 A CN111382177 A CN 111382177A CN 202010156747 A CN202010156747 A CN 202010156747A CN 111382177 A CN111382177 A CN 111382177A
- Authority
- CN
- China
- Prior art keywords
- task
- pool
- tasks
- queue
- sorting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims description 14
- 238000012163 sequencing technique Methods 0.000 claims abstract description 76
- 238000012545 processing Methods 0.000 claims abstract description 27
- 238000000034 method Methods 0.000 claims abstract description 21
- 230000008569 process Effects 0.000 claims description 13
- 238000003780 insertion Methods 0.000 claims description 4
- 230000037431 insertion Effects 0.000 claims description 4
- 230000007246 mechanism Effects 0.000 abstract description 10
- 238000010586 diagram Methods 0.000 description 10
- 238000012986 modification Methods 0.000 description 6
- 230000004048 modification Effects 0.000 description 6
- 238000004590 computer program Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000012795 verification Methods 0.000 description 5
- 238000012546 transfer Methods 0.000 description 4
- 238000013475 authorization Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000000926 separation method Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007717 exclusion Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Telephonic Communication Services (AREA)
Abstract
The invention discloses a method, a device and a system for processing a service data task, and belongs to the technical field of computer networks. The method comprises the following steps: pushing tasks to a sorting pool through a queue management service, and sorting in the sorting pool; refreshing the tasks in the sequencing pool to a consumption pool at regular time; obtaining tasks from the consumption pool via the queue management service. According to the invention, through the double-buffer queue task queuing mechanism of the sequencing pool and the consumption pool, the queuing of the database is converted into the queuing in the memory, and the sequencing pool is separated from the consumption pool, so that the smoothness and the high efficiency of task acquisition are not influenced by sequencing operation, and the system efficiency is greatly improved.
Description
Technical Field
The present invention relates to the field of computer network technologies, and in particular, to a method, an apparatus, and a system for processing a service data task.
Background
To realize the preemptive task acquisition mechanism, a priority task queue is needed to realize the scene, and the task queuing mechanism is a key. The early task queuing mechanism generally inquires the most prior task from a relational database, then takes the first task and modifies the state of the first task, so as to prevent other people from acquiring the task. The disadvantages of this mechanism are:
1) the database is called to queue greatly every time, so that the efficiency is extremely low;
2) under a concurrent scene, the condition that centralized operators inquire the same priority task easily occurs, only a first person can acquire the task, other persons cannot acquire the task, only the most priority task can be inquired again, and then the state is modified, so that the efficiency is extremely low.
The queue improved in the later stage is to regularly refresh the priority tasks of the relational database into the cache, lock the queue cache when the client acquires the tasks, and then return the most priority tasks to the client. The design reduces the times of accessing the relational database and the sequencing pressure of the database, and simultaneously avoids the situation that people simultaneously seize one task, but the following defects still exist:
1) large queuing of the database is required at regular time, and the pressure of the database is high;
2) during queuing, the client cannot smoothly acquire tasks, and an instant pause phenomenon occurs.
Disclosure of Invention
Based on the defects in the prior art, the inventor discovers that a centralized operation system needs an efficient task pulling mechanism through exploration, but the system is different from a traditional task pushing mode, the data access is more centralized through the pulling mode, and the efficiency requirement is more difficult to meet. Therefore, by providing the improved business data task processing method, device and system, the relational database task is firstly cached in the memory, and the double-buffer queue task queuing mechanism of the sequencing pool and the consumption pool is set, so that the queuing of the database is converted into the queuing in the memory, and the sequencing pool and the consumption pool are separated, so that the smoothness and the high efficiency of task acquisition cannot be influenced by sequencing operation, and the system efficiency is greatly improved.
In one aspect, a method for processing a service data task is provided, where the method includes:
pushing tasks to a sorting pool through a queue management service, and sorting in the sorting pool;
refreshing the tasks in the sequencing pool to a consumption pool at regular time;
obtaining tasks from the consumption pool via the queue management service.
Further, pushing the task to a sorting pool through a queue management service, and sorting in the sorting pool, including:
classifying the tasks to be processed according to a preset task classification rule through a queue management service to form a plurality of task queues, pushing each task queue to a corresponding sequencing pool of each task queue, and sequencing in the corresponding sequencing pool.
Further, classifying the tasks to be processed according to a preset task classification rule through a queue management service to form a plurality of task queues, pushing each task queue to a corresponding sorting pool of each task queue, and sorting in the corresponding sorting pool, including:
classifying the acquired tasks to be processed according to the flow links, the operation center and the transaction categories through a queue management service to form a plurality of task queues, pushing each task queue to a unique corresponding sorting pool of each task queue, and performing insertion sorting operation in the unique corresponding sorting pool.
Further, the classifying operation includes: classifying the tasks into a plurality of task queues corresponding to corresponding operation centers and/or operation personnel according to classification rules of preset flow links, operation centers and transaction categories, and determining task priorities of all tasks in each task queue; and/or the presence of a gas in the gas,
the insert ordering operation comprises: and determining the arrangement order of each task in the task queue according to the task priority, and adjusting the arrangement order in real time according to a task priority adjustment strategy, wherein the task priority adjustment strategy at least comprises service emergency degree and service processing condition.
Further, refreshing the tasks in the sequencing pool to a consumption pool in a timed mode comprises the following steps:
and transferring the most prior task of the sequencing pool to the consumption pool at regular time according to a preset task refreshing rule.
Further, obtaining tasks from the consumption pool by the queue management service includes:
and acquiring the task from the top end of the consumption pool through the queue management service according to a first-in first-out rule.
In another aspect, a service data task processing apparatus is provided, including:
a task ordering module to: pushing tasks to a sorting pool through a queue management service, and sorting in the sorting pool;
the task refreshing module is used for refreshing the tasks in the sequencing pool to a consumption pool at regular time;
and the task acquisition module is used for acquiring tasks from the consumption pool through the queue management service.
Further, the task ordering module is configured to:
classifying the tasks to be processed according to a preset task classification rule through a queue management service to form a plurality of task queues, pushing each task queue to a corresponding sequencing pool of each task queue, and sequencing in the corresponding sequencing pool.
Further, the task ordering module is configured to: classifying the acquired tasks to be processed according to the flow links, the operation center and the transaction categories through a queue management service to form a plurality of task queues, pushing each task queue to a unique corresponding sorting pool of each task queue, and performing insertion sorting operation in the unique corresponding sorting pool.
Further, the classifying operation includes: classifying the tasks into a plurality of task queues corresponding to corresponding operation centers and/or operation personnel according to classification rules of preset flow links, operation centers and transaction categories, and determining task priorities of all tasks in each task queue; and/or the presence of a gas in the gas,
the insert ordering operation comprises: and determining the arrangement order of each task in the task queue according to the task priority, and adjusting the arrangement order in real time according to a task priority adjustment strategy, wherein the task priority adjustment strategy at least comprises service emergency degree and service processing condition.
Further, the task refresh module is configured to:
and transferring the most prior task of the sequencing pool to the consumption pool at regular time according to a preset task refreshing rule.
Further, the task obtaining module obtains the task from the top of the consumption pool through the queue management service according to a first-in first-out rule.
In another aspect, a service data task processing system is provided, including a process engine, a queue management server and a cache, where the queue management server includes a task priority calculator, and the cache includes a sorting pool, a consumption pool, a task priority adjustment polling processor, a task refresher and a task recoverer, where the queue management server obtains tasks to be processed from the process engine, performs a classification operation on the obtained tasks to be processed, forms a plurality of task queues, and determines task priorities of all tasks in each task queue through the task priority calculator; the queue management server pushes the tasks of each task queue to a unique corresponding sorting pool, the tasks are sorted in the sorting pool, the sorting order of each task in the task queue is determined according to the task priority, and the task priority adjustment polling processor adjusts the sorting order in real time according to a task priority adjustment strategy; and the task refresher regularly refreshes the tasks in the sequencing pool to the consumption pool, and the task recoverer recovers the tasks which are not grabbed for a long time in the consumption pool overtime.
The technical scheme provided by the embodiment of the invention has the following beneficial effects:
1. the relational database tasks are firstly cached in the memory, and are arranged through a double-buffer queue task queuing mechanism of the sequencing pool and the consumption pool, when the tasks are listed, the tasks firstly enter the sequencing pool for sequencing, then the tasks meeting the transfer condition in the sequencing pool are transferred or carried to the consumption pool, and the tasks entering the consumption pool are not sequenced any more, because the queue of the database is converted into the queue in the memory and the sequencing pool is separated from the consumption pool, the smoothness and the high efficiency of the task acquisition cannot be influenced by the sequencing operation, the blockage during the task queuing is avoided, and the system efficiency is greatly improved;
2. dividing tasks into different task queues according to a flow link, an operation center and a transaction category, wherein each task queue is provided with a pair of sequencing pools, a consumption pool and a refresher; the sequencing pool adopts a Sorted set data structure and is responsible for sequencing the tasks according to two dimensions of priority and transaction initiation time; the consumption pool adopts a List data structure and is used for realizing smooth acquisition tasks of the client. When the tasks are listed, the tasks enter the sequencing pool firstly, the tasks are inserted into the proper sequencing position, the refresher can carry the most prior tasks in the sequencing pool to the consumption pool at regular time, and the tasks entering the consumption pool are not sequenced any more. When the client side obtains the tasks, the tasks are directly obtained from the top end of the consumption pool, and due to the separation of the sequencing pool and the consumption pool, the smoothness and the high efficiency of the task obtaining cannot be influenced by sequencing operation;
3. converting the queue of the database into queue in a memory (such as a redis memory database), separating a queue library from an acquisition library, avoiding pause during the queue, and showing a preliminary test result that: 2 ten thousand Tps can be achieved under the condition that tasks are not stacked, and more than 8 thousand Tps can be achieved under the condition that the stacking of the sorting pool is 1 million; the response time is less than 20 milliseconds.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
Fig. 1 is a flowchart of a service data task processing method provided in embodiment 1 of the present invention;
fig. 2 is a schematic structural diagram of a service data task processing device provided in embodiment 2 of the present invention;
fig. 3 is a schematic view of a business operation of a business data task processing system provided in embodiment 3 of the present invention in an exemplary business scenario.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention. In the description of the present invention, "a plurality" means two or more unless specifically defined otherwise.
According to the business data task processing method, device and system provided by the embodiment of the invention, the relational database tasks are firstly cached in the memory, and are set through the double-buffer queue task queuing mechanism of the sequencing pool and the consumption pool, when the tasks are listed, the tasks firstly enter the sequencing pool for sequencing, and then the tasks meeting the transfer condition in the sequencing pool are transferred or carried to the consumption pool, and the tasks entering the consumption pool are not sequenced any more.
The following describes in detail a method, an apparatus, and a system for processing a service data task according to an embodiment of the present invention with reference to specific embodiments and drawings.
Example 1
Fig. 1 is a flowchart of a service data task processing method provided in embodiment 1 of the present invention, and as shown in fig. 1, the service data task processing method provided in the embodiment of the present invention includes the following steps:
101. and pushing the tasks to the sorting pool through the queue management service, and sorting in the sorting pool.
For the task queue, in many service scenarios, the system is required to be able to handle 500 ten thousand or even higher traffic volumes per day, each service includes multiple (for example, more than 10) links on average, and tasks are to be ordered and accessed continuously, the performance requirement is high, and if the database is used directly, the database performance becomes a bottleneck. Therefore, the tasks are acquired by adopting a buffer queue mode, and the functions of sequencing and accessing the database are transferred to the buffer, so that the access throughput capacity is improved, and the pressure of the database is relieved.
If the cache is sorted and accessed at the same time, short time blocking can occur, the efficiency of an operator is influenced, for this reason, double-buffer queues are adopted to solve the problem, and task acquisition and task sorting actions are divided into two subdivided task pools, namely a sorting pool and a consumption pool, namely a consumption pool (a queue head task pool) and a sorting pool (a queue body task pool), so that the priority sorting rule of service requirements is ensured, and the operation efficiency of the system is also considered. The consumption pool (queue head task pool) is used for meeting the requirement of quickly acquiring tasks by operators, is a most-priority task buffer pool, and only retains the tasks with the highest priority which can be acquired by the operators. The sequencing pool (queue body task pool) stores the residual full amount of tasks, the task priority sequencing is carried out in the queue body large pool, and the most prior task is extracted in time and refreshed into the queue head.
Specifically, the process engine pushes the tasks to the sequencing pool through the queue management service, and the system sequences the tasks in the sequencing pool at the same time. Preferably, the tasks to be processed are classified by the queue management service according to a preset task classification rule to form a plurality of task queues, and then each task queue is pushed to a corresponding sorting pool of each task queue and sorted in the corresponding sorting pool. Preferably, the obtained tasks to be processed are classified by the queue management service according to the process links, the operation center and the transaction categories to form a plurality of task queues, each task queue is pushed to the unique corresponding sorting pool of each task queue, and the inserting sorting operation is carried out in the unique corresponding sorting pool. Further preferably, the classifying operation includes: classifying the tasks into a plurality of task queues corresponding to corresponding operation centers and/or operation personnel according to classification rules of preset flow links, operation centers and transaction categories, and determining task priorities of all tasks in each task queue; and/or, the insert ordering operation comprises: and determining the arrangement order of each task in the task queue according to the task priority, and adjusting the arrangement order in real time according to a task priority adjustment strategy, wherein the task priority adjustment strategy at least comprises a service emergency degree (such as a transaction emergency degree) and a service processing condition (such as long-time unprocessed service).
In addition, preferably, the sorting pool adopts a Sorted set data structure or a List data structure, and the consumption pool adopts a List data structure; and/or performing task adjustment operation by the task priority adjustment polling processor according to the task priority adjustment strategy.
Illustratively, as in a certain project, the flow links include a recording link, an authorization link, a telephone verification link, a seal verification link and the like, the operation centers include a southern operation center and a northern operation center, and the transaction category includes account type and settlement type transactions; tasks can be divided into different task queues according to flow links, operation centers and transaction categories, and the task queues comprise an input link, a southern operation center and an account queue, an input link, a northern operation center and an account queue, an authorization link, a southern operation center and a settlement category, a printing verification link, a northern operation center and a settlement category and the like. Specifically, for example, in a process of entering a task queue, a process engine pushes an entered task to a sorting pool through a queue management service, the entered task is inserted into a position corresponding to a score according to a priority score, and when the scores are the same, sorting is performed according to time. The queues are divided according to link types and operation centers, different operation centers and different operators are guaranteed to access different queues, efficiency is improved, and queue overlength and resource waste are prevented. Most links are special queues, but for links of one record and two records, because the requirements of the skills of personnel are the same, the business processing logic is the same, and the links can be combined into one queue.
Different service priorities are set according to different services, so that priority processing of the services such as standing and the like can be guaranteed, the requirements of the services such as client urgent service and standing and the like are met, the client urgent service is handled at the first time, and the services such as standing and the like quickly acquire results; distinguishing the handling time (different charging and different fund accounting time) of the client real-time service and the common service, and ensuring that the real-time service is accounted within the promised time; the system provides a basis for task management and scheduling, queues and schedules tasks according to the service priority, and solves the problem of task queue priority sequencing management.
It is noted that, the process of step 101 may be implemented in other ways besides the way described in the above steps without departing from the inventive concept, and the specific way is not limited by the embodiments of the present invention.
102. Refreshing the tasks in the sequencing pool to a consumption pool at regular time;
specifically, according to a preset task refreshing rule, the most prior task of the sequencing pool is transferred to the consumption pool at regular time. Preferably, the tasks are refreshed from the sorting pool to the consumption pool by the task refresher, and the size threshold of the consumption pool is set, such as 5000, and the tasks are really refreshed only when the number of the consumption pools is less than the refresh threshold (such as 1%), and are not refreshed otherwise.
It is noted that, the process of step 102 may be implemented in other ways besides the way described in the above steps without departing from the inventive concept, and the specific way is not limited by the embodiments of the present invention.
103. Tasks are obtained from the consumption pool by a queue management service.
Specifically, the tasks are obtained from the top of the consumption pool by the queue management service according to a first-in first-out rule. Preferably, the operator pulls tasks from the consumption pool on a first-in-first-out basis through the queue management service.
Illustratively, the entry personnel corresponding to the center of the area pull the first-in first-out entry task in the pool to perform entry operation. In the information acquisition link, when the whole order/slice adopts a secondary data acquisition mode, two task items are not distributed to the same input person; when the two-time input data are inconsistent in verification and need to be input again, the task item is not allocated to any one of the first-time input personnel and the second-time input personnel of the business element; when the mode of combining OCR recognition and manual entry is adopted, task items are not distributed to primary entry personnel of the business elements when entry needs to be carried out again due to the fact that the OCR recognition data and the manual entry data are inconsistent in verification. In implementation, the task mutual exclusion of the queue is to implement task filtering by a regular expression.
It is to be noted that, the process of step 103 may be implemented in other ways besides the way described in the above steps without departing from the inventive concept, and the specific way is not limited by the embodiments of the present invention.
Example 2
Fig. 2 is a schematic structural diagram of a service data task processing device provided in embodiment 2 of the present invention, and as shown in fig. 2, the service data task processing device provided in the embodiment of the present invention includes a task sequencing module 21, a task refreshing module 22, and a task obtaining module 23.
Wherein, the task ordering module 21 is configured to: and pushing the tasks to the sorting pool through the queue management service, and sorting in the sorting pool. Preferably, the task ordering module 21 is configured to: the method comprises the steps of classifying tasks to be processed according to a preset task classification rule through a queue management service to form a plurality of task queues, pushing each task queue to a corresponding sorting pool of each task queue, and sorting in the corresponding sorting pool. Further preferably, the task ordering module 21 is configured to: classifying the acquired tasks to be processed according to the flow links, the operation center and the transaction categories through the queue management service to form a plurality of task queues, pushing each task queue to a unique corresponding sorting pool of each task queue, and performing insertion sorting operation in the unique corresponding sorting pool. Further preferably, the classifying operation includes: classifying the tasks into a plurality of task queues corresponding to corresponding operation centers and/or operation personnel according to classification rules of preset flow links, operation centers and transaction categories, and determining task priorities of all tasks in each task queue; and/or, the insert ordering operation comprises: and determining the arrangement order of each task in the task queue according to the task priority, and adjusting the arrangement order in real time according to a task priority adjustment strategy, wherein the task priority adjustment strategy at least comprises a service emergency degree (such as a transaction emergency degree) and a service processing condition (such as long-time unprocessed service).
And the task refreshing module 22 is used for refreshing the tasks in the sequencing pool to the consumption pool at regular time. Preferably, the task refreshing module 22 periodically transfers the top-priority task of the sorting pool to the consumption pool according to a preset task refreshing rule. Further preferably, the task refreshing module 22 refreshes the tasks from the sorting pool to the consumption pool at regular time by the task refresher, and sets a size threshold of the consumption pool, and the tasks are really refreshed only when the number of the consumption pools is less than the refresh threshold, otherwise the tasks are not refreshed.
And the task obtaining module 23 is used for obtaining the tasks from the consumption pool through the queue management service. Preferably, the task obtaining module 23 obtains the task from the top of the consumption pool through the queue management service according to a first-in first-out rule. Further preferably, the operator pulls the task from the consumption pool on a first-in-first-out basis through the queue management service.
Example 3
Fig. 3 is a schematic view of a service operation of a service data task processing system provided in embodiment 3 of the present invention in a certain exemplary service scenario, and as shown in fig. 3, the service data task processing system provided in the embodiment of the present invention includes a process engine, a queue management server, and a cache, where the queue management server includes a task priority calculator (not shown in the figure), and the cache includes a sorting pool, a consumption pool, a task priority adjustment polling processor, a task refresher, and a task recoverer, where the queue management server acquires a task to be processed from the process engine, performs a classification operation on the acquired task to be processed, forms a plurality of task queues, and determines task priorities of all tasks in each task queue through the task priority calculator; the queue management server pushes the tasks of each task queue to a unique corresponding sorting pool, the tasks are sorted in the sorting pool, the sorting order of each task in the task queue is determined according to the task priority, and the task priority adjustment polling processor adjusts the sorting order in real time according to a task priority adjustment strategy; and the task refresher regularly refreshes the tasks in the sequencing pool to the consumption pool, and the task recoverer recovers the tasks which are not grabbed for a long time in the consumption pool overtime.
Compared with the prior art, the business data task processing method, the device and the system provided by the embodiment of the invention have the following beneficial effects:
1. the relational database tasks are firstly cached in the memory, and are arranged through a double-buffer queue task queuing mechanism of the sequencing pool and the consumption pool, when the tasks are listed, the tasks firstly enter the sequencing pool for sequencing, then the tasks meeting the transfer condition in the sequencing pool are transferred or carried to the consumption pool, and the tasks entering the consumption pool are not sequenced any more, because the queue of the database is converted into the queue in the memory and the sequencing pool is separated from the consumption pool, the smoothness and the high efficiency of the task acquisition cannot be influenced by the sequencing operation, the blockage during the task queuing is avoided, and the system efficiency is greatly improved;
2. dividing tasks into different task queues according to a flow link, an operation center and a transaction category, wherein each task queue is provided with a pair of sequencing pools, a consumption pool and a refresher; the sequencing pool adopts a Sorted set data structure and is responsible for sequencing the tasks according to two dimensions of priority and transaction initiation time; the consumption pool adopts a List data structure and is used for realizing smooth acquisition tasks of the client. When the tasks are listed, the tasks enter the sequencing pool firstly, the tasks are inserted into the proper sequencing position, the refresher can carry the most prior tasks in the sequencing pool to the consumption pool at regular time, and the tasks entering the consumption pool are not sequenced any more. When the client side obtains the tasks, the tasks are directly obtained from the top end of the consumption pool, and due to the separation of the sequencing pool and the consumption pool, the smoothness and the high efficiency of the task obtaining cannot be influenced by sequencing operation;
3. converting the queue of the database into queue in a memory (such as a redis memory database), separating a queue library from an acquisition library, avoiding pause during the queue, and showing a preliminary test result that: 2 ten thousand Tps can be achieved under the condition that tasks are not stacked, and more than 8 thousand Tps can be achieved under the condition that the stacking of the sorting pool is 1 million; the response time is less than 20 milliseconds.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present application have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all changes and modifications that fall within the true scope of the embodiments of the present application.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Those not described in detail in this specification are within the skill of the art.
Claims (10)
1. A business data task processing method is characterized by comprising the following steps:
pushing tasks to a sorting pool through a queue management service, and sorting in the sorting pool;
refreshing the tasks in the sequencing pool to a consumption pool at regular time;
obtaining tasks from the consumption pool via the queue management service.
2. The business data task processing method of claim 1, wherein pushing the tasks to a sorting pool by a queue management service and sorting in the sorting pool comprises:
classifying the tasks to be processed according to a preset task classification rule through a queue management service to form a plurality of task queues, pushing each task queue to a corresponding sequencing pool of each task queue, and sequencing in the corresponding sequencing pool.
3. The business data task processing method of claim 2, wherein the task to be processed is classified by the queue management service according to a preset task classification rule to form a plurality of task queues, and then each task queue is pushed to a corresponding sorting pool of each task queue and sorted in the corresponding sorting pool, comprising:
classifying the acquired tasks to be processed according to the flow links, the operation center and the transaction categories through a queue management service to form a plurality of task queues, pushing each task queue to a unique corresponding sorting pool of each task queue, and performing insertion sorting operation in the unique corresponding sorting pool.
4. The business data task processing method of claim 3,
the classification operation includes: classifying the tasks into a plurality of task queues corresponding to corresponding operation centers and/or operation personnel according to classification rules of preset flow links, operation centers and transaction categories, and determining task priorities of all tasks in each task queue; and/or the presence of a gas in the gas,
the insert ordering operation comprises: and determining the arrangement order of each task in the task queue according to the task priority, and adjusting the arrangement order in real time according to a task priority adjustment strategy, wherein the task priority adjustment strategy at least comprises service emergency degree and service processing condition.
5. The business data task processing method according to any one of claims 1 to 4, wherein the task in the sequencing pool is refreshed to a consumption pool at regular time, and the refreshing comprises:
and transferring the most prior task of the sequencing pool to the consumption pool at regular time according to a preset task refreshing rule.
6. The business data task processing method of any one of claims 1 to 4, wherein obtaining tasks from the consumption pool by the queue management service comprises:
and acquiring the task from the top end of the consumption pool through the queue management service according to a first-in first-out rule.
7. A business data task processing apparatus, comprising:
a task ordering module to: pushing tasks to a sorting pool through a queue management service, and sorting in the sorting pool;
the task refreshing module is used for refreshing the tasks in the sequencing pool to a consumption pool at regular time;
and the task acquisition module is used for acquiring tasks from the consumption pool through the queue management service.
8. The business data task processing device of claim 7, wherein the task ordering module is configured to:
classifying the tasks to be processed according to a preset task classification rule through a queue management service to form a plurality of task queues, pushing each task queue to a corresponding sequencing pool of each task queue, and sequencing in the corresponding sequencing pool.
9. The business data task processing device of claim 7, wherein the task refreshing module is configured to:
and transferring the most prior task of the sequencing pool to the consumption pool at regular time according to a preset task refreshing rule.
10. A business data task processing system is characterized by comprising a process engine, a queue management server and a cache, wherein the queue management server comprises a task priority calculator, and the cache comprises a sequencing pool, a consumption pool, a task priority adjustment polling processor, a task refresher and a task recoverer; the queue management server pushes the tasks of each task queue to a unique corresponding sorting pool, the tasks are sorted in the sorting pool, the sorting order of each task in the task queue is determined according to the task priority, and the task priority adjustment polling processor adjusts the sorting order in real time according to a task priority adjustment strategy; and the task refresher regularly refreshes the tasks in the sequencing pool to the consumption pool, and the task recoverer recovers the tasks which are not grabbed for a long time in the consumption pool overtime.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010156747.1A CN111382177A (en) | 2020-03-09 | 2020-03-09 | Service data task processing method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010156747.1A CN111382177A (en) | 2020-03-09 | 2020-03-09 | Service data task processing method, device and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111382177A true CN111382177A (en) | 2020-07-07 |
Family
ID=71218706
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010156747.1A Pending CN111382177A (en) | 2020-03-09 | 2020-03-09 | Service data task processing method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111382177A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000476A (en) * | 2020-08-21 | 2020-11-27 | 南京星邺汇捷网络科技有限公司 | Queue-based task execution flow control method and system |
| CN112468551A (en) * | 2020-11-16 | 2021-03-09 | 浪潮云信息技术股份公司 | Intelligent scheduling working method based on service priority |
| CN113326150A (en) * | 2021-05-31 | 2021-08-31 | 中国工商银行股份有限公司 | Online small-batch message processing method and device |
| CN113342871A (en) * | 2021-05-17 | 2021-09-03 | 浪潮软件股份有限公司 | Queue design method applied to government affair service entity hall queuing |
| CN113342484A (en) * | 2021-05-14 | 2021-09-03 | 深圳奥哲网络科技有限公司 | Process engine method, system, device and storage medium |
| CN113657863A (en) * | 2021-08-23 | 2021-11-16 | 交通银行股份有限公司 | Centralized operation processing system and method thereof |
| CN114418562A (en) * | 2022-01-18 | 2022-04-29 | 北京融威众邦电子技术有限公司 | Real-time account checking method based on multi-party payment |
| CN114489867A (en) * | 2022-04-19 | 2022-05-13 | 浙江大华技术股份有限公司 | Algorithm module scheduling method, algorithm module scheduling device and readable storage medium |
| CN114662866A (en) * | 2022-03-01 | 2022-06-24 | 达而观信息科技（上海）有限公司 | Queuing method for robot process automation tasks |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020087757A1 (en) * | 2001-01-04 | 2002-07-04 | International Business Machines Corporation | Priority queue with arbitrary queuing criteria |
| US20030233392A1 (en) * | 2002-06-12 | 2003-12-18 | Microsoft Corporation | Method and system for managing the execution of threads and the processing of data |
| US20050047425A1 (en) * | 2003-09-03 | 2005-03-03 | Yonghe Liu | Hierarchical scheduling for communications systems |
| WO2017028724A1 (en) * | 2015-08-19 | 2017-02-23 | 阿里巴巴集团控股有限公司 | Service request adjustment method and device |
| US20180300174A1 (en) * | 2017-04-17 | 2018-10-18 | Microsoft Technology Licensing, Llc | Efficient queue management for cluster scheduling |
| CN109710387A (en) * | 2018-12-06 | 2019-05-03 | 成都佰纳瑞信息技术有限公司 | A kind of policy engine and its application method for block chain affairs priority ranking |
| CN110096353A (en) * | 2019-05-14 | 2019-08-06 | 厦门美图之家科技有限公司 | Method for scheduling task and device |
-
2020
- 2020-03-09 CN CN202010156747.1A patent/CN111382177A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020087757A1 (en) * | 2001-01-04 | 2002-07-04 | International Business Machines Corporation | Priority queue with arbitrary queuing criteria |
| US20030233392A1 (en) * | 2002-06-12 | 2003-12-18 | Microsoft Corporation | Method and system for managing the execution of threads and the processing of data |
| US20050047425A1 (en) * | 2003-09-03 | 2005-03-03 | Yonghe Liu | Hierarchical scheduling for communications systems |
| WO2017028724A1 (en) * | 2015-08-19 | 2017-02-23 | 阿里巴巴集团控股有限公司 | Service request adjustment method and device |
| US20180300174A1 (en) * | 2017-04-17 | 2018-10-18 | Microsoft Technology Licensing, Llc | Efficient queue management for cluster scheduling |
| CN109710387A (en) * | 2018-12-06 | 2019-05-03 | 成都佰纳瑞信息技术有限公司 | A kind of policy engine and its application method for block chain affairs priority ranking |
| CN110096353A (en) * | 2019-05-14 | 2019-08-06 | 厦门美图之家科技有限公司 | Method for scheduling task and device |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112000476A (en) * | 2020-08-21 | 2020-11-27 | 南京星邺汇捷网络科技有限公司 | Queue-based task execution flow control method and system |
| CN112468551A (en) * | 2020-11-16 | 2021-03-09 | 浪潮云信息技术股份公司 | Intelligent scheduling working method based on service priority |
| CN113342484A (en) * | 2021-05-14 | 2021-09-03 | 深圳奥哲网络科技有限公司 | Process engine method, system, device and storage medium |
| CN113342871A (en) * | 2021-05-17 | 2021-09-03 | 浪潮软件股份有限公司 | Queue design method applied to government affair service entity hall queuing |
| CN113326150A (en) * | 2021-05-31 | 2021-08-31 | 中国工商银行股份有限公司 | Online small-batch message processing method and device |
| CN113326150B (en) * | 2021-05-31 | 2024-11-05 | 中国工商银行股份有限公司 | Online small batch message processing method and device |
| CN113657863A (en) * | 2021-08-23 | 2021-11-16 | 交通银行股份有限公司 | Centralized operation processing system and method thereof |
| CN114418562A (en) * | 2022-01-18 | 2022-04-29 | 北京融威众邦电子技术有限公司 | Real-time account checking method based on multi-party payment |
| CN114662866A (en) * | 2022-03-01 | 2022-06-24 | 达而观信息科技（上海）有限公司 | Queuing method for robot process automation tasks |
| CN114489867A (en) * | 2022-04-19 | 2022-05-13 | 浙江大华技术股份有限公司 | Algorithm module scheduling method, algorithm module scheduling device and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111382177A (en) | Service data task processing method, device and system | |
| CN104239956B (en) | On schedule preferential physical examination subscription services management method and system | |
| US20070206770A1 (en) | Method, apparatus, and computer product for computing skill value | |
| CN112200484B (en) | Intelligent case division system and method based on policy definition | |
| CN111079557A (en) | Face recognition-based automatic distribution method and system for electric power business hall service terminals | |
| CN102929961A (en) | Data processing method and device thereof based on building quick data staging channel | |
| CN112288341B (en) | Credit factory order scheduling method and device based on multi-agent reinforcement learning | |
| CN107273200A (en) | A kind of method for scheduling task stored for isomery | |
| CN110991846B (en) | Service personnel task allocation method, device, equipment and storage medium | |
| CN105955807A (en) | System and method for processing task | |
| CN109783225A (en) | A kind of tenant's priority management method and system of multi-tenant big data platform | |
| CN112468551A (en) | Intelligent scheduling working method based on service priority | |
| CN115237568A (en) | Mixed weight task scheduling method and system for edge heterogeneous equipment | |
| CN211352458U (en) | Ship VDES communication system for channel scheduling | |
| CN115481297A (en) | Data management method of data lake | |
| US8341012B2 (en) | Working skill estimating program | |
| CN117193992B (en) | Model training method, task scheduling device and computer storage medium | |
| CN113326150A (en) | Online small-batch message processing method and device | |
| CN108073658A (en) | A kind of data synchronous system and method | |
| CN111580991A (en) | Computer data processing method and system | |
| CN111428973A (en) | Virtual resource allocation method capable of effectively improving resource utilization rate | |
| CN109670027A (en) | A kind of image query, caching, storing method and system | |
| CN111210150A (en) | Intelligent park system based on edge calculation | |
| CN101587461A (en) | Memory access dispatching device, dispatching method and memory access control system | |
| Dimitriou | A batch arrival priority queue with recurrent repeated demands, admission control and hybrid failure recovery discipline |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200707 |
|
| RJ01 | Rejection of invention patent application after publication |