CN114090205A

CN114090205A - Task processing method, device, equipment and storage medium

Info

Publication number: CN114090205A
Application number: CN202111371465.4A
Authority: CN
Inventors: 唐成山; 卞嘉骏; 陈军
Original assignee: China Construction Bank Corp
Current assignee: China Construction Bank Corp
Priority date: 2021-11-18
Filing date: 2021-11-18
Publication date: 2022-02-25

Abstract

The invention provides a task processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: determining a target main task and a target task type corresponding to a target task fragment, wherein the target task fragment comprises a plurality of subtasks of the target main task; acquiring a first subtask concurrency number of a task fragment corresponding to the target main task; acquiring a second subtask concurrency number of the task fragment corresponding to the target task type; and comparing the first subtask concurrency number and the second subtask concurrency number with corresponding concurrency thresholds respectively, and processing the subtasks in the target task fragment according to a comparison result. The task processing method can realize flow control of multiple dimensions, meets the processing requirement of subtasks of tens of millions of orders every day, avoids overhigh pressure bearing of the system, and improves the stability and reliability of the system.

Description

Task processing method, device, equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a method, an apparatus, a device, and a storage medium for processing a task.

Background

In recent years, with the development of banking, the scale of the backend system is also enlarged, and the number and complexity of batch processing tasks (also called main tasks) are increasing. When the batch processing tasks are scheduled to be processed on a single host, the single host cannot provide enough performance due to overhigh task quantity and complexity, so that the load of the host is overlarge, and the processing of the tasks is influenced.

With the rise of the distributed architecture, all banks actively explore solutions for host downward movement, and build a core bank system by relying on an enterprise-level distributed platform and adopting an open technology, so that the batch processing function of the original host is realized through a batch processing framework of the distributed architecture.

Since a batch processing task in banking (such as batch collecting and paying, splitting and merging customer information, entering account and the like) often has as many as 10-20 ten thousand subtasks to be processed, it is necessary to design a subtask high-concurrency processing mechanism to quickly execute the batch processing task. In the prior art, a batch processing framework provides a sub-task fragmentation processing mechanism, which can fragment a batch processing task and provide functions of fragment distribution, fragment claiming, fragment execution and the like, so that the purposes of reducing complexity and shortening task execution time are achieved, and the requirement of efficiently executing the batch processing task is met. However, the prior art lacks a flow control mechanism for task execution, which can result in that a downstream transaction system cannot bear in the case of a large number of tasks.

Disclosure of Invention

In view of the foregoing problems in the prior art, an object of the present invention is to provide a task processing method, device, apparatus, and storage medium, which can implement flow control in multiple dimensions and improve stability and reliability of a system.

In order to solve the above problem, the present invention provides a task processing method, including:

determining a target main task and a target task type corresponding to a target task fragment, wherein the target task fragment comprises a plurality of subtasks of the target main task;

acquiring a first subtask concurrency number of the task fragment corresponding to the target main task;

acquiring a second subtask concurrency number of the task fragment corresponding to the target task type;

and comparing the first subtask concurrency number and the second subtask concurrency number with corresponding concurrency thresholds respectively, and processing the subtasks in the target task fragment according to a comparison result.

Further, the acquiring the first subtask concurrence number of the task fragment corresponding to the target main task includes:

acquiring a first task fragment set corresponding to the target main task;

and determining the sum of the concurrent subtask numbers of all task fragments in the first task fragment set to obtain the first concurrent subtask number.

Further, the acquiring the second subtask concurrence number of the task fragment corresponding to the target task type includes:

acquiring a second task fragment set corresponding to the target task type;

and determining the sum of the concurrent numbers of the subtasks of all the task slices in the second task slice set to obtain the concurrent number of the second subtask.

Further, the comparing the first subtask concurrency number and the second subtask concurrency number with the corresponding concurrency threshold respectively, and processing the subtasks in the target task fragment according to the comparison result includes:

determining a first concurrency threshold for the target primary task and a second concurrency threshold for the target task type;

comparing the first subtask concurrency number with the first concurrency threshold to obtain a first comparison result;

comparing the second concurrency number of the subtasks with the second concurrency threshold value to obtain a second comparison result;

determining whether subtasks in the target task fragment can be executed according to the first comparison result and the second comparison result;

and when determining that the subtask in the target task fragment can be executed, executing at least one subtask in the target task fragment.

Further, the determining a first concurrency threshold for the target primary task and a second concurrency threshold for the target task type includes:

acquiring a target configuration file corresponding to the target task fragment;

and determining the first concurrency threshold and the second concurrency threshold according to configuration items of the target configuration file.

Further, after determining the first concurrency threshold for the target primary task and the second concurrency threshold for the target task type, the method further includes:

acquiring current time information;

determining target control parameters according to the current time information;

correcting the first concurrency threshold and/or the second concurrency threshold according to the target control parameter to obtain a corrected first concurrency threshold and/or a corrected second concurrency threshold;

wherein the comparing the first concurrent subtask number with the first concurrent threshold to obtain a first comparison result includes:

comparing the first subtask concurrency number with a corrected first concurrency threshold value to obtain a first comparison result;

and/or the presence of a gas in the atmosphere,

the comparing the second concurrency number of the subtasks with the second concurrency threshold to obtain a second comparison result includes:

and comparing the second subtask concurrency number with a modified second concurrency threshold value to obtain a second comparison result.

Further, the determining whether to execute the subtasks in the target task slice according to the first comparison result and the second comparison result includes:

when the first comparison result and the second comparison result are both smaller than the first comparison result, determining that the subtasks in the target task fragment can be executed;

otherwise, determining that the subtasks in the target task fragment cannot be executed.

Another aspect of the present invention provides a task processing apparatus, including:

the system comprises a determining module, a task dividing module and a task classifying module, wherein the determining module is used for determining a target main task and a target task type corresponding to a target task dividing piece, and the target task dividing piece comprises a plurality of subtasks of the target main task;

the first acquisition module is used for acquiring a first subtask concurrency number of the task fragment corresponding to the target main task;

the second acquisition module is used for acquiring a second subtask concurrency number of the task fragment corresponding to the target task type;

and the task processing module is used for comparing the first subtask concurrency number and the second subtask concurrency number with corresponding concurrency thresholds respectively and processing the subtasks in the target task fragment according to a comparison result.

Another aspect of the present invention provides an electronic device, including a processor and a memory, where at least one instruction or at least one program is stored in the memory, and the at least one instruction or the at least one program is loaded by the processor and executed to implement the task processing method as described above.

Another aspect of the present invention provides a computer-readable storage medium, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the task processing method as described above.

Due to the technical scheme, the invention has the following beneficial effects:

according to the task processing method provided by the embodiment of the invention, the target main task and the target task type corresponding to the target task fragment are determined, the first subtask concurrency number of the task fragment corresponding to the target main task and the second subtask concurrency number of the task fragment corresponding to the target task type are respectively compared with the corresponding concurrency threshold values, and the subtasks in the target task fragment are processed according to the comparison result, so that the flow control of a single task type and a single main task dimension can be realized, the overhigh pressure bearing of the system is avoided while the processing requirement of the subtasks in the tens of millions of orders every day is met, and the stability and the reliability of the system are improved.

Drawings

In order to more clearly illustrate the technical solution of the present invention, the drawings used in the description of the embodiment or the prior art will be briefly described below. It is obvious that the drawings in the following description are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.

FIG. 1 is a schematic illustration of an implementation environment provided by an embodiment of the invention;

FIG. 2 is a flow diagram of a method for task processing provided by one embodiment of the invention;

FIG. 3 is a flowchart of a task processing method provided by another embodiment of the invention;

FIG. 4 is a flowchart of a task processing method provided by another embodiment of the invention;

FIG. 5 is a schematic structural diagram of a task processing device according to an embodiment of the present invention;

fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, are within the scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, apparatus, article, or device that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or device.

The embodiments of the present invention will be described in further detail below with reference to the drawings and embodiments.

Referring to the specification fig. 1, which is a schematic diagram illustrating an implementation environment of a task processing method according to an embodiment of the present invention, as shown in fig. 1, the implementation environment may include at least a task scheduling node 110, at least one task execution node 120, and a publish-and-subscribe message system 130. The publish-subscribe message system 130 may be communicatively coupled to the task scheduling node 110 and the at least one task execution node 120, respectively.

The task scheduling node 110 and the at least one task performing node 120 may include a smart phone, a tablet computer, a laptop computer, a desktop computer, a digital assistant, a smart speaker, a smart wearable device, a vehicle-mounted terminal, a server, and other types of physical devices, and may also include software running in the physical devices, such as an application program, but not limited thereto.

The publish-subscribe message system 130 may be a Kafka system, and the publish-subscribe message system 130 may asynchronously communicate task fragment data between the task scheduling node 110 and the at least one task execution node 120 based on reliable message queues.

In a specific example, the task scheduling node 110 may be a high-frequency scheduler, and the high-frequency scheduler may segment sub-tasks in a batch task (i.e., a main task) to obtain a plurality of corresponding task segments, and send the plurality of task segments to the publish-and-subscribe message system 130. The at least one task execution node 120 may subscribe to and consume messages in the publish-subscribe message system 130 to obtain pending task segments. The at least one task execution node 120 may process the task fragments to be processed by using the task processing method provided in the embodiment of the present invention, determine the execution status or obtain the execution result, which not only can ensure the processing speed of the task, but also can avoid non-functional problems such as database table locking caused by too high system pressure bearing.

It should be noted that fig. 1 is only an example. Those skilled in the art will appreciate that although only 2 task execution nodes 120 are shown in fig. 1, this is not a limitation of the embodiments of the present invention and that more or fewer task execution nodes 120 may be included than shown.

Referring to the specification, fig. 2 illustrates a flow of a task processing method according to an embodiment of the present invention, which may be applied to the task execution node in fig. 1, and specifically as shown in fig. 2, the method may include the following steps:

s210: determining a target main task and a target task type corresponding to a target task fragment, wherein the target task fragment comprises a plurality of subtasks of the target main task.

In the embodiment of the present invention, the task scheduling node may split the target main task into a plurality of subtasks, and then segment the plurality of subtasks, so as to obtain one or more task segments corresponding to the target main task. After the fragmentation is completed, the task scheduling node may distribute the one or more task fragments to different task execution nodes for processing, so as to obtain a processing result. And when the execution of the subtasks in the task fragments in all the task execution nodes is finished, the task scheduling node can collect the execution results of all the task execution nodes to summarize to form a report or perform other required processing. In practical application, various fragmentation methods in the prior art may be adopted to fragment multiple sub-tasks obtained by splitting, which is not limited in the embodiment of the present invention.

In the embodiment of the invention, after each task execution node acquires the target task fragment to be executed, each task execution node can analyze the target task fragment to determine the target main task and the target task type corresponding to the target task fragment, wherein the target task type is the task type of the target main task corresponding to the target task fragment. Specifically, the main tasks may include, but are not limited to, batch collection and payment, splitting and merging of customer information, entering account, collecting and gathering of public deposit, settlement of account, processing of register, daily settlement, production of various reports, batch file service, large message transmission service, insurance company insurance policy acceptance, premium renewal payment, settlement of claim and other batch processing services.

In a specific embodiment, the high-frequency processing framework can be used for processing the main task, and the task processing process is divided into 3 stages of main task preprocessing, sub task execution processing and main task post-processing. The main task preprocessing stage divides the main task into a plurality of subtasks, the subtask execution processing stage executes the subtasks, and the main task post-processing stage summarizes the execution states of the subtasks to form a report or perform other required processing.

The high-frequency processing frame comprises a high-frequency scheduling device, and the high-frequency scheduling device is used for polling the states of each main task and each subtask by setting a plurality of timing polling devices and is responsible for distributing and scheduling batch processing tasks. Specifically, the method can comprise a main task preprocessing polling device, a subtask execution completion polling device, a main task post-processing polling device and the like, wherein the main task preprocessing polling device is used for screening out a main task meeting the condition and distributing the main task to enter a preprocessing stage, the subtask execution completion polling device is used for counting the execution condition of the subtask under a certain main task, the subtask execution completion distribution enters a post-processing stage, and the main task post-processing polling device is used for screening out the main task meeting the condition and distributing the main task to enter a post-processing stage.

After the main task preprocessing stage is finished, a sub-task fragmentation and distribution stage can be added, a plurality of sub-tasks obtained by fragmentation are fragmented, and a plurality of task fragments obtained by fragmentation are distributed to different task execution nodes for execution. The sub-task fragmentation and distribution phase may comprise the steps of:

the method comprises the following steps that firstly, a subtask fragmentation timer can be set and used for regularly screening out a main task meeting subtask fragmentation conditions, and the main task meeting the subtask fragmentation conditions is as follows: the task state is the main task in the state that the main task is preprocessed to be completed (state code 11) and the Kafka message is not transmitted (state code 0), and the main task is preprocessed to be capable of performing sub-task fragmentation.

The states of the main tasks and their corresponding state codes are shown in table 1.

TABLE 1

The status of the Kafka message and its corresponding status code are shown in table 2.

TABLE 2

Status of state	Status code
		Is not sent down	0
Has already issued	1
		Has been consumed	2

And secondly, after the main task meeting the sub-task fragmentation condition is screened out, the execution modes of a plurality of sub-tasks of the main task can be determined, wherein the execution modes comprise a serial mode and a parallel mode. Specifically, the execution manner may be determined according to the task type of the main task, for example, the execution manner of each task type may be configured in advance, and marked by using a serial-parallel flag, and according to the serial-parallel flag corresponding to the task type of the main task, it may be determined whether multiple sub-tasks of the main task need to be serially or quickly parallel.

And thirdly, fragmenting the plurality of subtasks of the main task based on the execution mode to obtain one or more task fragments, wherein each task fragment comprises a plurality of subtasks of the main task. Specifically, when the execution mode is a serial mode, only one fragment is generated, and fragment information is written into a fragment information table; and when the execution mode is the parallel mode, generating a plurality of fragments, and writing the fragment information of the plurality of fragments into a fragment information table.

Optionally, when the execution mode is a parallel mode, a fixed fragmentation mode may be used for fragmentation, that is, the number of task fragments is fixed, and the number of subtasks per task fragment is not fixed. The number of the task fragments can be adjusted according to the configuration information, and the number of the task fragments is not limited in the embodiment of the invention. For example, assuming that one main task includes 10000 subtasks and the number of task slices is fixed to 40, 40 task slices with 250 subtasks may be generated, and information records of the 40 task slices are written into the slice information table.

Optionally, when the execution mode is a parallel mode, a fixed subtask number mode may be used for fragmentation, that is, the number of subtasks of a generated task fragment is fixed, and how many fragments are generated is indefinite. The number of subtasks of the task fragment may be preset. Exemplarily, assuming that one main task includes 10000 subtasks, and the number of subtasks of the generated task fragment is set to 500, 20 task fragments may be generated, and information records of the 20 task fragments are written into the fragment information table.

After fragmentation is complete, the state of the main task may be modified to be in subtask execution (state code 20).

And fourthly, setting a task fragment distribution timer for regularly screening the task fragment messages to be distributed in the fragment information table and sending the task fragment messages to the Kafka system. When screening task fragments, a message distribution filter can be designed for dynamically sending the task fragments to be executed to Kafka, and the working principle of the message distribution filter is as follows: screening task fragments with task states of not starting (state code 00) and not issuing Kafka messages (state code 0), determining task states of other task fragments belonging to the corresponding main task, and if the Kafka messages belonging to the corresponding main task are issued (state code 1) or the other task fragments with messages consumed (state code 2) exceed N, not issuing the task fragments; and if the number of the task fragments is not more than N, sending the task fragments to the Kafka system.

The value of N may be set and adjusted according to actual conditions, for example, may be set to 3, which means that at most 3 task slices may be in parallel processing at the same time, so that not only is the task concurrency ensured, but also the situation that a single task blocks other tasks is avoided. When there are more task execution nodes, the downstream processing capacity is larger, and N may be set to be larger, for example, to be 5, so that 5 task execution nodes may process the same main task at the same time.

Accordingly, task shards distributed to the Kafka system may be consumed by a subtask processing cluster, which may include a plurality of task execution nodes, each of which may manage concurrent shard tasks through its own thread pool. When the task execution node receives the task fragments which need to be executed serially, the task fragments can be only distributed to 1 thread to ensure that each subtask is executed serially, and when the task execution node receives the task fragments which can be executed in parallel, each subtask can be executed by a plurality of threads concurrently.

In the field of financial services, subtask processing clusters are distinguished according to provinces, task fragmentation messages topic are divided according to provinces, and the province subtask processing clusters only subscribe messages belonging to own provinces, so that task blocking does not occur between provinces.

In the subtask execution processing stage, a subtask fragment completion detection timer may be set for monitoring the task fragment processing condition, the subtask fragment completion detection timer may implement timing scheduling through a Schedule timing task of a Spring frame, and a time interval of the timing scheduling may be set according to an actual condition, for example, a check time interval may be set to 5 seconds, and the setting of the time interval takes into account not excessively frequent checks to increase server pressure, and is also not suitable for setting a processing time in which a state transition time is excessively long due to an excessively long detection time interval to affect a single main task.

Specifically, the detection flow of the subtask fragment completion detection timer is as follows:

the method comprises the steps of screening out a main task with a task state being in subtask processing (state code 20) from a database, namely, the main task executing the subtask processing, then inquiring the task states of all task fragments corresponding to the main task in a fragment information table, if the task states of all task fragments corresponding to the main task are all completed (state code 40), indicating that all subtasks of the main task are completely executed, modifying the task state of the main task to be the completion of the subtask processing (state code 21) and the Kafka message state to be undelivered (state code 0), and waiting for the post-processing stage of a subsequent main task to process.

It should be noted that, the above process of performing fragmentation processing on the main task is only an example, and does not limit the embodiment of the present invention, and in some other embodiments, other fragmentation methods may also be used to fragment multiple sub-tasks of the main task to obtain one or multiple task fragments.

S220: and acquiring the first subtask concurrency number of the task fragment corresponding to the target main task.

In this embodiment of the present invention, the task execution node may receive one or more task fragments of the target main task, when the task execution node receives only one task fragment of the target main task, the first subtask concurrency number is a subtask concurrency number of the task fragment (i.e., the target task fragment), and when the task execution node receives a plurality of task fragments of the target main task, the first subtask concurrency number is a sum of subtask concurrency numbers of the plurality of task fragments.

Specifically, the obtaining of the first subtask concurrence number of the task fragment corresponding to the target main task may include:

acquiring a first task fragment set corresponding to the target main task;

In practical application, each task fragment corresponding to the target main task may be determined first, and then the concurrent subtask numbers of each task fragment are added to obtain the first concurrent subtask number.

Specifically, a thread pool may be established in advance for the task execution node, a certain number of threads are stored in the thread pool, an idle thread may be acquired from the thread pool to execute a subtask in a task segment corresponding to each main task, and a concurrent number of subtasks of a certain task segment is a number of threads for executing a subtask in the task segment.

S230: and acquiring a second subtask concurrency number of the task fragment corresponding to the target task type.

In the embodiment of the present invention, the task execution node may execute task fragments corresponding to one or more main tasks in parallel, when the task execution node only executes task fragments of the target main task, the second subtask concurrency number is a sum of subtask concurrency numbers of the task fragments of the target main task, and when the task execution node executes task fragments of a plurality of main tasks, the second subtask concurrency number is a sum of subtask concurrency numbers of task fragments of each main task whose task type is the target task type.

Specifically, the obtaining of the second subtask concurrency number of the task fragment corresponding to the target task type may include:

acquiring a second task fragment set corresponding to the target task type;

and determining the sum of the concurrent numbers of the subtasks of each task fragment in the second task fragment set to obtain the concurrent number of the second subtask.

In practical application, the task type of the main task corresponding to each task fragment in the task execution node may be determined first, each task fragment whose task type of the corresponding main task is the target task type is determined, and then the concurrent numbers of the subtasks of each task fragment are added to obtain the second concurrent subtask number.

It should be noted that the subtask concurrency number mentioned in the embodiment of the present invention refers to the current real-time concurrency number.

S240: and comparing the first subtask concurrency number and the second subtask concurrency number with corresponding concurrency thresholds respectively, and processing the subtasks in the target task fragment according to a comparison result.

In the embodiment of the present invention, two-dimensional flow control may be set: the first dimension is a single main task dimension and is used for controlling the concurrency of subtasks of a certain main task at each task execution node; the second dimension is a single task type dimension and is used for controlling the concurrency of subtasks of a main task of a certain task type at each task execution node. Before executing the subtasks in the target task fragment, whether resources exist in the task execution node can be judged in advance from the two dimensions, if so, at least one subtask in the target task fragment is executed, otherwise, the subtasks in the target task fragment are not executed, and judgment is carried out again after a preset time interval is waited. The preset time interval may be set according to actual conditions, for example, may be set to 1 minute, and the like, which is not limited in this embodiment of the present invention.

In a possible embodiment, with reference to fig. 3 of the specification, the comparing the first and second subtask concurrency numbers with the corresponding concurrency thresholds respectively, and processing the subtasks in the target task slice according to the comparison result may include:

s241: determining a first concurrency threshold for the target primary task and a second concurrency threshold for the target task type.

Specifically, the determining a first concurrency threshold for the target primary task and a second concurrency threshold for the target task type may include:

In this embodiment of the present invention, the target configuration file may be configured in advance, and the target configuration file includes a first concurrency threshold for the target main task and a second concurrency threshold for the target task type. The first concurrency threshold and the second concurrency threshold may be set according to actual situations, and the embodiment of the present invention is not limited to this.

Alternatively, a configuration file may be configured for each task type in advance, and the configuration file corresponding to the target task type corresponding to the target task segment may be used as the target configuration file corresponding to the target task segment.

Optionally, a configuration file may be configured for each primary task, and the configuration file corresponding to the target primary task corresponding to the target task segment is used as the target configuration file corresponding to the target task segment.

Illustratively, the target configuration file may be a TASK type configuration table HFT _ TASK _ CFG, where the TASK type configuration table includes a TASK _ UPLM _ DNUM field for indicating a highest concurrency number of a single main TASK at a TASK execution node, i.e., the first concurrency threshold, and the TASK type configuration table further includes a FWCTL _ UPLM _ DNUM field for indicating a highest concurrency number of a single TASK type at a TASK execution node, i.e., the second concurrency threshold. The TASK type configuration table of the TASK type debit card cost information maintenance (a01925644) is shown in table 3, and it can be seen from table 3 that the value of the TASK _ UPLM _ DNUM field is 20, which means that the highest concurrency number of the subtasks on one node after a single main TASK is sliced is 20, and the value of the FWCTL _ UPLM _ DNUM field is 50, which means that when a plurality of TASK slices are distributed on one node, the highest concurrency number of the subtasks of the TASK type a019225644 is 50. The value of the TASK _ UPLM _ DNUM field and the value of the FWCTL _ UPLM _ DNUM field can be empirical values, and through production verification, the processing speed of TASKs can be guaranteed, and non-functional problems such as database table locking and the like caused by overhigh pressure bearing of a downstream system can be avoided.

TABLE 3

Configuration item	Value of
		HGFRQY_SVC_ECD	A019225644
HGFRQY_SVC_DSC	Debit card cost information maintenance
		TSK_PRTY_CD	5
STSK_MAX_NUM	20,000
		FWCTL_UPLM_DNUM	50
TSK_OVTM_TM	60
		LAST_MOD_TM	2020-12-19 12:30:00
PRMT_ONLN_TXN_CD	[NULL]
		BTCHPCS_DPT_TPCD	5
TASK_UPLM_DNUM	20
		EVENT_ID	[NULL]
TASK_CODE	[NULL]
		RETRY_NUM	5
RETRY_INTERVAL	[NULL]
		TSK_EXEC_TP	1
STSK_EXEC_TP	[NULL]
		CNU_PRP_DT_CFG	[NULL]
TSK_PRTN_TP	[NULL]
		TSK_PRTN_PARM	[NULL]
STSK_OVTM_TM	[NULL]
		PPCS_PTR_CFG	[NULL]
AF_PCSG_RTR_CFG	[NULL]
		STSK_RTR_CFG	[NULL]
PRTN_RETRY_NUM	[NULL]
		RCRD_RGTM	2020-12-19 12:30:00
SYS_RSRN_DMN1	[NULL]
		SYS_RSRN_DMN2	[NULL]
APP_RSRN_DMN1	[NULL]
		LOCK_ECD	[NULL]
TSK_MANAGE_STATUS	0
		STSK_RTR_MDL	0

According to the embodiment of the invention, the configuration file is utilized to maintain the flow control parameters of the single main task dimension and the single task type dimension, so that a user can set the concurrency threshold of the single main task dimension and the single task type dimension according to the actual situation, and the simplification and visualization of flow control adjustment are realized.

S242: comparing the first concurrent number of the subtasks with the first concurrent threshold to obtain a first comparison result;

s243: comparing the second concurrency number of the subtasks with the second concurrency threshold value to obtain a second comparison result;

s244: determining whether subtasks in the target task fragment can be executed according to the first comparison result and the second comparison result;

specifically, the determining whether to execute the subtask in the target task slice according to the first comparison result and the second comparison result may include:

In the embodiment of the present invention, when the concurrency number of the first subtask is smaller than the first concurrency threshold and the concurrency number of the second subtask is smaller than the second concurrency threshold, it indicates that resources of the node still can execute the task, otherwise, it indicates that no resources of the node can execute the task.

In practical applications, two static maps can be designed in the code: the key main key of the sublTaskConcurMap is a task type, and the value is a second subtask concurrence number of the current task type on the node; the key main key of subUnitConcurMap is task identification, and the value is the first subtask concurrency number of the current task identification on the node. The checkuseblethread method can be used to determine whether a single task type dimension can be distributively executed, and the checkusedredgedteadbytaskid method can be used to determine whether a single main task dimension can be distributively executed. If the single task type dimension cannot be distributed and executed or the single main task dimension cannot be distributed, the sleep can wait for a period of time and then judge again.

Specifically, when determining whether a single main task dimension can be distributed and executed by using a checkUsedThreadByTaskId method, a first thread number of a subtask executing a task fragment corresponding to a target main task in a thread pool (that is, a first subtask concurrency number of the task fragment corresponding to the target main task) may be obtained, the first thread number is compared with a first concurrency threshold, if the first thread number is less than the first concurrency threshold, it is determined that a resource exists, a task may be executed, and if the first thread number is greater than or equal to the first concurrency threshold, it is determined that no resource exists, and a task may not be executed.

When judging whether the single task type dimension can be distributed and executed by using a checkusebleThread method, a second thread number (namely, a second subtask concurrency number of the task fragment corresponding to the target task type) for executing the subtask corresponding to the task type in the thread pool can be obtained, the second thread number is compared with a second concurrency threshold, if the second thread number is smaller than the second concurrency threshold, it is judged that a resource exists, the task can be executed, and if the second thread number is larger than or equal to the second concurrency threshold, it is judged that no resource exists, and the task cannot be executed.

S245: and when determining that the subtask in the target task fragment can be executed, executing at least one subtask in the target task fragment.

In the embodiment of the present invention, when it is determined that a subtask in the target task fragment can be executed, at least one subtask in the target task fragment can be executed in parallel through one or more idle threads in a thread pool.

Specifically, the target number of executable subtasks may be determined according to the first concurrency number of subtasks, the first concurrency threshold, the second concurrency number of subtasks, and the second concurrency threshold, and the target number of subtasks may be executed in parallel. For example, a first number of subtasks executable by a single main task dimension may be calculated according to the first concurrency number of subtasks and the first concurrency threshold, a second number of subtasks executable by a single task type dimension may be calculated according to the second concurrency number of subtasks and the second concurrency threshold, and the larger of the first number and the second number may be used as the target number.

In practical applications, before a certain subtask of the target task fragment is executed, the target task type is incremented by one in subtaskcurcpmap, and the target main task is incremented by one in subunitconurcmap. And after the thread pool asynchronously executes a certain subtask of the target task fragment, the two values are correspondingly reduced by one, so that the dynamic realization of flow control is realized.

In a possible embodiment, referring to fig. 4 in combination with the description, after determining the first concurrency threshold for the target main task and the second concurrency threshold for the target task type, the method may further include:

s2411': acquiring current time information;

s2412': determining target control parameters according to the current time information;

s2413': and correcting the first concurrency threshold and/or the second concurrency threshold according to the target control parameter to obtain a corrected first concurrency threshold and/or a corrected second concurrency threshold.

Correspondingly, the comparing the first concurrency number of subtasks with the first concurrency threshold to obtain a first comparison result (i.e., S242) specifically includes:

s242': comparing the first subtask concurrency number with a corrected first concurrency threshold value to obtain a first comparison result;

and/or the presence of a gas in the gas,

the comparing the second concurrency number of the subtasks with the second concurrency threshold to obtain a second comparison result (i.e., S243) specifically includes:

and S243': and comparing the second subtask concurrency number with a modified second concurrency threshold value to obtain a second comparison result.

Specifically, a corresponding relationship between a time period and a control parameter may be preconfigured, and when the current time information is obtained, a target control parameter corresponding to the current time information may be determined according to the time period in which the current time information is located. After determining the target control parameter, the first concurrency threshold and/or the second concurrency threshold may be modified by using the target control parameter, specifically, a product of the target control parameter and the first concurrency threshold may be used as a modified first concurrency threshold, and/or a product of the target control parameter and the second concurrency threshold may be used as a modified second concurrency threshold.

In practical application, the corresponding relation between the time period and the control parameter can be stored in a Redis database to be shared by each task execution node. For example, the rule controlling the parameter may be set to "when: the "percentage" indicates that the time period control parameter of 0 point to 8 points is 100% and the time period control parameter of 8 point to 24 points is 80% if the value is "00: 00@100, 08:00@ 80", that is, the concurrence threshold value in the daytime is controlled to be 80% of the highest concurrence threshold value.

According to the embodiment of the invention, the corresponding relation between the time period and the control parameter is configured in advance, so that the time period dimension flow control is realized, the subtask processing flow can be controlled at a lower level in daily transaction peak periods such as a meal point and a twenty-first-class time period, the too high pressure bearing of a downstream transaction system is avoided, and the stability of the system is ensured.

In summary, according to the task processing method of the embodiment of the present invention, the target main task and the target task type corresponding to the target task fragment are determined, the first subtask concurrency number of the task fragment corresponding to the target main task and the second subtask concurrency number of the task fragment corresponding to the target task type are respectively compared with the corresponding concurrency threshold, and the subtasks in the target task fragment are processed according to the comparison result, so that the flow control of the single task type and the single main task dimension can be realized, while the requirement of processing the subtasks of tens of millions of levels every day is met, the too high pressure bearing of the system is avoided, and the stability and reliability of the system are improved.

The embodiment of the invention designs a set of stateless flow control model, realizes rich and diverse flow control schemes through the flow control design with multiple dimensions, can be selected in a customized manner for a single main task, a single task type and a time period, increases the concurrency capability of subtasks or reduces the concurrency capability of the subtasks, can be operated simply by simply increasing or decreasing task execution nodes, and has clear flow control effect and better user experience.

Referring to fig. 5, a task processing device 500 according to an embodiment of the present invention is shown. As shown in fig. 5, the apparatus 500 may include:

a determining module 510, configured to determine a target main task and a target task type corresponding to a target task fragment, where the target task fragment includes multiple subtasks of the target main task;

a first obtaining module 520, configured to obtain a first subtask concurrence number of a task fragment corresponding to the target main task;

a second obtaining module 530, configured to obtain a second subtask concurrency number of the task fragment corresponding to the target task type;

and the task processing module 540 is configured to compare the first subtask concurrency number and the second subtask concurrency number with corresponding concurrency thresholds, and process the subtasks in the target task fragment according to a comparison result.

In one possible embodiment, the task processing module 540 may include:

a first determining unit configured to determine a first concurrency threshold for the target main task and a second concurrency threshold for the target task type;

the first comparing unit is used for comparing the first subtask concurrency number with the first concurrency threshold value to obtain a first comparison result;

the second comparing unit is used for comparing the second concurrent subtask number with the second concurrent threshold value to obtain a second comparison result;

a second determining unit, configured to determine whether a subtask in the target task fragment can be executed according to the first comparison result and the second comparison result;

and the task execution unit is used for executing at least one subtask in the target task fragment when determining that the subtask in the target task fragment can be executed.

In one possible embodiment, the task processing module 540 may further include:

an acquisition unit configured to acquire current time information;

a third determining unit, configured to determine a target control parameter according to the current time information;

a correction unit, configured to correct the first concurrency threshold and/or the second concurrency threshold according to the target control parameter, so as to obtain a corrected first concurrency threshold and/or a corrected second concurrency threshold;

the first comparing unit is configured to compare the first subtask concurrency number with a modified first concurrency threshold to obtain a first comparison result;

and/or the presence of a gas in the gas,

and the second determining unit is used for comparing the second subtask concurrency number with the modified second concurrency threshold value to obtain a second comparison result.

It should be noted that, when the apparatus provided in the foregoing embodiment implements the functions thereof, only the division of the functional modules is illustrated, and in practical applications, the functions may be distributed by different functional modules according to needs, that is, the internal structure of the apparatus may be divided into different functional modules to implement all or part of the functions described above. In addition, the apparatuses provided in the above embodiments and the corresponding method embodiments belong to the same concept, and specific implementation processes thereof are detailed in the corresponding method embodiments, and are not described herein again.

An embodiment of the present invention further provides an electronic device, which includes a processor and a memory, where the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the task processing method provided by the above method embodiment.

The memory may be used to store software programs and modules, and the processor may execute various functional applications and data processing by operating the software programs and modules stored in the memory. The memory can mainly comprise a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs needed by functions and the like; the storage data area may store data created according to use of the apparatus, and the like. Further, the memory may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory may also include a memory controller to provide the processor access to the memory.

In a specific embodiment, fig. 6 is a schematic diagram illustrating a hardware structure of an electronic device for implementing the task processing method provided by the embodiment of the present invention, where the electronic device may be a computer terminal, a mobile terminal, or another device, and the electronic device may also participate in forming or including the task processing apparatus provided by the embodiment of the present invention. As shown in fig. 6, the electronic device 600 may include one or more computer-readable storage media of the memory 610, one or more processing cores of the processor 620, an input unit 630, a display unit 640, a Radio Frequency (RF) circuit 650, a wireless fidelity (WiFi) module 660, and a power supply 670. Those skilled in the art will appreciate that the electronic device configuration shown in fig. 6 does not constitute a limitation of electronic device 600, and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components. Wherein:

the memory 610 may be used to store software programs and modules, and the processor 620 may execute various functional applications and data processing by operating or executing the software programs and modules stored in the memory 610 and calling data stored in the memory 610. The memory 610 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to use of the electronic device, and the like. In addition, the memory 610 may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device. Accordingly, the memory 610 may also include a memory controller to provide the processor 620 with access to the memory 610.

The processor 620 is a control center of the electronic device 600, connects various parts of the whole electronic device by using various interfaces and lines, and performs various functions of the electronic device 600 and processes data by operating or executing software programs and/or modules stored in the memory 610 and calling data stored in the memory 610, thereby performing overall monitoring of the electronic device 600. The Processor 620 may be a central processing unit, or may be other general-purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, a discrete hardware component, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The input unit 630 may be used to receive input numeric or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control. In particular, the input unit 630 may include a touch sensitive surface 631 as well as other input devices 632. In particular, the touch-sensitive surface 631 may include, but is not limited to, a touch pad or touch screen, and the other input devices 632 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

The display unit 640 may be used to display information input by or provided to a user and various graphical user interfaces of an electronic device, which may be made up of graphics, text, icons, video, and any combination thereof. The Display unit 640 may include a Display panel 641, and optionally, the Display panel 641 may be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like.

The RF circuit 650 may be used for receiving and transmitting signals during information transmission and reception or during a call, and in particular, for receiving downlink information of a base station and then processing the received downlink information by the one or more processors 620; in addition, data relating to uplink is transmitted to the base station. In general, RF circuit 650 includes, but is not limited to, an antenna, at least one Amplifier, a tuner, one or more oscillators, a Subscriber Identity Module (SIM) card, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like. In addition, RF circuit 650 may also communicate with networks and other devices via wireless communications. The wireless communication may use any communication standard or protocol, including but not limited to Global System for Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), and the like.

WiFi belongs to short-distance wireless transmission technology, and the electronic equipment 600 can help a user to send and receive e-mails, browse webpages, access streaming media and the like through the WiFi module 660, and provides wireless broadband Internet access for the user. Although fig. 6 shows WiFi module 660, it is understood that it does not belong to the essential components of electronic device 600, and may be omitted entirely as needed within the scope not changing the essence of the invention.

The electronic device 600 also includes a power supply 670 (e.g., a battery) for powering the various components, which may be logically coupled to the processor 620 via a power management system to manage charging, discharging, and power consumption management functions via the power management system. The power supply 670 may also include one or more dc or ac power sources, recharging systems, power failure detection circuitry, power converters or inverters, power status indicators, or any other component.

It should be noted that, although not shown, the electronic device 600 may further include a bluetooth module, and the like, which is not described herein again.

An embodiment of the present invention further provides a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program for implementing a task processing method, where the at least one instruction or the at least one program is loaded and executed by the processor to implement the task processing method provided by the foregoing method embodiment.

Optionally, in an embodiment of the present invention, the storage medium may include, but is not limited to: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.

An embodiment of the invention also provides a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the task processing method provided in the above-mentioned various alternative embodiments.

It should be noted that: the precedence order of the above embodiments of the present invention is only for description, and does not represent the merits of the embodiments. And specific embodiments thereof have been described above. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing may also be possible or may be advantageous.

The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.

Claims

1. A task processing method, comprising:

acquiring a first subtask concurrency number of a task fragment corresponding to the target main task;

and comparing the first subtask concurrent number and the second subtask concurrent number with corresponding concurrent thresholds respectively, and processing the subtasks in the target task fragment according to a comparison result.

2. The method according to claim 1, wherein the obtaining the first subtask concurrency number of the task fragment corresponding to the target main task comprises:

acquiring a first task fragment set corresponding to the target main task;

3. The method according to claim 1, wherein the obtaining the second subtask concurrency number of the task slice corresponding to the target task type includes:

acquiring a second task fragment set corresponding to the target task type;

4. The method according to claim 1, wherein the comparing the first and second subtask concurrency numbers with corresponding concurrency thresholds respectively, and wherein processing the subtasks in the target task slice according to the comparison result comprises:

5. The method of claim 4, wherein determining a first concurrency threshold for the target primary task and a second concurrency threshold for the target task type comprises:

6. The method of claim 4, wherein after determining the first concurrency threshold for the target primary task and the second concurrency threshold for the target task type, further comprising:

acquiring current time information;

comparing the first concurrent number of the first subtask with a corrected first concurrent threshold to obtain a first comparison result;

and/or the presence of a gas in the gas,

7. The method of claim 4, wherein determining whether a sub-task in the target task slice can be executed according to the first comparison result and the second comparison result comprises:

8. A task processing apparatus, comprising:

9. An electronic device, comprising a processor and a memory, wherein at least one instruction or at least one program is stored in the memory, and the at least one instruction or the at least one program is loaded by the processor and executed to implement the task processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, in which at least one instruction or at least one program is stored, the at least one instruction or the at least one program being loaded and executed by a processor to implement the task processing method according to any one of claims 1 to 7.