CN113485806A - Method, device, equipment and computer readable medium for processing task - Google Patents
Method, device, equipment and computer readable medium for processing task Download PDFInfo
- Publication number
- CN113485806A CN113485806A CN202110752934.0A CN202110752934A CN113485806A CN 113485806 A CN113485806 A CN 113485806A CN 202110752934 A CN202110752934 A CN 202110752934A CN 113485806 A CN113485806 A CN 113485806A
- Authority
- CN
- China
- Prior art keywords
- subtasks
- subtask
- main task
- task
- processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012545 processing Methods 0.000 title claims abstract description 130
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000012216 screening Methods 0.000 claims abstract description 35
- 238000013467 fragmentation Methods 0.000 claims abstract description 25
- 238000006062 fragmentation reaction Methods 0.000 claims abstract description 25
- 238000004590 computer program Methods 0.000 claims description 9
- 238000007781 pre-processing Methods 0.000 claims description 9
- 230000008569 process Effects 0.000 abstract description 19
- 238000010586 diagram Methods 0.000 description 16
- 239000012634 fragment Substances 0.000 description 12
- 230000006870 function Effects 0.000 description 11
- 238000004891 communication Methods 0.000 description 7
- 238000012805 post-processing Methods 0.000 description 5
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000011143 downstream manufacturing Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method, a device, equipment and a computer readable medium for processing tasks, and relates to the technical field of automatic program design. One embodiment of the method comprises: regularly screening out the main tasks meeting the sub-task fragmentation according to the processing state of the main tasks; splitting the main task into one or more subtasks according to the type of the main task, and storing the information of the subtasks in a subtask information table; in the subtask information table, periodically determining the subtasks sent to the distributed cluster according to the execution state of the subtasks under the main task; and in the distributed cluster, executing the subtasks sent to the distributed cluster. According to the method and the device, the current resource processing capacity of other example machines can be dynamically acquired in the processing process of the subtasks, and the resource utilization efficiency is improved.
Description
Technical Field
The present invention relates to the field of automatic programming technology, and in particular, to a method, an apparatus, a device, and a computer-readable medium for processing tasks.
Background
Global IT markets are changing dramatically, and new technologies such as cloud computing, big data, internet of things, and artificial intelligence are rewriting game rules. The host is always favored by large banks due to its high availability and high throughput, but the host device is dependent on a single device, which is not favorable for autonomous availability, the core traffic is rapidly increasing, and the host faces a more and more serious challenge.
With the rise of the distributed architecture, banks actively explore solutions for host downward movement, and can support the batch processing function of the original host through a batch processing framework suitable for the distributed architecture. A subtask fragment processing mechanism is established based on a high-frequency framework, a subtask fragment function is provided, and functions of fragment distribution, fragment claiming, fragment execution and the like are provided so as to meet the requirement of efficiently executing batch processing.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art: in the process of processing the subtasks, the current resource processing capacity of other example machines cannot be dynamically acquired, so that the resource utilization efficiency is low.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method, an apparatus, a device, and a computer-readable medium for processing a task, which can dynamically obtain the current resource processing capacity of other example machines in the process of processing a sub-task, thereby improving the resource utilization efficiency.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a method of processing a task, including:
regularly screening out the main tasks meeting the sub-task fragmentation according to the processing state of the main tasks;
splitting the main task into one or more subtasks according to the type of the main task, and storing the information of the subtasks in a subtask information table;
in the subtask information table, periodically determining the subtasks sent to the distributed cluster according to the execution state of the subtasks under the main task;
and in the distributed cluster, executing the subtasks sent to the distributed cluster.
The method for regularly screening the main tasks meeting the sub-task fragmentation according to the processing state of the main tasks comprises the following steps:
and according to the processing state of the main task, regularly screening out the main task which is finished in the preprocessing of the main task and is not issued by the distributed message.
The processing state of the main task is known by the state code of the main task.
The splitting the main task into one or more subtasks according to the type of the main task includes:
and splitting the main task into a serial subtask or a plurality of parallel subtasks according to the mark of the type of the main task.
The identification of the type of the main task is a serial identification;
the splitting the main task into one or more subtasks according to the type of the main task, and storing the information of the subtasks in a subtask information table includes:
and splitting the main task into a serial subtask according to the type of the main task, and storing the information of the serial subtask in a subtask information table.
The identification of the type of the main task is a parallel identification;
the splitting the main task into one or more subtasks according to the type of the main task, and storing the information of the subtasks in a subtask information table includes:
and splitting the main task into a plurality of parallel subtasks according to the type of the main task, and storing the information of the parallel subtasks in a subtask information table.
The splitting of the main task into a plurality of parallel subtasks includes:
and splitting the main task into sub task groups with preset fragmentation number according to the preset fragmentation number, wherein the sub task groups comprise one or more parallel sub tasks.
The splitting of the main task into a plurality of parallel subtasks includes:
and splitting the main task into one or more subtask groups according to the number of the parallel subtasks and the number of the parallel subtasks in the subtask group.
After storing the information of the subtask in a subtask information table, the method further includes:
and updating the processing state of the main task.
The execution state of the subtasks sent to the distributed cluster comprises not-started and not-issued messages.
The periodically determining the subtasks sent to the distributed cluster according to the execution state of the subtasks under the main task in the subtask information table includes:
screening an unexecuted subtask group in the subtask information table, wherein the execution state of the unexecuted subtask group comprises that the subtask group is not started and a message is not issued;
and sending the unexecuted subtask group to a distributed cluster if the number of the issued subtask groups does not exceed a preset fragmentation threshold, wherein the issued subtask group and the unexecuted subtask group belong to the same main task, and the unexecuted subtask group comprises one or more unexecuted subtasks.
And the issued subtask group and the unexecuted subtask group are processed in parallel.
The periodically determining the subtasks sent to the distributed cluster according to the execution state of the subtasks under the main task in the subtask information table includes:
screening out unexecuted subtasks in the subtask information table, wherein the execution state of the unexecuted subtasks comprises that the subtasks are not started and messages are not issued;
And if the unexecuted subtask group belongs to the serial execution task, sending the unexecuted subtask group to one thread of the distributed cluster.
In the distributed cluster, executing the subtasks sent to the distributed cluster includes:
in the distributed cluster, the subtasks sent to the distributed cluster are executed according to the geographic area.
The method further comprises the following steps:
and inquiring the execution state of the subtask under the main task in the subtask information table to update the processing state of the main task.
The querying the execution state of the subtask under the main task in the subtask information table to update the processing state of the main task includes:
and inquiring the execution state of the subtasks subordinate to the main task in the subtask information table according to a preset inquiry period so as to update the processing state of the main task.
According to a second aspect of the embodiments of the present invention, there is provided an apparatus for processing a task, including:
the screening module is used for regularly screening the main tasks meeting the sub-task fragmentation according to the processing state of the main tasks;
the splitting module is used for splitting the main task into one or more subtasks according to the type of the main task and storing the information of the subtasks in a subtask information table;
The sending module is used for periodically determining the subtasks sent to the distributed cluster in the subtask information table through the execution state of the subtasks under the main task;
and the execution module is used for executing the subtasks sent to the distributed cluster in the distributed cluster.
According to a third aspect of embodiments of the present invention, there is provided an electronic device for processing a task, including:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method as described above.
According to a fourth aspect of embodiments of the present invention, there is provided a computer readable medium, on which a computer program is stored, which when executed by a processor, implements the method as described above.
One embodiment of the above invention has the following advantages or benefits: regularly screening out the main tasks meeting the sub-task fragmentation according to the processing state of the main tasks; splitting the main task into one or more subtasks according to the type of the main task, and storing the information of the subtasks in a subtask information table; in the subtask information table, periodically determining the subtasks sent to the distributed cluster according to the execution state of the subtasks under the main task; and in the distributed cluster, executing the subtasks sent to the distributed cluster. The processing process of each subtask can be obtained through the subtask information table, the current resource processing capacity of other example machines can be dynamically obtained in the processing process of the subtask, and therefore the resource use efficiency is improved.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of a main flow of a method of processing tasks according to an embodiment of the invention;
FIG. 2 is a schematic diagram of a main task split into one or more subtasks, according to an embodiment of the present invention;
FIG. 3 is a flow diagram of a periodic determination of subtasks sent to a distributed cluster, according to an embodiment of the present invention;
FIG. 4 is a flow diagram of another process for periodically determining subtasks to send to a distributed cluster, according to an embodiment of the present invention;
FIG. 5 is a flow diagram illustrating updating a processing state of a primary task according to an embodiment of the invention;
FIG. 6 is a schematic diagram of a main structure of an apparatus for processing tasks according to an embodiment of the present invention;
FIG. 7 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 8 is a schematic structural diagram of a computer system suitable for implementing a terminal device or a server according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
With the rise of the distributed architecture, the future development initiative can be mastered by relying on an enterprise-level distributed platform and adopting an open technology to build a core system.
The method is suitable for a batch processing framework of a distributed architecture and supports the batch processing function of the original host. The high-frequency scheduling device in the parallel computing of the batch processing framework is responsible for the functions of disassembling, distributing, executing, summarizing and the like of batch disassembling single services, and needs to support the processing capacity of about 2000-3000 ten thousand services in one day, such as batch collection and payment, splitting and merging of client information, posting and the like. The initiated batch main task is about 200- > 300 thousands of batches, and most of the batch single tasks are large in batch and single task splitting, and one main task has up to 10 thousands of subtasks to be processed.
Specifically, the high-frequency processing framework divides the batch order splitting task into 3 stages, namely main task preprocessing, sub-task execution processing and main task post-processing. The main task preprocessing stage is responsible for splitting the main task into single subtasks, the subtask execution processing is responsible for executing single online transactions, and the main task post-processing is responsible for summarizing the execution conditions of the subtasks to form a report or perform other required processing.
The high-frequency scheduling device inspects each main task through a plurality of timing inspectors, and the state of the subtasks is responsible for distributing and scheduling batch processing tasks. The timing polling device comprises a main task preprocessing polling device: screening out the main tasks meeting the conditions and distributing the main tasks to enter a preprocessing stage; and the subtask execution completion polling device: counting the execution condition of a subtask under a certain main task, and entering a post-processing stage after the distribution of the execution completion; a main task post-processing polling device: and screening out the main tasks meeting the conditions and distributing the main tasks to enter a post-processing stage.
In the original processing system, after a main task enters a sub-task processing stage, a single sub-task can be executed on one processor only, and the sub-tasks are managed through a thread pool concurrently. The processing mode can well complete the task under the condition of small task amount, and is simple to realize and easy to understand. However, in the presence of tens of millions of batch splitting tasks, the cluster parallel processing capacity is necessarily maximized, and the batch processing speed is increased.
However, existing subtask processing is managed concurrently through a thread pool, and cannot be concurrent on a multi-instance machine. Because the subtask processing cannot dynamically acquire the current resource processing capacity of other example machines, the resources are blocked or idle.
In order to solve the technical problem of low resource utilization efficiency, the following technical scheme in the embodiment of the invention can be adopted.
Referring to fig. 1, fig. 1 is a schematic diagram of a main flow of a method for processing a task according to an embodiment of the present invention, and an execution state of the subtask is known through a subtask information table, and the subtask is processed in time. As shown in fig. 1, the method specifically comprises the following steps:
s101, screening out the main tasks meeting the sub-task fragmentation at regular time according to the processing state of the main tasks.
In embodiments of the present invention, each main task can be divided into one or more subtasks. That is, a main task is subordinate to one or more subtasks, each of which has a subordinate main task.
To process tasks in time, a table marks the processing state of each primary task for the primary tasks. As an example, the processing state of the main task is known by the state code of the main task.
Referring to table 1, table 1 includes a status code corresponding to the processing status of the main task, and it can be known that the processing status of the main task can be clearly and timely obtained through the status code of the main task.
TABLE 1
| Status of state | Status code |
| Not started | 00 |
| In the pretreatment | 10 |
| Completion of pretreatment | 11 |
| In the sub-task processing | 20 |
| Subtask processing completion | 21 |
| In the post-treatment | 30 |
| Complete the process | 40 |
In an embodiment of the invention, the main tasks which are preprocessed and distributed messages are not issued are screened out at regular time according to the processing state of the main tasks.
During specific implementation, the subtask fragment timer is used for regularly screening out the main tasks which are preprocessed and distributed messages are not issued according to the processing state of the main tasks. Namely, the screening conditions were: the main task pre-processing is completed and the distributed message is not sent down.
The screening conditions were as specified in table 1: the main task pre-processes the complete (11) state and the distributed message is not sent down (00) state. The main task meeting the screening condition represents that the main task has finished preprocessing, and a sub-task fragmentation process can be performed.
S102, splitting the main task into one or more subtasks according to the type of the main task, and storing the information of the subtasks in a subtask information table.
In the sub-task fragment execution process, the main task needs to be split into one or more sub-tasks according to the type of the main task, and the information of the sub-tasks is stored in a sub-task information table. Therefore, the subtasks can be processed efficiently in time based on the content in the subtask information table.
Referring to fig. 2, fig. 2 is a schematic diagram of a main task being split into one or more subtasks according to an embodiment of the present invention.
In particular, the type of primary task is determined by the identity. Then, in fig. 2, the main task is split into one serial subtask or a plurality of parallel subtasks according to the flag of the type of the main task. As one example, the identification of the primary task includes serial identification and parallel identification.
In an embodiment of the present invention, the identifier of the type of the main task is a serial identifier, and then the main task is split into serial subtasks according to the type of the main task, and the information of the serial subtasks is stored in the subtask information table.
In an embodiment of the present invention, the identifier of the type of the main task is a parallel identifier, and then the main task is divided into a plurality of parallel subtasks according to the type of the main task, and the information of the parallel subtasks is stored in the subtask information table.
It is considered that in the distributed cluster, tasks are executed in units of slices in many cases. Then, in the case where there are a plurality of subtasks, the plurality of subtasks may be divided into slices so that the tasks are performed in the slices. The method specifically comprises the following two modes, namely a mode one and a mode two.
The first method is as follows:
the fixed slicing mode means that the number of slices is fixed, and the number of subtasks per slice is not fixed. Specifically, according to the preset number of fragments, the main task is divided into sub-task groups with the preset number of fragments, and each sub-task group comprises one or more parallel sub-tasks. Wherein the subtask group corresponds to the above slice.
As an example, a main task has 10000 subtasks, and the preset number of the pieces is 10, the main task is divided into 10 subtask groups, and each subtask group includes 1000 subtasks.
The second method comprises the following steps:
the fixed subtask number mode means that the number of subtasks for a slice is fixed, and how many slices are generated is not fixed. Specifically, the main task is divided into one or more subtask groups according to the number of parallel subtasks and the number of parallel subtasks in the subtask group.
As an example, a main task has 10000 subtasks, and the number of parallel subtasks in a subtask group is 500, the main task is divided into 20 subtask groups.
In an embodiment of the present invention, after the information of the subtask is stored in the subtask information table, which indicates that the main task to which the subtask belongs is processing, the processing state of the main task may be updated, so as to filter the main task again.
S103, in the subtask information table, the subtask sent to the distributed cluster is periodically determined according to the execution state of the subtask under the main task.
The subtask information table includes the execution states of a plurality of subtasks, such as: the subtask information table includes an execution state of the subtask group, and the execution state of each subtask group includes an execution state of a plurality of subtasks.
As an example: if the status code is 0, the execution status of the subtask is not issued; if the status code is 1, the execution status of the subtask is issued; state code 2, then the execution state of the subtask is consumed.
In an embodiment of the present invention, the execution state of the subtask under the main task is obtained in the subtask information table, and the subtask sent to the distributed cluster is periodically determined. The execution status of the subtasks sent to the distributed cluster includes not started and messages not sent down. That is, the subtask can only be sent to the distributed cluster if the execution state includes not started and the message is not delivered.
Referring to fig. 3, fig. 3 is a flowchart of periodically determining a subtask sent to a distributed cluster according to an embodiment of the present invention, which specifically includes the following steps:
s301, screening out the unexecuted subtask group in the subtask information table, wherein the execution state of the unexecuted subtask group comprises that the information is not issued and the information is not started.
The subtask information table includes an execution state of the subtask group. And screening the unexecuted subtask group based on the execution state of the subtask group. The execution state of the unexecuted subtask group includes not started and not transmitted.
As one example, the unexecuted subtask group is filtered out periodically by the subtask fragment distribution timer.
S302, if the number of the issued subtask groups does not exceed the preset fragmentation threshold, sending the unexecuted subtask groups to the distributed cluster, wherein the issued subtask groups and the unexecuted subtask groups belong to the same main task, and the unexecuted subtask groups comprise one or more unexecuted subtasks.
After the unexecuted subtask group is screened out, in order to ensure the concurrent execution of the subtasks, the situation that other tasks are blocked by a single task is avoided. It is desirable to limit the execution of tasks, in particular by a preset allocation threshold.
The main task is subordinate to the issued subtask group and the unexecuted subtask group. The issued subtask group is a subtask group that has been sent to execution in the distributed cluster. The set of unexecuted subtasks is a set of subtasks that are not sent to execution in the distributed cluster. As one example, the set of issued subtasks is processed in parallel with the set of unexecuted subtasks. It will be appreciated that the issued subtask group and the unexecuted subtask group belong to parallel tasks.
If the number of the issued subtask groups does not exceed the preset fragmentation threshold, sending the unexecuted subtask groups to the distributed cluster; correspondingly, if the number of the issued subtask groups exceeds the preset fragmentation threshold, the transmission of the unexecuted subtask groups to the distributed cluster is stopped.
As an example, a preset allocation threshold equal to 3 would represent that at most 3 subtask groups are in parallel processing for the same main task at the same time. Therefore, the concurrency degree is ensured, and the condition that a single task blocks other tasks is also ensured.
The preset allocation threshold may be adjusted based on the number of machines. If the number of machines is larger and the downstream processing capacity is larger, the preset allocation threshold may be set larger, for example, the preset allocation threshold is equal to 5. Thus, 5 instance machines can process the same main task at the same time, and each instance manages concurrent fragmentation tasks through the thread pool of the instance.
In embodiments of the invention, the message distribution filter dynamically sends and does not perform subtasks to organize the distributed cluster. As one example, the distributed cluster is a kafka cluster.
In the embodiment of fig. 3, the parallel subtasks are sent to the distributed cluster by presetting an allocation threshold.
Referring to fig. 4, fig. 4 is another flowchart of periodically determining a subtask sent to a distributed cluster according to an embodiment of the present invention, which specifically includes the following steps:
s401, screening out the unexecuted subtask group in the subtask information table, wherein the execution state of the unexecuted subtask group comprises that the information is not issued and the information is not started.
Similarly to S301, the unexecuted subtask group is screened out through the subtask information table.
S402, if the unexecuted subtask group belongs to the serial execution task, the unexecuted subtask group is sent to one thread of the distributed cluster.
And the unexecuted subtask group belongs to a serial execution task, and is sent to one thread of the distributed cluster in order to ensure that the unexecuted subtask group is executed in a serial mode.
As an example, when a subtask consumption instance receives a group of unexecuted subtasks that require serial execution, only 1 thread of the group of unexecuted subtasks will be allocated to ensure that the subtasks execute serially, rather than concurrently. The clustered subtask fragmented messages that are distributed to the kafka messages will be consumed by the subtask processing clusters.
In the embodiment of FIG. 4, for serially executing tasks, the set of unexecuted subtasks is sent to one thread of the distributed cluster.
And S104, in the distributed cluster, executing the subtasks sent to the distributed cluster.
The distributed cluster includes a plurality of instance machines that execute the subtasks sent to the distributed cluster.
In one embodiment of the invention, in order to reduce instances of task congestion, in a distributed cluster, the subtasks sent to the distributed cluster are executed by geographical area.
As an example, the subtask processing clusters are divided according to provinces, the subtask fragmentation messages are divided according to provinces, and the province subtask processing clusters only subscribe to messages belonging to the own province, so that no task congestion occurs between provinces.
In one embodiment of the present invention, in order to monitor the current task processing situation, the execution state of the subtask subordinate to the main task in the subtask information table may be queried to update the processing state of the main task.
As an example, according to a preset query period, the execution state of the subtask subordinate to the main task in the subtask information table is queried to update the processing state of the main task. And the processing state of the main task can be updated in time by periodical query. Such as: the preset query period is determined by the Schedule timing task of the Spring framework.
Specifically, the scheduling is realized through a Schedule timing task of a Spring framework, and the preset query period is 5 seconds. Presetting the setting of a query period, and considering excessively frequent queries to increase the pressure of the server; if the set time interval of the preset query period is too long, the processing state transfer time is too long, and the processing time of a single main task is affected.
Referring to fig. 5, fig. 5 is a schematic flowchart of a process state of updating a main task according to an embodiment of the present invention, which specifically includes:
s501, inquiring a main task.
And (5) screening out the main task with the main task state of 20 from the database, wherein 20 represents the sub-task processing. The screened main task is the main task which is executing the sub-task processing.
And S502, judging whether the main task exists.
If the main task exists, executing S503; and if the main task does not exist, ending the process.
S503, traversing all the subordinate subtask groups of the main task.
And inquiring the execution state of the sub-task group under the screened main task in the sub-task information table. If the execution state of all the subtask groups subordinate to the main task is complete and the completed code is 40, all the subtasks subordinate to the main task are completed.
And S504, updating the processing state of the main task.
The processing state of the modification master task is 21 and the message state is 0. Wherein, 21 represents that the subtask processing is completed, and 0 represents that the subtask is not transmitted.
Referring to fig. 6, fig. 6 is a schematic diagram of a main structure of a device for processing a task according to an embodiment of the present invention, where the device for processing a task may implement a method for processing a task, and as shown in fig. 6, the device for processing a task specifically includes:
the screening module 601 is used for regularly screening the main tasks meeting the sub-task fragmentation according to the processing state of the main tasks;
a splitting module 602, configured to split the main task into one or more subtasks according to the type of the main task, and store information of the subtasks in a subtask information table;
a sending module 603, configured to periodically determine, in the subtask information table, a subtask to be sent to the distributed cluster according to an execution state of a subtask subordinate to the main task;
an executing module 604, configured to execute, in the distributed cluster, the sub-task sent to the distributed cluster.
In an embodiment of the present invention, the screening module 601 is specifically configured to screen out, at regular time, a main task for which preprocessing of the main task is completed and the distributed message is not delivered according to a processing state of the main task.
In one embodiment of the invention, the processing state of the primary task is known by the state code of the primary task.
In an embodiment of the present invention, the splitting module 602 is specifically configured to split the main task into one serial subtask or multiple parallel subtasks according to the flag of the type of the main task.
In one embodiment of the invention, the identification of the type of the primary task is a serial identification;
the splitting module 602 is specifically configured to split the main task into a serial subtask according to the type of the main task, and store information of the serial subtask in a subtask information table.
In one embodiment of the invention, the identification of the type of the primary task is a parallel identification;
the splitting module 602 is specifically configured to split the main task into multiple parallel subtasks according to the type of the main task, and store information of the parallel subtasks in a subtask information table.
In an embodiment of the present invention, the splitting module 602 is specifically configured to split the main task into sub-task groups with a preset number of fragments according to the preset number of fragments, where the sub-task group includes one or more parallel sub-tasks.
In an embodiment of the present invention, the splitting module 602 is specifically configured to split the main task into one or more subtask groups according to the number of the parallel subtasks and the number of the parallel subtasks in the subtask group.
In an embodiment of the present invention, the splitting module 602 is further configured to update the processing status of the main task.
In an embodiment of the present invention, the execution status of the subtasks sent to the distributed cluster includes not started and not sent down.
In an embodiment of the present invention, the sending module 603 is specifically configured to screen out an unexecuted subtask group from the subtask information table, where an execution state of the unexecuted subtask group includes that the subtask group is not started and a message is not issued;
and sending the unexecuted subtask group to a distributed cluster if the number of the issued subtask groups does not exceed a preset fragmentation threshold, wherein the issued subtask group and the unexecuted subtask group belong to the same main task, and the unexecuted subtask group comprises one or more unexecuted subtasks.
In one embodiment of the invention, the set of issued subtasks is processed in parallel with the set of unexecuted subtasks.
In an embodiment of the present invention, the sending module 603 is specifically configured to screen out an unexecuted subtask in the subtask information table, where an execution state of the unexecuted subtask includes that the unexecuted subtask is not started and a message is not issued;
and if the unexecuted subtask group belongs to the serial execution task, sending the unexecuted subtask group to one thread of the distributed cluster.
In an embodiment of the present invention, the executing module 604 is specifically configured to execute, in the distributed cluster, the subtasks sent to the distributed cluster according to the geographic area.
In an embodiment of the present invention, the screening module 601 is further configured to query an execution state of a subtask subordinate to the main task in the subtask information table, so as to update the processing state of the main task.
In an embodiment of the present invention, the screening module 601 is specifically configured to query, according to a preset query period, an execution state of a subtask subordinate to the main task in the subtask information table, so as to update the processing state of the main task.
In an embodiment of the present invention, the preset query period is determined by a Schedule timing task of a Spring framework.
Fig. 7 illustrates an exemplary system architecture 700 of a method of processing tasks or a device for processing tasks to which embodiments of the invention may be applied.
As shown in fig. 7, the system architecture 700 may include terminal devices 701, 702, 703, a network 704, and a server 705. The network 704 serves to provide a medium for communication links between the terminal devices 701, 702, 703 and the server 705. Network 704 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
A user may use the terminal devices 701, 702, 703 to interact with a server 705 over a network 704, to receive or send messages or the like. The terminal devices 701, 702, 703 may have installed thereon various communication client applications, such as a shopping-like application, a web browser application, a search-like application, an instant messaging tool, a mailbox client, social platform software, etc. (by way of example only).
The terminal devices 701, 702, 703 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 705 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the terminal devices 701, 702, 703. The backend management server may analyze and perform other processing on the received data such as the product information query request, and feed back a processing result (for example, target push information, product information — just an example) to the terminal device.
It should be noted that the method for processing the task provided by the embodiment of the present invention is generally executed by the server 705, and accordingly, a device for processing the task is generally disposed in the server 705.
It should be understood that the number of terminal devices, networks, and servers in fig. 7 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 8, shown is a block diagram of a computer system 800 suitable for use with a terminal device implementing an embodiment of the present invention. The terminal device shown in fig. 8 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 8, the computer system 800 includes a Central Processing Unit (CPU)801 that can perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)802 or a program loaded from a storage section 808 into a Random Access Memory (RAM) 803. In the RAM 803, various programs and data necessary for the operation of the system 800 are also stored. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.
The following components are connected to the I/O interface 805: an input portion 806 including a keyboard, a mouse, and the like; an output section 807 including a signal such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN card, a modem, or the like. The communication section 809 performs communication processing via a network such as the internet. A drive 810 is also connected to the I/O interface 805 as necessary. A removable medium 811 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 810 as necessary, so that a computer program read out therefrom is mounted on the storage section 808 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and/or installed from the removable medium 811. The computer program executes the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 801.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a screening module, a splitting module, a sending module, and an execution module. Where the names of these modules do not in some cases constitute a limitation on the module itself, for example, a screening module may also be described as "for timing out a primary task that satisfies a sub-task slice according to the processing state of the primary task".
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, cause the device to comprise:
regularly screening out the main tasks meeting the sub-task fragmentation according to the processing state of the main tasks;
splitting the main task into one or more subtasks according to the type of the main task, and storing the information of the subtasks in a subtask information table;
in the subtask information table, periodically determining the subtasks sent to the distributed cluster according to the execution state of the subtasks under the main task;
and in the distributed cluster, executing the subtasks sent to the distributed cluster.
According to the technical scheme of the embodiment of the invention, the main task meeting the sub-task fragmentation is screened out at regular time according to the processing state of the main task; splitting the main task into one or more subtasks according to the type of the main task, and storing the information of the subtasks in a subtask information table; in the subtask information table, periodically determining the subtasks sent to the distributed cluster according to the execution state of the subtasks under the main task; and in the distributed cluster, executing the subtasks sent to the distributed cluster. The processing process of each subtask can be obtained through the subtask information table, the current resource processing capacity of other example machines can be dynamically obtained in the processing process of the subtask, and therefore the resource use efficiency is improved.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (19)
1. A method of processing a task, comprising:
regularly screening out the main tasks meeting the sub-task fragmentation according to the processing state of the main tasks;
splitting the main task into one or more subtasks according to the type of the main task, and storing the information of the subtasks in a subtask information table;
in the subtask information table, periodically determining the subtasks sent to the distributed cluster according to the execution state of the subtasks under the main task;
and in the distributed cluster, executing the subtasks sent to the distributed cluster.
2. The method for processing tasks according to claim 1, wherein the step of periodically screening out the main tasks satisfying the sub-task fragmentation according to the processing state of the main task comprises the following steps:
And according to the processing state of the main task, regularly screening out the main task which is finished in the preprocessing of the main task and is not issued by the distributed message.
3. A method of processing tasks according to claim 1, characterized in that the processing state of the main task is known by the state code of the main task.
4. The method for processing tasks according to claim 1, wherein the splitting the main task into one or more subtasks according to the type of the main task comprises:
and splitting the main task into a serial subtask or a plurality of parallel subtasks according to the mark of the type of the main task.
5. A method of processing tasks according to claim 1, characterized in that the identification of the type of the primary task is a serial identification;
the splitting the main task into one or more subtasks according to the type of the main task, and storing the information of the subtasks in a subtask information table includes:
and splitting the main task into a serial subtask according to the type of the main task, and storing the information of the serial subtask in a subtask information table.
6. A method of processing tasks according to claim 1, characterized in that the identification of the type of the main task is a parallel identification;
The splitting the main task into one or more subtasks according to the type of the main task, and storing the information of the subtasks in a subtask information table includes:
and splitting the main task into a plurality of parallel subtasks according to the type of the main task, and storing the information of the parallel subtasks in a subtask information table.
7. The method of processing tasks according to claim 6, wherein the splitting the main task into a plurality of parallel subtasks comprises:
and splitting the main task into sub task groups with preset fragmentation number according to the preset fragmentation number, wherein the sub task groups comprise one or more parallel sub tasks.
8. The method of processing tasks according to claim 6, wherein the splitting the main task into a plurality of parallel subtasks comprises:
and splitting the main task into one or more subtask groups according to the number of the parallel subtasks and the number of the parallel subtasks in the subtask group.
9. The method for processing tasks according to claim 1, wherein after storing the information of the subtasks in a subtask information table, the method further comprises:
And updating the processing state of the main task.
10. The method of claim 1, wherein the status of the sending of the subtasks to the distributed cluster includes not started and not sent messages.
11. The method for processing tasks according to claim 1, wherein the periodically determining the subtasks sent to the distributed cluster according to the execution status of the subtasks under the main task in the subtask information table comprises:
screening an unexecuted subtask group in the subtask information table, wherein the execution state of the unexecuted subtask group comprises that the subtask group is not started and a message is not issued;
and sending the unexecuted subtask group to a distributed cluster if the number of the issued subtask groups does not exceed a preset fragmentation threshold, wherein the issued subtask group and the unexecuted subtask group belong to the same main task, and the unexecuted subtask group comprises one or more unexecuted subtasks.
12. The method of processing tasks of claim 11, wherein the set of issued subtasks is processed in parallel with the set of unexecuted subtasks.
13. The method for processing tasks according to claim 1, wherein the periodically determining the subtasks sent to the distributed cluster according to the execution status of the subtasks under the main task in the subtask information table comprises:
Screening out unexecuted subtasks in the subtask information table, wherein the execution state of the unexecuted subtasks comprises that the subtasks are not started and messages are not issued;
and if the unexecuted subtask group belongs to the serial execution task, sending the unexecuted subtask group to one thread of the distributed cluster.
14. The method for processing tasks according to claim 1, wherein the executing the subtasks sent to the distributed cluster in the distributed cluster comprises:
in the distributed cluster, the subtasks sent to the distributed cluster are executed according to the geographic area.
15. The method of processing tasks according to claim 1, further comprising:
and inquiring the execution state of the subtask under the main task in the subtask information table to update the processing state of the main task.
16. The method for processing tasks according to claim 15, wherein the querying the execution status of the subtasks subordinate to the main task in the subtask information table to update the processing status of the main task comprises:
and inquiring the execution state of the subtasks subordinate to the main task in the subtask information table according to a preset inquiry period so as to update the processing state of the main task.
17. An apparatus for processing tasks, comprising:
the screening module is used for regularly screening the main tasks meeting the sub-task fragmentation according to the processing state of the main tasks;
the splitting module is used for splitting the main task into one or more subtasks according to the type of the main task and storing the information of the subtasks in a subtask information table;
the sending module is used for periodically determining the subtasks sent to the distributed cluster in the subtask information table through the execution state of the subtasks under the main task;
and the execution module is used for executing the subtasks sent to the distributed cluster in the distributed cluster.
18. An electronic device for processing tasks, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-16.
19. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-16.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110752934.0A CN113485806A (en) | 2021-07-02 | 2021-07-02 | Method, device, equipment and computer readable medium for processing task |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110752934.0A CN113485806A (en) | 2021-07-02 | 2021-07-02 | Method, device, equipment and computer readable medium for processing task |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113485806A true CN113485806A (en) | 2021-10-08 |
Family
ID=77939746
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110752934.0A Pending CN113485806A (en) | 2021-07-02 | 2021-07-02 | Method, device, equipment and computer readable medium for processing task |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113485806A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114020434A (en) * | 2021-11-09 | 2022-02-08 | 中国建设银行股份有限公司 | Task processing method and device, electronic equipment and storage medium |
| CN115237576A (en) * | 2022-08-10 | 2022-10-25 | 中国建设银行股份有限公司 | Serial-parallel hybrid processing method, device, equipment and medium |
| CN115866017A (en) * | 2023-02-27 | 2023-03-28 | 天翼云科技有限公司 | Message processing method, message processing device, communication equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107479962A (en) * | 2016-06-08 | 2017-12-15 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus for mission dispatching |
| US20180276031A1 (en) * | 2015-09-15 | 2018-09-27 | Alibaba Group Holding Limited | Task allocation method and system |
| CN110673959A (en) * | 2019-10-14 | 2020-01-10 | 京东数字科技控股有限公司 | System, method and apparatus for processing tasks |
| CN111274052A (en) * | 2020-01-19 | 2020-06-12 | 中国平安人寿保险股份有限公司 | Data distribution method, server, and computer-readable storage medium |
| CN111400012A (en) * | 2020-03-20 | 2020-07-10 | 中国建设银行股份有限公司 | Data parallel processing method, device, equipment and storage medium |
-
2021
- 2021-07-02 CN CN202110752934.0A patent/CN113485806A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180276031A1 (en) * | 2015-09-15 | 2018-09-27 | Alibaba Group Holding Limited | Task allocation method and system |
| CN107479962A (en) * | 2016-06-08 | 2017-12-15 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus for mission dispatching |
| CN110673959A (en) * | 2019-10-14 | 2020-01-10 | 京东数字科技控股有限公司 | System, method and apparatus for processing tasks |
| CN111274052A (en) * | 2020-01-19 | 2020-06-12 | 中国平安人寿保险股份有限公司 | Data distribution method, server, and computer-readable storage medium |
| CN111400012A (en) * | 2020-03-20 | 2020-07-10 | 中国建设银行股份有限公司 | Data parallel processing method, device, equipment and storage medium |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114020434A (en) * | 2021-11-09 | 2022-02-08 | 中国建设银行股份有限公司 | Task processing method and device, electronic equipment and storage medium |
| CN115237576A (en) * | 2022-08-10 | 2022-10-25 | 中国建设银行股份有限公司 | Serial-parallel hybrid processing method, device, equipment and medium |
| CN115866017A (en) * | 2023-02-27 | 2023-03-28 | 天翼云科技有限公司 | Message processing method, message processing device, communication equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108388479B (en) | Delayed message pushing method and device, computer equipment and storage medium | |
| CN104915407B (en) | A kind of resource regulating method based under Hadoop multi-job environment | |
| CN113485806A (en) | Method, device, equipment and computer readable medium for processing task | |
| US20190228303A1 (en) | Method and apparatus for scheduling resource for deep learning framework | |
| US10515326B2 (en) | Database systems and related queue management methods | |
| CN109408205B (en) | Task scheduling method and device based on hadoop cluster | |
| CN108549583B (en) | Big data processing method and device, server and readable storage medium | |
| CN110019339B (en) | Data query method and system | |
| CN110858197A (en) | Method and device for synchronizing data | |
| CN112099937A (en) | Resource management method and device | |
| CN109428926B (en) | Method and device for scheduling task nodes | |
| CN111478781A (en) | Message broadcasting method and device | |
| CN115658311A (en) | Resource scheduling method, device, equipment and medium | |
| CN111461583B (en) | Inventory checking method and device | |
| CN112398669A (en) | Hadoop deployment method and device | |
| CN117331678B (en) | Heterogeneous computing power federation-oriented multi-cluster job resource specification computing method and system | |
| CN114116173A (en) | Method, device and system for dynamically adjusting task allocation | |
| CN112395337B (en) | Data export method and device | |
| CN113742389A (en) | Service processing method and device | |
| Liu et al. | KubFBS: A fine‐grained and balance‐aware scheduling system for deep learning tasks based on kubernetes | |
| CN113821506A (en) | Task execution method, device, system, server and medium for task system | |
| CN113760638A (en) | Log service method and device based on kubernets cluster | |
| CN111767126A (en) | System and method for distributed batch processing | |
| CN109146672B (en) | Matching method and device based on production mode | |
| CN113407108A (en) | Data storage method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |