CN110928655A

CN110928655A - Task processing method and device

Info

Publication number: CN110928655A
Application number: CN201911095979.4A
Authority: CN
Inventors: 旷波; 王大飞; 文海荣; 江旻
Original assignee: WeBank Co Ltd
Current assignee: WeBank Co Ltd
Priority date: 2019-11-11
Filing date: 2019-11-11
Publication date: 2020-03-27

Abstract

The embodiment of the invention relates to the technical field of financial technology (Finech) and discloses a task processing method and a task processing device. The resource using condition of the executing node is monitored in real time, so that the executing node executes the schedulable task under the condition that the task concurrency amount of the executing node and the task concurrency amount of the queue to which the task belongs are not fully loaded, the task to be processed in the executing node can be always within the available range of the resource, the task to be processed in the executing node can be prevented from being blocked, and the task processing efficiency is improved.

Description

Task processing method and device

Technical Field

The present invention relates to the field of data processing, and in particular, to a task processing method and apparatus.

Background

With the development of computer technology, more and more technologies are applied in the financial field, and the traditional financial industry is gradually changing to financial technology (Fintech), however, due to the requirements of security and real-time performance of the financial industry, the financial technology field puts higher demands on the technologies. The financial industry generally relates to batch processing operation, and for a large number of tasks related to the batch processing operation, the financial industry needs to ensure the accuracy, safety and unreleasability of task processing as much as possible, so that the financial industry is required to rapidly and timely schedule various tasks when the batch processing operation is executed, and the efficiency of batch task processing is ensured.

The Quartz frame is a commonly used scheduling frame, developers can develop their own task scheduling system based on the frame, the task scheduling system developed based on the Quartz frame usually needs to set a task distribution platform, and the developers set the corresponding relationship between tasks and nodes in the task distribution platform in advance, so that the task distribution platform distributes each task in the task scheduling system based on the corresponding relationship between tasks and nodes. However, when a batch processing job is executed, there may be a node in the task scheduling system that needs to execute a large number of tasks, and the resource of the node cannot support the corresponding number of tasks, so if the tasks are still distributed based on this method, the tasks to be processed in the node may be blocked, and thus the efficiency of task processing is low.

In summary, a task processing method is needed to solve the technical problems of low task processing efficiency and easy task blocking caused by distributing batch tasks based on a task distribution platform in the prior art.

Disclosure of Invention

The embodiment of the invention provides a task processing method and a task processing device, which are used for solving the technical problems of low task processing efficiency and easy task blockage caused by distributing batch tasks based on a task distribution platform in the prior art.

In a first aspect, a task processing method provided in an embodiment of the present invention is applied to any execution node in a task scheduling system, where a plurality of queues are arranged in the execution node, and a sum of full-load task concurrency amounts of the plurality of queues is greater than a full-load task concurrency amount of the execution node, and the method includes:

monitoring the scheduling state of each task in a task set to be processed, and after monitoring that the task is in a schedulable state, if the task concurrency amount of a queue to which the task belongs in the execution node is smaller than the full-load task concurrency amount of the queue to which the task belongs and the task concurrency amount of the execution node is smaller than the full-load task concurrency amount of the execution node, setting the state of the task in the task set to be a locked state, and executing the task; wherein the locked state of the task is used to avoid other executing nodes from executing the task.

In the design, the execution node in the task scheduling system can monitor the use condition of the resource in real time, and before the schedulable task is executed, the task concurrency amount of the queue to which the task in the execution node belongs and the task concurrency amount of the execution node are inquired, and the task is executed again under the condition that the execution node and the execution node are determined to be not fully loaded, so that the task to be processed in the execution node is always within the available range of the resource, the task to be processed in the execution node can be prevented from being blocked, and the task processing efficiency is improved.

In a possible design, after the task is monitored to be in a schedulable state, determining a superior task on which the task depends according to a dependency relationship of a plurality of tasks in the task set to be processed, determining an execution state of the superior task on which the task depends from the task set to be processed, if the execution state of the superior task is completed, determining whether a task concurrency amount of a queue to which the task belongs in the execution node is smaller than a full-load task concurrency amount of the queue to which the task belongs, and if the execution state of the task is any one of executing, non-scheduling and non-executing, determining whether the task concurrency amount of the queue to which the task belongs in the execution node is smaller than the full-load task concurrency amount of the queue to which the task belongs after the superior task is completed.

In the design, the dependency relationship of each task, the scheduling state and the execution state of each task in the task scheduling system are managed by using the task set to be processed, so that the execution node starts to schedule the task after determining that the superior task of the task is executed, and the task can be correctly and orderly executed according to the set dependency relationship; that is, the above design can ensure that the executing node correctly executes each task according to the dependency relationship of each task under limited resources.

In one possible design, the performing the task includes: determining the number of tasks currently executable by the queue to which the task belongs in the execution node according to the task concurrency amount of the queue to which the task belongs in the execution node and the task concurrency amount of the execution node; further, determining the ranking of the priority of the task in the queue to which the task belongs, and if the ranking is within the number of tasks currently executable in the queue to which the task belongs, starting to execute the task.

In the above design, when the priority ranking of the task in the queue is within the number of the tasks currently executable by the queue, the execution node may start executing the task, and otherwise, the execution node does not execute the task; therefore, the execution nodes can be ensured to correctly execute the tasks according to the sequence from high priority to low priority under the limited resources, and the scheduling execution of the high-timeliness tasks is preferentially ensured.

In one possible design, the task scheduling system further includes a management node and a scheduling node; the management node is used for acquiring task configuration information, and the scheduling node is used for creating one or more corresponding tasks according to the task configuration information acquired by the management node, and adding the one or more tasks, the dependency relationship of the one or more tasks, the scheduling condition and the execution state of each task, and the queue identifier to which each task belongs to the task set to be processed; in specific implementation, the monitoring that the task is in a schedulable state includes: and acquiring the scheduling condition of the task from the task set to be processed, and if the scheduling condition of the task is determined to be met, determining that the task is in a schedulable state.

In the design, by setting the management node, the scheduling node and the execution node, a series of processes of generating a task by task configuration information, determining the state of the task by monitoring the trigger information of the task and automatically scheduling the task according to the resource availability condition of the execution node can be automatically realized, so that the automatic process of task processing in the task scheduling system is completed, the process is flexible and reliable, and the task can be ensured to be smoothly and orderly executed.

In one possible design, the method further includes: counting the value of the task under a duration index aiming at any task, calculating the duration fluctuation rate of the task under the duration index according to the value of the task under the duration index and the threshold value of the duration index, and giving an alarm to the task if the duration fluctuation rate of the task under the duration index is greater than the preset duration fluctuation rate corresponding to the duration index; the time index comprises any one or more of time consumption time, overtime time and delay time; or, for each batch of tasks in the task scheduling system, determining an execution completion rate of the tasks at a set time, and if the execution completion rate is less than a preset completion rate, expanding the capacity of the task scheduling system; or determining a task increase rate of each execution node in the task scheduling system relative to a second period according to an increase of an average task execution amount of each execution node in the first period compared with an average task execution amount of each execution node in the second period; the first period is after the second period; if the task growth rate is larger than or equal to the average resource occupancy rate of each execution node, expanding the capacity of the task scheduling system; the average resource occupancy rate of each execution node is the ratio of the average resource amount of each execution node to the total resource amount of the task scheduling system; the capacity expansion mode comprises any one or more of the following modes: the method comprises the steps of increasing the number of execution nodes in the task scheduling system, increasing the task concurrency of any one or more execution nodes, increasing the task concurrency of queues to which tasks belong in any one or more execution nodes, and configuring the core number of Central Processing Units (CPUs) of any one or more execution nodes.

In the design, by monitoring the task execution condition of the task scheduling system, the resource use condition and the task execution condition of the execution node, various abnormalities in the task scheduling system can be found in time, and warning or capacity expansion can be automatically performed based on the various abnormalities, so that tasks, resources and execution nodes in the task scheduling system can be flexibly managed, and the usability of the task scheduling system is improved.

In a second aspect, a task processing device provided in an embodiment of the present invention is an execution node in a task scheduling system, where the execution node is provided with a plurality of queues, and a sum of full-load task concurrency amounts of the plurality of queues is greater than a full-load task concurrency amount of the execution node, and the device includes:

the monitoring module is used for monitoring the scheduling state of each task in the task set to be processed;

an execution module, configured to, after monitoring that the task is in a schedulable state for any task in the to-be-processed task set, if it is determined that a task concurrency amount of a queue to which the task belongs in the execution node is smaller than a full-load task concurrency amount of the queue to which the task belongs, and the task concurrency amount of the execution node is smaller than the full-load task concurrency amount of the execution node, set the state of the task in the to-be-processed task set to be a locked state, and execute the task; the locked state of the task is used to avoid other executing nodes from executing the task.

In one possible design, after monitoring that the task is in the schedulable state, the execution module is further configured to: determining a superior task on which the task depends according to the dependency relationship of a plurality of tasks in the task set to be processed, determining the execution state of the superior task on which the task depends from the task set to be processed, if the execution state of the superior task is the execution completion, determining whether the task concurrency amount of a queue to which the task belongs in the execution node is less than the full-load task concurrency amount of the queue to which the task belongs, and if the execution state of the task is any one of the execution state, the non-scheduling state and the non-execution state, determining whether the task concurrency amount of the queue to which the task belongs in the execution node is less than the full-load task concurrency amount of the queue to which the task belongs after the execution of the superior task is completed.

In one possible design, the execution module is specifically configured to: determining the number of tasks currently executable by the queue to which the task belongs in the execution node according to the task concurrency amount of the queue to which the task belongs in the execution node and the task concurrency amount of the execution node; further, determining the ranking of the priority of the task in the queue to which the task belongs, and if the ranking is within the number of tasks currently executable in the queue to which the task belongs, starting to execute the task.

In one possible design, the task scheduling system further includes a management node and a scheduling node; the management node is used for acquiring task configuration information, and the scheduling node is used for creating one or more corresponding tasks according to the task configuration information acquired by the management node, and adding the one or more tasks, the dependency relationship of the one or more tasks, the scheduling condition and the execution state of each task, and the queue identifier to which each task belongs to the task set to be processed; correspondingly, the monitoring module is specifically configured to: and acquiring the scheduling condition of the task from the task set to be processed, and if the scheduling condition of the task is determined to be met, determining that the task is in a schedulable state.

In one possible design, the apparatus further includes a capacity expansion module, where the capacity expansion module is configured to, for any task, count a value of the task under a duration index, calculate a duration fluctuation rate of the task under the duration index according to the value of the task under the duration index and a threshold of the duration index, and alarm the task if the duration fluctuation rate of the task under the duration index is greater than a preset duration fluctuation rate corresponding to the duration index; the time index comprises any one or more of time consumption time, overtime time and delay time; or, for each batch of tasks in the task scheduling system, determining an execution completion rate of the tasks at a set time, and if the execution completion rate is less than a preset completion rate, expanding the capacity of the task scheduling system; or determining a task increase rate of each execution node in the task scheduling system relative to a second period according to an increase of an average task execution amount of each execution node in the first period compared with an average task execution amount of each execution node in the second period; the first period is after the second period; if the task growth rate is larger than or equal to the average resource occupancy rate of each execution node, expanding the capacity of the task scheduling system; the average resource occupancy rate of each execution node is the ratio of the average resource amount of each execution node to the total resource amount of the task scheduling system; the capacity expansion mode comprises any one or more of the following modes: the method comprises the steps of increasing the number of execution nodes in the task scheduling system, increasing the task concurrency of any one or more execution nodes, increasing the task concurrency of queues to which tasks belong in any one or more execution nodes, and configuring the core number of Central Processing Units (CPUs) of any one or more execution nodes.

In a third aspect, an embodiment of the present invention provides a computing device, including at least one processor and at least one memory, where the memory stores a computer program, and when the program is executed by the processor, the processor is caused to execute the task processing method according to any of the first aspect.

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program executable by a computing device, and when the program runs on the computing device, the computer program causes the computing device to execute the task processing method according to any of the first aspects.

These and other aspects of the invention are apparent from and will be elucidated with reference to the embodiments described hereinafter.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.

Fig. 1 is a schematic structural diagram of a task scheduling system according to an embodiment of the present invention;

fig. 2 is a flowchart illustrating a task processing method according to an embodiment of the present invention;

fig. 3 is a schematic structural diagram of a task processing device according to an embodiment of the present invention;

fig. 4 is a schematic structural diagram of a computing device according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail with reference to the accompanying drawings, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Financial technology (Fintech) refers to a new innovative technology brought to the financial field after information technology is integrated into the financial field, and by using advanced information technology to assist in realizing financial operation, transaction execution and financial system improvement, the processing efficiency and the business scale of the financial system can be improved, and the cost and the financial risk can be reduced.

In the field of financial technology, a plurality of business systems are generally provided, such as a marketing business system, a loan business system, an insurance business system, and the like, and each business system may be responsible for executing a business related to the business system. Taking a marketing service system as an example, generally, the marketing service system can generate thousands or even tens of thousands of services every day, and accordingly, the marketing service system needs to manage a large amount of services, for example, checking the daily account information, checking the user information, and the like. Therefore, each business system may need to process a great number of tasks every day, so that a great number of tasks need to be processed in a batch processing job manner in the field of financial technology.

Fig. 1 is a schematic structural diagram of a task scheduling system according to an embodiment of the present invention, and as shown in fig. 1, a management layer, a scheduling layer, and an execution layer may be sequentially arranged in the task scheduling system, where the management layer is a topmost layer of the task scheduling system, the execution layer is a bottommost layer of the task scheduling system, and the scheduling layer is an intermediate layer arranged between the management layer and the execution layer. Wherein, a plurality of execution nodes, such as execution node 131, execution node 132 and execution node 133, may be disposed in the execution layer, and the plurality of execution nodes may be arranged in a cluster; accordingly, one or more management nodes, such as management node 110, may be disposed in the management layer, and one or more scheduling nodes, such as scheduling node 120, may also be disposed in the scheduling layer.

In the embodiment of the present invention, the execution layer may provide a standard interface to the scheduling layer, so that each execution node in the execution layer may be connected to the scheduling node 120 through the standard interface; the scheduling node 120 may also be connected to the management node 110, for example, through a wired connection, or through a wireless connection, which is not limited in particular.

As shown in fig. 1, in the embodiment of the present invention, the management node 110 may further be connected to one or more clients, such as a client 141, a client 142, and a client 143; the client is a client in a business system, the business system may refer to a business department or a group in the business department, and different business systems may correspond to different businesses. For example, the client 141 may be a client in a loan transaction system, the client 142 may be a client in an insurance transaction system, the client 143 may be a client in a sales transaction system, and so on.

It should be noted that the node in the embodiment of the present invention may refer to a server or a process in the server, for example, the execution node 131 may be an execution server or a process in the server, and is not limited specifically.

Based on the system architecture illustrated in fig. 1, fig. 2 is a flowchart illustrating a task processing method according to an embodiment of the present invention, and as shown in fig. 2, the method includes:

step 201, monitoring the scheduling state of each task in the set of tasks to be processed.

In the embodiment of the present invention, the management node 110 is configured to obtain task configuration information, where the task configuration information refers to configuration information that can be expanded into task instances, for example, when a certain piece of task configuration information is "checking account for account transfer transactions generated in 9:00-10:00 time period of the previous day when 8:00 days, the task configuration information can be expanded into multiple task instances each day. For example, if the transfer transaction 1, the transfer transaction 2 and the transfer transaction 3 are generated in the period of 9:00 to 10:00 of 29 days of 10 and 29 months in 2019, the task configuration information can be expanded into the task instance 1, the task instance 2 and the task instance 3 in 30 days of 10 and 30 months in 2019, the transfer transaction 1 is checked when the task instance 1 is "8: 00 of 30 days of 10 and 30 months in 2019", the transfer transaction 2 is checked when the task instance 2 is "8: 00 of 30 days of 10 and 30 months in 2019", and the transfer transaction 3 is checked when the task instance 3 is "8: 00 of 30 days of 10 and 30 months in 2019".

The manner of acquiring the task configuration information by the management node 110 may be various, for example, the management node 110 may acquire the task configuration information uploaded to the management node 110 by each service system through respective client by performing interactive communication with the client of each service system, or may also set a task configuration interface on the management node 110, so that a developer inputs the task configuration information on the task configuration interface of the management node 110 based on the requirement of each service system, and the like, which is not particularly limited.

For convenience of description, the following embodiments of the present invention simply refer to task instances as tasks.

In this embodiment of the present invention, the scheduling node 120 may obtain task configuration information in the management node 110 according to a preset period or in real time, then create one or more corresponding tasks according to the obtained task configuration information, and add the one or more tasks, the dependency relationship of the one or more tasks, the scheduling condition and the execution state of each task, the queue identifier to which each task belongs, and the like to the set of tasks to be processed. For example, when executing a batch task, the preset period may be set to 1 day, so that the scheduling node 120 may pull the task configuration information corresponding to this day from the management node 110 at a fixed time every day, generate a batch task to be processed today according to the task configuration information corresponding to this day, and generate a task set to be processed today according to a dependency relationship between any two tasks in the batch task to be processed today, an execution state of each task (unlocked, locked, or completed), a scheduling condition of each task, a scheduling state of each task (scheduled or not), and a queue identifier to which each task belongs.

It should be noted that, the above is only an exemplary and simple description, the listed set of tasks to be processed is only for convenience of describing the scheme, and does not constitute a limitation to the scheme, and in a specific implementation, the relevant information of the tasks in the set of tasks to be processed may be configured by a person skilled in the art according to an actual need, for example, the relevant information may also include priorities of the tasks, and is not specifically limited.

In one possible implementation manner, the tasks in the set of tasks to be processed may be divided into script tasks and modular tasks, where the script tasks refer to tasks that are expanded based on the script task configuration information, and the modular tasks refer to tasks that are expanded based on the modular task configuration information. The script tasks can be isolated through task main directories, each task main directory can be used for identifying one service system, if a task set is divided for each product, a task main directory 1 can be set for the service system 141 in the task set, a task main directory 2 is set for the service system 142, and a task main directory 3 is set for the service system 143, the task main directory 1 is used for storing tasks of the service system 141 on the product, the task main directory 2 is used for storing tasks of the service system 142 on the product, and the task main directory 3 is used for storing tasks of the service system 143 on the product; accordingly, the modularized tasks may be isolated via the execution users, each of which is used to identify one execution node, for example, if one task set is divided for each product, an execution user directory 1 may be set for the execution node 131 in the task set, an execution user directory 2 may be set for the execution node 132, an execution user directory 3 may be set for the execution node 133, the user directory 1 is used to store the tasks of the execution node 131 for the product, the user directory 2 is used to store the tasks of the execution node 132 for the product, and the user directory 3 is used to store the tasks of the execution node 133 for the product.

Correspondingly, for any execution node, a plurality of execution groups can be arranged in the execution node, and the execution groups can be development groups, so that the execution node can arrange at least one task main directory queue and at least one execution user queue for each execution group; the task master catalog queue is established based on a task master catalog, tasks in the task master catalog queue are all tasks which are expanded based on script task configuration information, the execution user queue is established based on a user catalog, and the tasks in the execution user queue are all tasks which are expanded based on modular task configuration information; therefore, different execution groups of the execution nodes respectively use the task main directory and the execution users independently through the task main directory queue and the execution user queue, and the execution groups respectively control the execution queues of the execution groups, so that resources of the execution tasks can be divided more finely, the task execution process is divided into different execution queues according to respective requirements, and the execution regularity is improved.

In the embodiment of the present invention, the task configuration information may directly include the identifier of the queue to which the task belongs, so that the scheduling node 120 may directly determine the identifier of the queue to which the task belongs when the task is expanded based on the task configuration information, and thus add the identifier of the queue to which the task belongs to the set of tasks to be processed.

In a specific implementation, for any execution node in the scheduling platform, the execution node may monitor a scheduling state of each task from a set of to-be-processed tasks in the scheduling node, where the scheduling state of each task may be a non-schedulable state and a schedulable state. Specifically, the scheduling node 120 may monitor the scheduling condition of each task in the set of tasks to be processed in real time, and if it is determined that the current time meets the scheduling condition of a certain task, the scheduling state of the task in the set of tasks to be processed may be modified from the non-schedulable state to the schedulable state.

Step 202, after monitoring that a task is in a schedulable state for any task in the set of tasks to be processed, if it is determined that the task concurrency amount of a queue to which the task belongs in the execution node is less than the full-load task concurrency amount of the queue to which the task belongs, and the task concurrency amount of the execution node is less than the full-load task concurrency amount of the execution node, setting the state of the task in the set of tasks to be processed to be a locked state, and executing the task.

In an example, after determining that a certain task in the set of tasks to be processed is in a schedulable state, the execution node may first determine, from the set of tasks to be processed, an upper task on which the task depends, and determine whether the task is executable according to an execution state or a scheduling state of the upper task in the set of tasks to be processed. For example, if the execution status of the upper level task is execution completion, it indicates that the upper level task on which the task depends has been execution completion, and thus the task can be executed; if the execution state of the upper task is the locked state, or the scheduling state of the upper task is the non-scheduling state, or the scheduling state of the upper task is the schedulable state, but the execution state is the unlocked state, it is described that the upper task depending on the task is not executed yet, and therefore the task cannot be executed.

In the embodiment of the invention, the dependency relationship of each task, the scheduling state and the execution state of each task in the task scheduling system are managed by using the task set to be processed, so that the execution node starts to schedule the task after determining that the superior task of the task is executed, and each task can be correctly and orderly executed according to the set dependency relationship; that is, the above design can ensure that the executing node correctly executes each task according to the dependency relationship of each task under limited resources.

In a specific implementation, if it is determined that the upper-level task on which the task depends has been executed, the execution node may determine the number of tasks currently executable by the execution node according to the current task concurrency amount of the execution node and the full-load task concurrency amount of the execution node, and if the current executable task number of the execution node is less than or equal to a first preset threshold (for example, 0), it is indicated that the execution node does not currently have redundant resources to process a new task, so the execution node may not process the task; correspondingly, if the number of tasks currently executable by the execution node is greater than the first preset threshold, it indicates that the execution node currently has redundant resources to process new tasks, in this way, the execution node can obtain the queue identification of the task from the set of tasks to be processed, then, the current task concurrency amount of the queue to which the task in the execution node belongs and the full task concurrency amount of the queue are inquired to determine the current executable task amount of the queue, if the current executable task amount of the queue is less than or equal to a second preset threshold (such as 0), the queue does not have redundant resources to process new tasks currently, therefore, the executing node may not process the task, if the number of tasks currently executable by the queue is greater than the second preset threshold, it means that there are currently excess resources in the queue to process the new task and the executing node can therefore determine to process the task.

For example, queue 1, queue 2, and queue 3 are provided in the execution node, the full-load concurrency of tasks of queue 1 is 30, the full-load concurrency of tasks of queue 2 is 20, the full-load concurrency of tasks of queue 3 is 20, the full-load concurrency of tasks of the execution node is 50, and thus, when a certain task is in a schedulable state and the queue to which the task belongs is queue 2, if the current task concurrency amounts of queue 1, queue 2 and queue 3 are 5, 20 and 5 respectively, the current task concurrency amount of the executing node is 30, although the current amount of task concurrency 30 for the executing node does not reach the full amount of task concurrency 50 for the executing node, however, the current task concurrency 20 of the queue 2 to which the task belongs reaches the full task concurrency 20 of the queue 2, and the queue 2 does not have redundant resources to process the task currently, so that the executing node does not process the task; accordingly, if the current task concurrency amounts of the queue 1, the queue 2 and the queue 3 are respectively 30, 10 and 10, the current task concurrency amount of the execution node is 50, although the current task concurrency amount 10 of the queue 2 to which the task belongs does not reach the full task concurrency amount 20 of the queue 2, the current task concurrency amount 50 of the execution node already reaches the full task concurrency amount 50 of the execution node, and the execution node does not currently have redundant resources to process the task, so the execution node does not process the task.

In the embodiment of the present invention, each execution node may be provided with a plurality of queues, the sum of the task concurrency amounts of the plurality of queues at any time is kept the same as the task concurrency amount of the execution node, and the sum of the full-load task concurrency amounts of the plurality of queues may be greater than or equal to the full-load task concurrency amount of the execution node. However, if the sum of the full-load task concurrency amounts of the plurality of queues is equal to the full-load task concurrency amount of the execution node, each queue can only acquire tasks less than or equal to the full-load task concurrency amount of the queue, and in actual operation, due to non-uniformity of the tasks, the task concurrency amounts of the plurality of queues are not in a full-load state in most cases, so that the task concurrency amount of the execution node in the scheme cannot be in a full-load state in most cases, and resources of the execution node cannot be fully utilized.

Based on this, the embodiment of the invention can set the sum of the full-load task concurrency of the queues to be greater than the full-load task concurrency of the execution node, so that each queue in the execution node can have the capability of acquiring more resources, and even if the task concurrency of the queues is not in a full-load state due to uneven tasks, the task concurrency of the execution node can be in a full-load state; therefore, the method can fully utilize the resources of the execution node and improve the use efficiency of the resources.

It should be noted that the above is only an exemplary and simple description, and the listed query sequence is only for convenience of describing the scheme, and does not constitute a limitation to the scheme, in a specific implementation, the execution node may query the task concurrency amount of the queue to which the task belongs to first determine that the queue has redundant resources to process the task, and then query the task concurrency amount of the execution node to determine whether the execution node has redundant resources to process the task, or may query the task concurrency amount of the queue to which the task belongs and the task concurrency amount of the execution node at the same time, determine whether the queue and the execution node have redundant resources to process the task, if the queue and the execution node both have redundant resources to process the task, the task may be processed, if any one or any one of the queue and the execution node does not have redundant resources to process the task, the task may not be processed.

Further, if the executing node determines to process the task, a first response message may be generated first, and the first response message is sent to the scheduling node 120, and then the task is added to the queue to which the task belongs; accordingly, after receiving the first response message, the scheduling node 120 may modify the state of the task in the pending task set from an unlocked state to a locked state, so as to avoid the other executing nodes from repeatedly executing the task.

In a possible implementation manner, each execution node may be provided with a resource management table, and resources in the execution nodes are managed through the resource management table; the resource management table may include resource information of the executing node and resource information of each queue of the executing node, the resource information of the executing node may include an Internet Protocol address (IP) of the executing node, the number of tasks executed by the executing node, the number of tasks being executed by the executing node, a maximum task concurrency amount of the executing node, a service state of the executing node, and the like, and the resource information of the queue may include an identifier of the queue (such as a task master directory queue or an executing user queue), a maximum task concurrency amount of the queue, a service state of the queue, and the like. The service state of the execution node or the service state of the queue may refer to a heartbeat of the execution node or the queue, and if the execution node or the queue is down (i.e., the heartbeat is 0), the service state of the execution node or the queue may be out of service, and if the execution node or the queue is not down (i.e., the heartbeat is greater than 0), the service state of the execution node or the queue may be in service.

Accordingly, in this implementation manner, when determining that a certain task is in a schedulable state, the execution node may first query the resource management table to determine a service state of the execution node, if the execution node is in an unserviceable state, the execution node may determine not to execute the task, if the execution node is in a serviceable state, the execution node may obtain an identifier of a queue to which the task belongs from a set of tasks to be processed of the scheduling node 120, then determine a service state of the queue to which the task belongs from the query resource management table, if the queue to which the task belongs is in the serviceable state, the execution node may query the resource management table to determine a maximum task concurrency amount of the execution node and a maximum task concurrency amount of the queue to which the task belongs, and determine whether the execution node and the queue to which the task belong have reached the maximum task concurrency amount, if any one has reached the maximum task concurrency amount, it is determined not to process the task, and if neither of them reaches the maximum amount of task concurrency, it may be determined to process the task.

For example, when the maximum task concurrency amount of the execution node is 30, if the execution node has a main directory queue of the first execution group and an execution user queue of the second execution group, the maximum task concurrency amount of the main directory queue of the first execution group is 20, and the maximum task concurrency amount of the execution user queue of the second execution group is 5, the execution node 131 can only process 25 tasks at most at the same time; accordingly, if the main directory queue of the second execution group still exists in the execution node, and the maximum task concurrency amount of the main directory queue of the second execution group is 15, the execution node 131 can only process 30 tasks at most at the same time.

In a possible implementation manner, for any queue in the execution nodes, the execution nodes may determine the number of tasks currently executable by the queue according to the task concurrency amount of the queue in the execution nodes and the task concurrency amount of the execution nodes, and if the number of tasks executable by the queue at the current time is T (an integer greater than 0), the execution nodes may first select all tasks executed and completed by the dependent previous-stage task from all tasks in the queue, then select T tasks with the largest priority from all tasks, and take the T tasks as the currently executable tasks; for example, the execution queue may sort all tasks that have been executed and completed by the dependent task of the previous stage according to the order from large to small or from small to large in priority, and then select T tasks from the side with the largest priority to start execution. Correspondingly, if the total task concurrency of the plurality of queues in the execution node is greater than the task concurrency of the execution node, the execution node may first select all tasks that have been executed and completed by the dependent previous task from all tasks in the plurality of queues, then select T tasks with the highest priority from all tasks, and then execute the T tasks based on the queues to which the T tasks belong.

In the implementation manner, when the priority ranking of the task in the queue is within the number of the tasks currently executable by the queue, the execution node may start to execute the task, and otherwise, the execution node does not execute the task; therefore, the execution nodes can be ensured to correctly execute the tasks according to the sequence from high priority to low priority under the limited resources, and the scheduling execution of the high-timeliness tasks is preferentially ensured.

In the embodiment of the present invention, the task scheduling system may further generate a resource visualization interface according to a condition that each execution node executes a task, and may display the resource visualization interface on the user terminal according to a user requirement, where the user terminal may be any one or more of a computer, a mobile phone, and an IPad, and is not limited specifically.

In a possible scenario, the task scheduling system may generate a batch of tasks to be executed each day according to task configuration information of the current day and data of a historical period (such as a previous day, a previous week, a previous month, and the like), and in this scenario, the task scheduling system may further record resource usage of each execution node each day and a progress of executing the batch of tasks each day, and may generate various types of visual data tables according to the record; therefore, if the visualization instruction sent by the user is received, the task scheduling system can obtain the corresponding visualization data table according to the visualization instruction query and display the visualization data table to the user.

In the implementation of the invention, the visual data table may include any one or more of a resource analysis chart, a task operation time consumption table, a task batch schedule, a task statistical report, a task dependency view and a system task management table.

The resource analysis chart can display the task scheduling system and the resource use condition of each execution node, such as the number of resources used by the task scheduling system to execute the batch tasks per minute and the number of resources used by each execution node per minute; in this way, the user may specify the query date on the resource analysis chart to query the resource usage of each execution node in the query date and the total resource usage of the task scheduling system in the query date, and if the number of resources used by the task scheduling system in the query date is the same as or close to the total resource usage of each execution node in the query date, it indicates that the task scheduling system is always in a full resource state in the query date.

The task operation time consumption table can display the time consumption situation of the batch tasks executed by each execution node every day, such as the time consumption duration of all tasks, the completion time of all tasks, the time consumption duration of each task and the completion time of each task; in this way, the user can specify a certain task and a query time period from the task running time table to check the time consumption duration of all tasks executed in the query time period, the time consumption duration and the completion time of each task executed in the query time period by the task.

The task batch schedule can display the execution condition of the batch tasks executed by the task scheduling system every day, such as the number of the batch tasks to be executed every day, the number of the tasks completed by being executed, the number of the tasks failed to be executed, the number of the tasks overtime in execution, the number of the tasks executed in a delayed way, and the like.

The task statistics report may display the execution status of the tasks in the task scheduling system, such as the number of tasks of the batch that the task scheduling system executes each day, the number of tasks corresponding to each task type, the number of tasks in an unscheduled state, a scheduled state, an unexecuted state, the number of tasks in an executed state, the number of tasks that are alerted, and the like.

The task dependency view may display the dependency relationship of each task in a tree structure, and may display the current running state of each task, for example, a user may obtain, by viewing the task dependency view, an upper task or a lower task on which each task depends, an execution state of the task, an execution state of the upper task on which the task depends, an execution state of the lower task of the task, and the like.

The system task management table may display the task execution status of each execution node, such as the number of tasks completed by each execution node, the number of tasks currently being executed, and the like.

In the embodiment of the invention, various views of the task scheduling system are displayed by using the resource visualization interface, so that a user can conveniently monitor the execution condition of the task scheduling system on the task in real time, the display mode is clear and clear, the watching experience of the user can be improved, the user can conveniently find and remove the fault in time, and the smooth operation of the task scheduling system is ensured.

In the embodiment of the present invention, the task scheduling system may also monitor and maintain the task scheduling system according to the resource usage of executing the batch tasks and the execution result of the batch tasks, and several possible monitoring and maintaining manners are described below:

in an example, the task scheduling system may further count end times of executing the batch tasks each day within a preset time period (e.g., 1 week), and if it is determined that the number of days in which the end time is later than the preset end time exceeds a preset number of days (e.g., 3 days), it may be determined that resources of the task scheduling system are insufficient, and thus, the capacity of the task scheduling system may be expanded; for example, if the historical period is 1 week, the preset end time may be an average end time of executing the batch tasks in the 1 week, or may also be a median of the end times of executing the batch tasks in the 1 week, or may also be a weighted average end time of executing the batch tasks in the 1 week, and a weight of a date in the later period in the 1 week is heavier than a weight of a date in the earlier period, which is not limited specifically.

In another example, for each batch of tasks in the task scheduling system, the task scheduling system may count an execution completion rate of the batch of tasks at a set time, and if the execution completion rate is less than a preset completion rate (for example, 90%), it indicates that a large number of tasks are still incomplete at the set time, so that the task scheduling system may automatically perform capacity expansion.

In yet another example, the task scheduling system may determine a task increase rate of the first period relative to the second period according to an increase amount of an average task execution amount of each execution node in the first period compared to an average task execution amount of each execution node in the second period, and may expand the task scheduling system if the task increase rate is greater than or equal to an average resource occupancy rate of each execution node. The first time period may be any time period after the second time period, and the average resource occupancy of each execution node may be a ratio of the average resource amount of each execution node to the total resource amount of the task scheduling system.

For example, if the first period is the tth day, the determination condition in this example can be expressed by the following formula:

(total(task)_T-AVG(total(task)))/AVG(total(task))>＝m/n

wherein, total (task)_TFor the total number of tasks executed by the task scheduling system on the Tth day, AVG (total (task) is the average number of tasks executed by the task scheduling system on each day in the second time interval (i.e. the ratio of the total number of tasks executed in the second time interval to the number of days in the second time interval), therefore (total (task)_TAVG (total) (task)) refers to the rate of increase of the number of tasks performed by the task scheduling system on day T compared to the average number of tasks per day; correspondingly, m is the average resource amount executed by each executing node in the task scheduling system (i.e. the ratio of the total resource amount of the task scheduling system per day to the number of executing nodes in the task scheduling system), and n is the total resource amount of the task scheduling system, so m/n is the ratio of the average resource amount of each executing node in the task scheduling system to the total resource amount of the task executing system, i.e. the average resource occupancy rate of each executing node.

Therefore, when the increase rate of the number of tasks executed by the task scheduling system on the T-th day compared with the average number of tasks per day is greater than or equal to the average resource occupancy rate of one execution node in the task scheduling system, it indicates that the number of tasks currently required to be executed exceeds the average task execution amount of one execution node, and thus, the task scheduling system can expand the capacity of the task scheduling system according to the number (or multiple) of the tasks exceeding the average task execution amount.

In yet another example, the task scheduling system may also determine, from the daily batch tasks, a resource profile required for the day before executing the daily batch tasks, and if it is determined that the resources of the task scheduling system are insufficient to support the resources required for the day, may calculate additional resources required to support the resources required for the day, and may expand the task scheduling system by an amount of resources greater than or equal to the additional resources. For example, the task scheduling system may compare a batch task of each day with a batch task of the previous day, and when the resource of the task scheduling system is just used to support the resource required for executing the batch task of the previous day, if the batch task of the day is found to be more than the batch task of the previous day, the capacity of the task scheduling system is expanded, and the expanded resource amount may be set according to the number of the excess tasks.

In the embodiment of the present invention, the capacity expansion manner of the task scheduling system may be various, for example, a plurality of idle nodes may be set in the task scheduling system in advance, and when it is determined that the capacity expansion of the task scheduling system is required, a corresponding number of spatial nodes may be activated according to the capacity expansion requirement, so as to increase the number of execution nodes in the task scheduling system; or software improvement or hardware improvement can be performed on any one or any plurality of execution nodes in the task scheduling system, so as to increase the task concurrency of the execution nodes by improving the task execution capacity of the execution nodes, for example, the core number of the Central Processing Unit (CPU) of any one or any plurality of execution nodes can be increased; or the task concurrency of the queues to which the tasks belong in any one or any plurality of execution nodes can be increased, so that the task execution capacity of each queue in the execution nodes is improved, and the task concurrency of each queue is increased.

In yet another example, the task scheduling system may also monitor the running conditions of the batch tasks in real time, and may record tasks that have failed to be executed in the batch tasks, tasks that have not been executed beyond the latest execution time, and tasks that have not been executed beyond the latest completion time, and push these abnormal tasks to the operation and maintenance staff. In specific implementation, the task scheduling system may generate the alarm information corresponding to an abnormal task every time an abnormal task is monitored, or may generate the alarm information according to all abnormal tasks after a batch of tasks in one day is executed, so that the task scheduling system may first send the alarm information to the operation and maintenance server, so that the operation and maintenance server may push the alarm information to the operation and maintenance personnel through any one or any multiple of stapling, mails, short messages, and WeChats.

In another example, if the batch tasks required to be executed by the task scheduling system every day are the same, the task scheduling system may further determine, according to the operation result of the batch tasks every day, 10 tasks that are longest in operation time consumption in the batch tasks every day, then, for each task of the 10 tasks, count a same-ratio increase time of the operation time consumption of the task and the operation time consumption of other tasks in the day and/or a ring-ratio increase time of the operation time consumption of the task and the operation time consumption of the task in the previous day, and if the same-ratio increase time or the ring-ratio increase time is greater than a preset time, determine that the task is an abnormal task; therefore, the task scheduling system can generate the alarm information corresponding to the task and can push the alarm information to operation and maintenance personnel.

In another example, for any task, the execution node may further count values of the task under each duration index, then, for any duration index, the duration fluctuation rate of the task under the duration index may be calculated according to the value of the task under the duration index and the threshold of the duration index, and if the duration fluctuation rate of the task under the duration index is greater than the preset duration fluctuation rate corresponding to the duration index, the capacity of the task scheduling system may be expanded. Each duration index may include any one or more of time-consuming duration, timeout duration and delay duration, and the threshold of each duration index may be any one or more of a historical average value of the duration index, a historical median value of the duration index, and a historical weighted average value of the duration index.

Taking each time index comprising time-consuming time, overtime time and delay time as an example, the specific implementation is total, aiming at each task in daily batch tasks, the task scheduling system can also calculate the time-consuming time from the beginning to the end of the execution of the task, the delay time of the actual starting time of the task relative to the set starting time and the overtime time of the actual ending time of the task relative to the set ending time according to the execution condition of the task, then calculate the fluctuation rate of the time-consuming time based on the time-consuming time and the average time-consuming time, calculate the fluctuation rate of the delay time based on the delay time and the average delay time and calculate the fluctuation rate of the overtime time based on the overtime time and the average time-consuming time; for example, when the time consumed by a task is costtime_iLeveling of tasks over historical periodsWhen the average elapsed time length is avg (costtime), the fluctuation rate of the elapsed time length may be:

(costtime_i-AVG(costtime))/AVG(costtime)

when the delay time of the task is delaytime_iWhen the average delay time of the task in the historical period is avg (delaytime), the fluctuation rate of the delay time may be:

(delaytime_i-AVG(delaytime))/AVG(delaytime)

correspondingly, when the overtime time of the task is overtime_iWhen the average timeout duration of the task in the historical period is avg (overtime), the fluctuation rate of the timeout duration may be:

(overtime_i-AVG(overtime))/AVG(overtime)

thus, if the preset time-consuming duration fluctuation threshold, the preset delay time duration fluctuation threshold and the preset timeout duration fluctuation threshold are all k, if any one or more of the fluctuation rate of the time-consuming duration, the fluctuation rate of the delay time duration and the fluctuation rate of the timeout duration satisfies the following conditions:

(costtime_i-AVG(costtime))/AVG(costtime)>k

(delaytime_i-AVG(delaytime))/AVG(delaytime)>k

(overtime_i-AVG(overtime))/AVG(overtime)>k

the task scheduling system may determine that the task is an abnormal task, and thus, the task scheduling system may generate alarm information corresponding to the task and may push the alarm information to the operation and maintenance staff.

It should be noted that, in the embodiment of the present invention, the preset time-consuming duration fluctuation threshold, the preset time-delaying duration fluctuation threshold, and the preset timeout duration fluctuation threshold may be set by a person skilled in the art according to experience, and the preset time-consuming duration fluctuation threshold, the preset time-delaying duration fluctuation threshold, and the preset timeout duration fluctuation threshold may be the same or different, and are not limited specifically.

In the above example, by determining the fluctuation condition of each task during execution, the task whose fluctuation condition exceeds the threshold value can be detected, and this detection method can detect the task with abnormality more comprehensively and can find the task with the overtime execution time as soon as possible, so that the overtime execution of a certain task can be avoided from hindering the execution of the whole batch of tasks, and the task execution efficiency can be ensured.

It should be noted that, the above are only exemplary and simple descriptions, and do not constitute a limitation to the solution, and in the specific implementation, the manners of alarming and capacity expansion may be set by those skilled in the art according to experience, and are not limited to the above described manners.

In the embodiment of the invention, by monitoring the task execution condition, the resource use condition and the task execution condition of the execution node of the task scheduling system, various abnormalities in the task scheduling system can be found in time, and warning or capacity expansion can be automatically carried out based on the various abnormalities, so that the tasks, resources and the execution node in the task scheduling system can be flexibly managed, and the usability of the task scheduling system is improved.

In the above embodiment of the present invention, any executing node in a task scheduling system monitors a scheduling state of each task in a set of tasks to be processed, and after monitoring that the task is in a schedulable state, if it is determined that task concurrency of a queue to which the task belongs in the executing node is not fully loaded and task concurrency of the executing node is not fully loaded, setting a state of the task in the set of tasks to be processed as a locked state, and executing the task; wherein the locked state of the task is used to avoid other executing nodes from executing the task. In the embodiment of the invention, the execution node in the task scheduling system can monitor the use condition of the resource in real time, and before the schedulable task is executed, the task concurrency quantity of the queue to which the task in the execution node belongs and the task concurrency quantity of the execution node are inquired, and the task is executed again under the condition that the execution node and the execution node are determined to be not fully loaded, so that the task to be processed in the execution node is always in the available range of the resource, the task to be processed in the execution node can be prevented from being blocked, and the task processing efficiency is improved.

In view of the above method flow, an embodiment of the present invention further provides a task processing device, and specific contents of the device may be implemented with reference to the above method.

Fig. 3 is a schematic structural diagram of a task processing device according to an embodiment of the present invention, where the device is any execution node in a task scheduling system, and a plurality of queues are arranged in the execution node, and a sum of full-load task concurrency amounts of the plurality of queues is greater than the full-load task concurrency amount of the execution node, where the device includes:

a monitoring module 301, configured to monitor a scheduling state of each task in a set of tasks to be processed;

an execution module 302, configured to, after monitoring that the task is in a schedulable state for any task in the to-be-processed task set, if it is determined that a task concurrency amount of a queue to which the task belongs in the execution node is smaller than a full-load task concurrency amount of the queue to which the task belongs, and the task concurrency amount of the execution node is smaller than the full-load task concurrency amount of the execution node, set the state of the task in the to-be-processed task set as a locked state, and execute the task; the locked state of the task is used to avoid other executing nodes from executing the task.

Optionally, after monitoring that the task is in the schedulable state, the executing module 302 is further configured to:

determining a superior task on which the task depends according to the dependency relationship of the tasks in the task set to be processed;

determining an execution state of a superior task on which the task depends from the set of tasks to be processed, if the execution state of the superior task is completed, determining whether the task concurrency of a queue to which the task belongs in the execution node is less than the full-load task concurrency of the queue to which the task belongs, and if the execution state of the task is any one of executing, unscheduled and unexecuted, after waiting for the completion of the execution of the superior task, determining whether the task concurrency of the queue to which the task belongs in the execution node is less than the full-load task concurrency of the queue to which the task belongs.

Optionally, the executing module 302 is specifically configured to:

determining the number of tasks currently executable by the queue to which the task belongs in the execution node according to the task concurrency amount of the queue to which the task belongs in the execution node and the task concurrency amount of the execution node;

and determining the ranking of the priority of the task in the queue to which the task belongs, and if the ranking is within the number of the tasks currently executable in the queue to which the task belongs, starting to execute the task.

Optionally, the task scheduling system further includes a management node and a scheduling node; the management node is used for acquiring task configuration information, and the scheduling node is used for creating one or more corresponding tasks according to the task configuration information acquired by the management node, and adding the one or more tasks, the dependency relationship of the one or more tasks, the scheduling condition and the execution state of each task, and the queue identifier to which each task belongs to the task set to be processed;

the monitoring module 301 is specifically configured to:

and acquiring the scheduling condition of the task from the task set to be processed, and if the scheduling condition of the task is determined to be met, determining that the task is in a schedulable state.

Optionally, the apparatus further includes a capacity expansion module 303, where the capacity expansion module 303 is configured to:

counting the value of the task under a duration index aiming at any task, calculating the duration fluctuation rate of the task under the duration index according to the value of the task under the duration index and the threshold value of the duration index, and giving an alarm to the task if the duration fluctuation rate of the task under the duration index is greater than the preset duration fluctuation rate corresponding to the duration index; the time index comprises any one or more of time consumption time, overtime time and delay time; alternatively, the first and second electrodes may be,

aiming at each batch of tasks in the task scheduling system, determining the execution completion rate of the tasks at a set moment, and if the execution completion rate is less than a preset completion rate, expanding the capacity of the task scheduling system; alternatively, the first and second electrodes may be,

determining a task increase rate of each execution node in the task scheduling system relative to a second period according to an increase of an average task execution amount of each execution node in the task scheduling system in the first period compared with an average task execution amount of each execution node in the second period; the first period is after the second period; if the task growth rate is larger than or equal to the average resource occupancy rate of each execution node, expanding the capacity of the task scheduling system; the average resource occupancy rate of each execution node is the ratio of the average resource amount of each execution node to the total resource amount of the task scheduling system;

the capacity expansion mode comprises any one or more of the following modes: the method comprises the steps of increasing the number of execution nodes in the task scheduling system, increasing the task concurrency of any one or more execution nodes, increasing the task concurrency of queues to which tasks belong in any one or more execution nodes, and configuring the core number of Central Processing Units (CPUs) of any one or more execution nodes.

From the above, it can be seen that: in the above embodiment of the present invention, any executing node in a task scheduling system monitors a scheduling state of each task in a set of tasks to be processed, and after monitoring that the task is in a schedulable state, if it is determined that task concurrency of a queue to which the task belongs in the executing node is not fully loaded and task concurrency of the executing node is not fully loaded, setting a state of the task in the set of tasks to be processed as a locked state, and executing the task; wherein the locked state of the task is used to avoid other executing nodes from executing the task. In the embodiment of the invention, the execution node in the task scheduling system can monitor the use condition of the resource in real time, and before the schedulable task is executed, the task concurrency quantity of the queue to which the task in the execution node belongs and the task concurrency quantity of the execution node are inquired, and the task is executed again under the condition that the execution node and the execution node are determined to be not fully loaded, so that the task to be processed in the execution node is always in the available range of the resource, the task to be processed in the execution node can be prevented from being blocked, and the task processing efficiency is improved.

Based on the same inventive concept, an embodiment of the present invention further provides a computing device, as shown in fig. 4, including at least one processor 401 and a memory 402 connected to the at least one processor, where a specific connection medium between the processor 401 and the memory 402 is not limited in the embodiment of the present invention, and the processor 401 and the memory 402 are connected through a bus in fig. 4 as an example. The bus may be divided into an address bus, a data bus, a control bus, etc.

In the embodiment of the present invention, the memory 402 stores instructions executable by the at least one processor 401, and the at least one processor 401 may execute the steps included in the foregoing task processing method by executing the instructions stored in the memory 402.

The processor 401 is a control center of the computing device, and may be connected to various parts of the computing device through various interfaces and lines, and may perform data processing by executing or executing instructions stored in the memory 402 and calling data stored in the memory 402. Optionally, the processor 401 may include one or more processing units, and the processor 401 may integrate an application processor and a modem processor, where the application processor mainly processes an operating system, a user interface, an application program, and the like, and the modem processor mainly processes an issued instruction. It will be appreciated that the modem processor described above may not be integrated into the processor 401. In some embodiments, processor 401 and memory 402 may be implemented on the same chip, or in some embodiments, they may be implemented separately on separate chips.

The processor 401 may be a general-purpose processor, such as a Central Processing Unit (CPU), a digital signal processor, an Application Specific Integrated Circuit (ASIC), a field programmable gate array or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof, configured to implement or perform the methods, steps, and logic blocks disclosed in embodiments of the present invention. A general purpose processor may be a microprocessor or any conventional processor or the like. The steps of a method disclosed in connection with the task processing embodiments may be embodied directly in a hardware processor, or in a combination of the hardware and software modules within the processor.

Memory 402, which is a non-volatile computer-readable storage medium, may be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. The Memory 402 may include at least one type of storage medium, and may include, for example, a flash Memory, a hard disk, a multimedia card, a card-type Memory, a Random Access Memory (RAM), a Static Random Access Memory (SRAM), a Programmable Read Only Memory (PROM), a Read Only Memory (ROM), a charge Erasable Programmable Read Only Memory (EEPROM), a magnetic Memory, a magnetic disk, an optical disk, and so on. The memory 402 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited to such. The memory 402 of embodiments of the present invention may also be circuitry or any other device capable of performing a storage function to store program instructions and/or data.

Based on the same inventive concept, the embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a computing device, and when the program runs on the computing device, the computer program causes the computing device to execute the task processing method according to any one of fig. 2.

It should be apparent to those skilled in the art that embodiments of the present invention may be provided as a method, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.

It will be apparent to those skilled in the art that various changes and modifications may be made in the present invention without departing from the spirit and scope of the invention. Thus, if such modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include such modifications and variations.

Claims

1. A task processing method is applied to any execution node in a task scheduling system, a plurality of queues are arranged in the execution node, and the sum of full-load task concurrency of the plurality of queues is greater than the full-load task concurrency of the execution node, and the method comprises the following steps:

monitoring the scheduling state of each task in the task set to be processed;

aiming at any task in the task set to be processed, after the task is monitored to be in a schedulable state, if the task concurrency amount of a queue to which the task belongs in the execution node is determined to be smaller than the full-load task concurrency amount of the queue to which the task belongs, and the task concurrency amount of the execution node is determined to be smaller than the full-load task concurrency amount of the execution node, setting the state of the task in the task set to be a locked state, and executing the task; the locked state of the task is used to avoid other executing nodes from executing the task.

2. The method of claim 1, wherein after monitoring that the task is in a schedulable state, further comprising:

determining the execution state of a superior task on which the task depends from the task set to be processed;

if the execution state of the superior task is the execution completion, determining whether the task concurrency of the queue to which the task belongs in the execution node is less than the full-load task concurrency of the queue to which the task belongs;

and if the execution state of the task is any one of executing, unscheduled and unexecuted, after the execution of the superior task is finished, determining whether the task concurrency amount of the queue to which the task belongs in the execution node is less than the full-load task concurrency amount of the queue to which the task belongs.

3. The method of claim 1, wherein the performing the task comprises:

4. The method of claim 1, wherein the task scheduling system further comprises a management node and a scheduling node; the management node is used for acquiring task configuration information, and the scheduling node is used for creating one or more corresponding tasks according to the task configuration information acquired by the management node, and adding the one or more tasks, the dependency relationship of the one or more tasks, the scheduling condition and the execution state of each task, and the queue identifier to which each task belongs to the task set to be processed;

the monitoring that the task is in a schedulable state comprises the following steps:

5. The method according to any one of claims 1 to 4, further comprising:

6. A task processing device, which is any execution node in a task scheduling system, wherein a plurality of queues are arranged in the execution node, and the sum of full-load task concurrency of the plurality of queues is greater than the full-load task concurrency of the execution node, the device comprising:

7. The apparatus of claim 6, wherein the execution module is specifically configured to:

8. The apparatus of claim 7, wherein the execution module is specifically configured to:

9. The apparatus of claim 6, wherein the task scheduling system further comprises a management node and a scheduling node; the management node is used for acquiring task configuration information, and the scheduling node is used for creating one or more corresponding tasks according to the task configuration information acquired by the management node, and adding the one or more tasks, the dependency relationship of the one or more tasks, the scheduling condition and the execution state of each task, and the queue identifier to which each task belongs to the task set to be processed;

the monitoring module is specifically configured to:

10. The apparatus of any one of claims 6 to 9, further comprising a capacity expansion module, the capacity expansion module configured to:

11. A computing device comprising at least one processor and at least one memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the method of any of claims 1 to 5.

12. A computer-readable storage medium storing a computer program executable by a computing device, the program, when run on the computing device, causing the computing device to perform the method of any of claims 1 to 5.