CN115509713A

CN115509713A - A task processing method, system device, electronic apparatus, and storage medium

Info

Publication number: CN115509713A
Application number: CN202211221525.9A
Authority: CN
Inventors: 阮彬彬
Original assignee: Hangzhou Hikvision Digital Technology Co Ltd
Current assignee: Hangzhou Hikvision Digital Technology Co Ltd
Priority date: 2022-09-30
Filing date: 2022-09-30
Publication date: 2022-12-23

Abstract

The embodiment of the application provides a task processing method, a task processing system, a task processing device, electronic equipment and a storage medium, and is applied to the technical field of information. The method comprises the steps of obtaining a task to be scheduled initiated by a first user; under the condition that the current queue resources of the first sub-queue are not enough to support the tasks to be scheduled and a second sub-queue meeting preset resource scheduling conditions exists in the first tenant node, selecting queue resources to be scheduled from the queue resources of the second sub-queue; changing the queue resources to be scheduled into the queue resources of the first sub-queue; and adding the task to be scheduled to the first sub-queue. By applying the method, when the current queue resource of the first queue is not enough to support the task to be scheduled, the selected queue resource in the second sub-queue is changed into the queue resource of the first sub-queue for completing the scheduling of the task to be scheduled, so that the utilization rate of the queue resource can be improved, and the rationality of the utilization of the queue resource is increased.

Description

Task processing method, system, device, electronic equipment and storage medium

Technical Field

The present application relates to the field of information technologies, and in particular, to a method, a system, an apparatus, an electronic device, and a storage medium for processing a task.

Background

With the continuous development of big data, more and more tasks can be performed by means of electronic devices. In the related art, when a task queue is established, a corresponding number of queue resources are selected to be allocated to the task queue according to the importance degree of the task queue. During the actual processing of tasks, the number of tasks that can be processed by the task queue depends on the queue resources allocated during the queue creation. However, the above method may cause unreasonable utilization of queue resources.

Disclosure of Invention

An object of the embodiments of the present application is to provide a task processing method, system, apparatus, electronic device, and storage medium, so as to solve the problem that queue resources cannot be reasonably utilized. The specific technical scheme is as follows:

in a first aspect of the embodiments of the present application, a task processing method is provided, which is applied to a task processing system, where the task processing system is provided with a plurality of tenant nodes, each of the tenant nodes corresponds to a corresponding number of queue resources, the tenant nodes include a plurality of sub-queues, and the sub-queues correspond to users one to one, and the method includes:

acquiring a task to be scheduled, which is initiated by a first user, wherein the first user corresponds to a first sub-queue, and the first sub-queue belongs to a first tenant node;

under the condition that the current queue resources of the first sub-queue are not enough to support the tasks to be scheduled and a second sub-queue meeting preset resource scheduling conditions exists in the first tenant node, selecting queue resources to be scheduled from the queue resources of the second sub-queue;

changing the queue resource to be scheduled into the queue resource of the first sub-queue;

and adding the task to be scheduled to the first sub-queue.

In a possible implementation manner, the current queue resource of the first sub-queue does not exceed the resource upper limit of the first sub-queue; for each sub-queue except the first sub-queue in the first tenant node, under the condition that the priority of the sub-queue is lower than that of the first sub-queue and the current queue resource of the sub-queue is greater than the lower resource limit of the sub-queue, judging that the sub-queue is a second sub-queue meeting a preset resource scheduling condition;

selecting queue resources to be scheduled from the queue resources of the second sub-queue, including:

selecting a specified number of queue resources to be scheduled from the queue resources of the second sub-queue, wherein the specified number satisfies that the queue resources of the second sub-queue are not less than the resource lower limit of the second sub-queue after the queue resources to be scheduled are scheduled out of the second sub-queue; and the specified quantity meets the condition that the queue resources of the first sub-queue do not exceed the resource upper limit of the first sub-queue after the queue resources to be scheduled are scheduled to be the queue resources of the first sub-queue.

In a possible implementation manner, the number of the tasks to be scheduled is multiple, and the adding the tasks to be scheduled to the first sub-queue includes:

and sequentially adding the tasks to be scheduled to the first sub-queue according to the sequence of the priorities of the tasks to be scheduled from high to low until the tasks to be scheduled are added completely or the queue resources of the first sub-queue are insufficient to continue adding the tasks to be scheduled.

In one possible embodiment, the method further comprises:

and under the condition that the queue resources of the first sub-queue are not enough to continue to add the task to be scheduled, storing the task to be scheduled which is not added into a task waiting storage area.

In one possible embodiment, the method further comprises:

and under the condition that the queue resources of the first sub-queue are not enough to load all the tasks to be scheduled and the priority of all the tasks to be scheduled is higher than the priority of the end-of-queue task of the first sub-queue, transferring the end-of-queue task of the first sub-queue to a task waiting storage area until the queue resources of the first sub-queue are enough to load all the tasks to be scheduled or the tasks to be scheduled with the priority lower than the priority of the end-of-queue task exist.

In one possible embodiment, the method further comprises:

when the queue resources of the first sub-queue which are free reach a first preset free resource threshold value, sequentially selecting tasks to be added into the first sub-queue according to the sequence of the priority from high to low in each task of the first sub-queue in the task waiting storage area until the tasks of the first sub-queue do not exist in the task waiting storage area or the queue resources of the first sub-queue which are free are smaller than a second preset free resource threshold value, wherein the first preset free resource threshold value is larger than the second preset free resource threshold value.

In one possible embodiment, the method further comprises:

acquiring a tenant node creating instruction, wherein the tenant node creating instruction comprises the total queue resource amount of the tenant node;

and establishing a corresponding tenant node on the basis of a preset root node, and distributing queue resources of the total queue resource amount for the established tenant node, wherein the total amount of the queue resources of all the sub queues in the tenant node does not exceed the total queue resource amount of the tenant node for any tenant node.

In one possible embodiment, the method further comprises:

acquiring a sub-queue creating request aiming at a second user, wherein the sub-queue creating request of the second user comprises a tenant node to which the second user belongs, a queue resource upper limit and a queue resource lower limit of a sub-queue of the second user, and the priority of the sub-queue of the second user;

and according to the sub-queue creation request of the second user, creating a sub-queue with corresponding priority on the basis of the tenant node to which the second user belongs, and allocating queue resources for the created sub-queue, wherein the allocated queue resources do not exceed the upper limit of the queue resources of the sub-queue of the second user and are not lower than the lower limit of the queue resources of the sub-queue of the second user.

In a possible implementation manner, a Yarn (Resource manager) component and a K8S (container orchestrator) component are provided in the task processing system, and creation, deletion, and queue Resource partitioning of the tenant node and the sub-queue are implemented by the Yarn component or the K8S component.

In a second aspect of the embodiments of the present application, a task processing system is provided, where the task processing system is provided with a plurality of tenant nodes, each of the tenant nodes corresponds to a corresponding number of queue resources, each of the tenant nodes includes a plurality of sub-queues, and the sub-queues correspond to users one to one, and the task processing system includes: a resource management component, the resource management component comprising:

the task running scheduling module is used for acquiring a task to be scheduled, which is initiated by a first user, wherein the first user corresponds to a first sub-queue, and the first sub-queue belongs to a first tenant node;

the queue operation service module is used for selecting queue resources to be scheduled from the queue resources of a second sub-queue under the condition that the current queue resources of the first sub-queue are not enough to support the tasks to be scheduled and the second sub-queue meeting preset resource scheduling conditions exists in the first tenant node; changing the queue resource to be scheduled into the queue resource of the first sub-queue;

and the task operation scheduling module is also used for adding the task to be scheduled to the first sub-queue.

In a possible implementation manner, the current queue resource of the first sub-queue does not exceed the resource upper limit of the first sub-queue;

the queue operation service module is specifically configured to, for each sub-queue in the first tenant node except the first sub-queue, determine that the sub-queue is a second sub-queue that meets a preset resource scheduling condition when the priority of the sub-queue is lower than the priority of the first sub-queue and a current queue resource of the sub-queue is greater than a resource lower limit of the sub-queue; selecting a specified number of queue resources to be scheduled from the queue resources of the second sub-queue, wherein the specified number satisfies that the queue resources of the second sub-queue are not less than the resource lower limit of the second sub-queue after the queue resources to be scheduled are scheduled out of the second sub-queue; and the specified quantity meets the condition that the queue resources of the first sub-queue do not exceed the resource upper limit of the first sub-queue after the queue resources to be scheduled are scheduled to be the queue resources of the first sub-queue.

In a possible implementation manner, the number of the tasks to be scheduled is multiple, and the task operation scheduling module is specifically configured to: and sequentially adding the tasks to be scheduled to the first sub-queue according to the sequence of the priorities of the tasks to be scheduled from high to low until the tasks to be scheduled are added completely or the queue resources of the first sub-queue are insufficient to continue adding the tasks to be scheduled.

In a possible implementation manner, the task operation scheduling module is further configured to: and under the condition that the queue resources of the first sub-queue are not enough to continue to add the task to be scheduled, storing the task to be scheduled which is not added into a task waiting storage area.

In a possible implementation manner, the task operation scheduling module is further configured to: and under the condition that the queue resources of the first sub-queue are not enough to load all the tasks to be scheduled and the priority of all the tasks to be scheduled is higher than the priority of the end-of-queue task of the first sub-queue, transferring the end-of-queue task of the first sub-queue to a task waiting storage area until the queue resources of the first sub-queue are enough to load all the tasks to be scheduled or the tasks to be scheduled with the priority lower than the priority of the end-of-queue task exist.

In a possible implementation manner, the task operation scheduling module is further configured to: when the queue resources of the first sub-queue which are free reach a first preset free resource threshold value, sequentially selecting tasks to be added into the first sub-queue according to the sequence of the priority from high to low in each task of the first sub-queue in the task waiting storage area until the tasks of the first sub-queue do not exist in the task waiting storage area or the queue resources of the first sub-queue which are free are smaller than a second preset free resource threshold value, wherein the first preset free resource threshold value is larger than the second preset free resource threshold value.

In a possible implementation manner, the queue operation service module is further configured to: acquiring a tenant node creating instruction, wherein the tenant node creating instruction comprises the total queue resource amount of the tenant node; and establishing a corresponding tenant node on the basis of a preset root node, and distributing queue resources of the total queue resource amount for the established tenant node, wherein the total amount of the queue resources of all the sub queues in the tenant node does not exceed the total queue resource amount of the tenant node for any tenant node.

In a possible implementation manner, the queue operation service module is further configured to: acquiring a sub-queue creating request aiming at a second user, wherein the sub-queue creating request of the second user comprises a tenant node to which the second user belongs, a queue resource upper limit and a queue resource lower limit of a sub-queue of the second user, and the priority of the sub-queue of the second user; and according to the sub-queue creation request of the second user, creating a sub-queue with corresponding priority on the basis of the tenant node to which the second user belongs, and allocating queue resources for the created sub-queue, wherein the allocated queue resources do not exceed the upper limit of the queue resources of the sub-queue of the second user and are not lower than the lower limit of the queue resources of the sub-queue of the second user.

In one possible implementation, the task processing system further includes: a Yarn module and a K8S module; and the queue operation service module realizes the creation, deletion and queue resource division of the tenant nodes and the sub-queues through the Yarn component or the K8S component.

In one possible embodiment, the resource management component further comprises:

the queue resource information management service module is used for displaying the current information view of each sub-queue, counting the queue resources of each sub-queue and triggering the updating of the queue resources when the tasks in the sub-queues are completed;

and the tenant user queue relation service module is used for maintaining the corresponding relation between the tenant node and the sub-queue.

A third aspect of the embodiments of the present application provides a task processing device, which is applied to a task processing system, where the task processing system is provided with a plurality of tenant nodes, each of the tenant nodes corresponds to a corresponding number of queue resources, each of the tenant nodes includes a plurality of sub-queues, and the sub-queues correspond to users one to one, and the device includes:

the scheduling method comprises the steps that a task to be scheduled acquisition module is used for acquiring a task to be scheduled initiated by a first user, wherein the first user corresponds to a first sub-queue, and the first sub-queue belongs to a first tenant node;

a queue resource selecting module, configured to select a queue resource to be scheduled from queue resources of a second sub-queue when a current queue resource of the first sub-queue is insufficient to support the task to be scheduled and the first tenant node has the second sub-queue meeting a preset resource scheduling condition;

a queue resource changing module, configured to change the queue resource to be scheduled to a queue resource of the first sub-queue;

and the task to be scheduled adding module is used for adding the task to be scheduled to the first sub-queue.

In a possible implementation manner, the current queue resource of the first sub-queue does not exceed the resource upper limit of the first sub-queue; for each sub-queue except the first sub-queue in the first tenant node, under the condition that the priority of the sub-queue is lower than the priority of the first sub-queue and the current queue resource of the sub-queue is larger than the resource lower limit of the sub-queue, the sub-queue is judged to be a second sub-queue meeting a preset resource scheduling condition;

the queue resource selecting module is specifically configured to select a specified number of queue resources to be scheduled from the queue resources of the second sub-queue, where the specified number satisfies that the queue resources of the second sub-queue are not less than a resource lower limit of the second sub-queue after the queue resources to be scheduled are scheduled by the second sub-queue; and the specified quantity meets the condition that the queue resources of the first sub-queue do not exceed the resource upper limit of the first sub-queue after the queue resources to be scheduled are scheduled to be the queue resources of the first sub-queue.

In a possible implementation manner, the number of the tasks to be scheduled is multiple, and the task to be scheduled adding module is specifically configured to sequentially add the tasks to be scheduled to the first sub-queue according to a sequence from high priority to low priority of the tasks to be scheduled, until the addition of the tasks to be scheduled is completed or queue resources of the first sub-queue are insufficient to continue adding the tasks to be scheduled.

In a possible embodiment, the apparatus further comprises:

and the task to be scheduled storage module is used for storing the task to be scheduled which is not added into a task waiting storage area under the condition that the queue resources of the first sub-queue are not enough to continue adding the task to be scheduled.

In a possible embodiment, the apparatus further comprises:

and the queue tail task unloading module is used for unloading the queue tail tasks of the first sub-queue into a task waiting storage area under the condition that the queue resources of the first sub-queue are not enough to load the tasks to be scheduled and the priority of the tasks to be scheduled is higher than that of the queue tail tasks of the first sub-queue until the queue resources of the first sub-queue are enough to load the tasks to be scheduled or the tasks to be scheduled with the priority lower than that of the queue tail tasks exist.

In a possible embodiment, the apparatus further comprises:

and the preset idle resource threshold setting module is used for sequentially selecting tasks to be added into the first sub-queue in the sequence from high priority to low priority in each task of the first sub-queue in the task waiting storage area when the idle queue resources of the first sub-queue reach a first preset idle resource threshold, until the tasks of the first sub-queue do not exist in the task waiting storage area or the idle queue resources of the first sub-queue are smaller than a second preset idle resource threshold, wherein the first preset idle resource threshold is larger than the second preset idle resource threshold.

In a possible embodiment, the apparatus further comprises:

the system comprises a creation instruction acquisition module, a queue management module and a queue management module, wherein the creation instruction acquisition module is used for acquiring a tenant node creation instruction, and the tenant node creation instruction comprises the total queue resource amount of the tenant node;

and the tenant node creating module is used for creating corresponding tenant nodes on the basis of a preset root node and distributing queue resources of the total queue resource amount for the created tenant nodes, wherein the total amount of the queue resources of all the sub-queues in any tenant node is not more than the total queue resource amount of the tenant node.

In a possible embodiment, the apparatus further comprises:

a second user queue creation request obtaining module, configured to obtain a sub-queue creation request for a second user, where the sub-queue creation request for the second user includes a tenant node to which the second user belongs, a queue resource upper limit and a queue resource lower limit of the sub-queue of the second user, and a priority of the sub-queue of the second user;

and the second user queue creating module is used for creating a sub-queue with corresponding priority on the basis of the tenant node to which the second user belongs according to the sub-queue creating request of the second user, and allocating queue resources for the created sub-queue, wherein the allocated queue resources do not exceed the upper limit of the queue resources of the sub-queue of the second user and are not lower than the lower limit of the queue resources of the sub-queue of the second user.

In a possible implementation manner, a yann component and a K8S component are arranged in the task processing system, and creation, deletion, and queue resource division of the tenant node and the sub-queue are realized by the yann component or the K8S component.

In a fourth aspect of the embodiments of the present application, there is also provided an electronic device, including:

a memory for storing a computer program;

a processor configured to implement the method according to any one of the first aspect of the embodiments of the present application when executing a program stored in a memory.

In a fifth aspect of the embodiments of the present application, a computer-readable storage medium is further provided, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements any one of the task processing methods described above.

The embodiments of the present application also provide a computer program product containing instructions, which when run on a computer, causes the computer to execute any one of the above task processing methods.

The embodiment of the application has the following beneficial effects:

according to the task processing method, the task processing system, the task processing device, the electronic equipment and the storage medium, a task to be scheduled, which is initiated by a first user, is obtained, wherein the first user corresponds to a first sub-queue, and the first sub-queue belongs to a first tenant node; under the condition that the current queue resources of the first sub-queue are not enough to support the tasks to be scheduled and a second sub-queue meeting preset resource scheduling conditions exists in the first tenant node, selecting queue resources to be scheduled from the queue resources of the second sub-queue; changing the queue resource to be scheduled into the queue resource of the first sub-queue; adding the task to be scheduled to in the first sub-queue. When the current queue resource of the first queue is not enough to support the task to be scheduled, the selected queue resource in the second sub-queue is changed into the queue resource of the first sub-queue for completing the scheduling of the task to be scheduled, so that the utilization rate of the queue resource can be improved, and the rationality of the utilization of the queue resource is increased.

Of course, not all advantages described above need to be achieved at the same time in the practice of any one product or method of the present application.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and it is also obvious for a person skilled in the art to obtain other embodiments according to the drawings.

Fig. 1 is a flowchart of a task processing method according to an embodiment of the present application.

Fig. 2 is a schematic structural diagram of a task processing system according to an embodiment of the present application.

Fig. 3 is a flowchart of a queue operation according to an embodiment of the present disclosure.

Fig. 4 is a schematic operation flow diagram of a resource management center according to an embodiment of the present disclosure.

Fig. 5 is a schematic diagram of a current information view of a queue according to an embodiment of the present application.

Fig. 6 is a schematic diagram of a task scheduling process according to an embodiment of the present application.

Fig. 7 is a schematic structural diagram of a task processing device according to an embodiment of the present application.

Fig. 8 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application are within the scope of protection of the present application.

In a first aspect of the present application, a task processing method is provided, as shown in fig. 1:

step S101: and acquiring a task to be scheduled initiated by a first user.

The first user corresponds to a first sub-queue, and the first sub-queue belongs to a first tenant node. In practical Application, the first user refers to an API (Application Program Interface) consuming entity with authentication information, the tenant refers to a resource set that can be accessed by each service, the first sub-queue is created by a parent queue of the hierarchical queue, and the parent queue and the plurality of sub-queues together form the hierarchical queue.

The method of the embodiment of the application can be applied to a task processing system, wherein the task processing system is provided with a plurality of tenant nodes, each tenant node corresponds to a corresponding amount of queue resources, each tenant node comprises a plurality of sub-queues, the sub-queues correspond to users one by one, one tenant corresponds to one father user, and one user can only bind one tenant.

The method of the embodiment of the application can be realized through terminal equipment, and specifically, the terminal equipment can be intelligent equipment such as a server, a computer, a mobile phone and the like.

Step S102: and under the conditions that the current queue resources of the first sub-queue are not enough to support the tasks to be scheduled and a second sub-queue meeting the preset resource scheduling conditions exists in the first tenant node, selecting the queue resources to be scheduled from the queue resources of the second sub-queue.

In practical application, each queue resource has a corresponding upper resource limit and a corresponding lower resource limit, wherein the resource of the child node can allocate the total resource of the parent node according to a certain proportion or be smaller than the total resource of the parent node.

In a possible implementation manner, the current queue resource of the first sub-queue does not exceed the resource upper limit of the first sub-queue; for each sub-queue except the first sub-queue in the first tenant node, under the condition that the priority of the sub-queue is lower than that of the first sub-queue and the current queue resource of the sub-queue is greater than the lower resource limit of the sub-queue, judging that the sub-queue is a second sub-queue meeting the preset resource scheduling condition;

selecting queue resources to be scheduled from the queue resources of the second sub-queue may include:

selecting a specified quantity of queue resources to be scheduled from the queue resources of the second sub-queue, wherein the specified quantity meets the condition that the queue resources of the second sub-queue are not less than the resource lower limit of the second sub-queue after the queue resources to be scheduled are scheduled by the second sub-queue; and the specified quantity meets the condition that the queue resources of the first sub-queue do not exceed the resource upper limit of the first sub-queue after the queue resources to be scheduled are scheduled to be the queue resources of the first sub-queue.

Specifically, the specified number may be determined according to the task to be scheduled. In practical application, when the task to be scheduled is finished running, the specified number of queue resources to be scheduled selected from the queue resources of the second sub-queue can be released and returned to the second sub-queue.

By applying the method of the embodiment, the resources of the second sub-queue can be preempted when the current queue resources of the first sub-queue are not enough to support the tasks to be scheduled, so that the tasks to be scheduled can be successfully operated and the resources in the second sub-queue can be reasonably utilized, the tasks with high priority can be operated first, and the smooth operation of the tasks and the reasonable utilization of the resources can be finally realized.

Step S103: and changing the queue resources to be scheduled into the queue resources of the first sub-queue.

Step S104: and adding the task to be scheduled to the first sub-queue.

In one possible implementation, the number of the tasks to be scheduled is multiple, and adding the tasks to be scheduled to the first sub-queue may include:

By applying the method of the embodiment, the task to be scheduled can be added into the first sub-queue from high to low according to the priority, so that the task to be executed can run according to the priority, and finally the task runs more smoothly.

By applying the method of the embodiment, when the current queue resource of the first queue is not enough to support the task to be scheduled, the selected queue resource in the second sub-queue is changed into the queue resource of the first sub-queue for completing the scheduling of the task to be scheduled, so that the utilization rate of the queue resource can be improved, and the rationality of the utilization of the queue resource can be increased.

In a possible implementation manner, the task processing method of this embodiment may further include:

and under the condition that the queue resources of the first sub-queue are not enough to continue adding the tasks to be scheduled, storing the tasks to be scheduled which are not added into the task waiting storage area.

In practical application, after the task to be scheduled that is not added is stored in the task waiting storage area, when the queue resources of the first sub-queue are enough to support the running of the task to be scheduled that is not added, the task to be scheduled that is not added can be read from the task waiting storage area and added into the first word queue.

By applying the method of the embodiment of the application, the task to be scheduled which is not added can be stored in the task waiting storage area under the condition that the queue resources of the first sub-queue are not enough to continue adding the task to be scheduled, so that the task to be scheduled which is not added can be rapidly added into the first sub-queue after the running condition is met, and the task processing efficiency is improved.

In a possible implementation manner, before the step of adding the task to be scheduled to the first sub-queue, the method may further include:

and under the condition that the queue resources of the first sub-queue are not enough to load all the tasks to be scheduled and the priority of all the tasks to be scheduled is higher than that of the tail task of the first sub-queue, the tail task of the first sub-queue is transferred to a task waiting storage area until the queue resources of the first sub-queue are enough to load all the tasks to be scheduled or the tasks to be scheduled with the priority lower than that of the tail task exist.

The queue tasks in the first sub-queue can be arranged according to the priority, and the high-priority tasks are preferentially executed, and when the priority of the task to be scheduled which is not added is higher than that of the task at the tail of the first sub-queue, the task at the tail of the queue can be placed in the task waiting storage area, and the task to be scheduled with the high priority can be preferentially executed. In practical application, the tasks with lower priority and short running time can also be dumped into the task waiting storage area.

By applying the method of the embodiment of the application, the task to be scheduled with higher priority is executed by judging the priority of the task to be scheduled and the task at the tail of the first sub-queue, so that the processing time of the task can be shortened, and the efficiency can be improved.

In a possible embodiment, the method may further include:

when the queue resources of the first sub-queue which are free reach a first preset free resource threshold value, sequentially selecting tasks to be added into the first sub-queue in each task of the first sub-queue of the task waiting storage area according to the sequence from high priority to low priority until no task of the first sub-queue exists in the task waiting storage area or the queue resources of the first sub-queue which are free are smaller than a second preset free resource threshold value, wherein the first preset free resource threshold value is larger than the second preset free resource threshold value.

The first preset idle resource threshold may be set according to each task to be scheduled in the task waiting storage area, so as to ensure that when the queue resource of the first sub-queue that is idle reaches the first preset idle resource threshold, the task to be scheduled in the first sub-queue in the task waiting storage area can run. The second preset free resource threshold may be a minimum resource that ensures that each task in the task waiting area can run, and when the queue resource that is free in the first sub-queue is less than the second preset free resource threshold, the task in the task waiting storage area is stopped being added to the first sub-queue.

By applying the method of the embodiment of the application, whether the queue resources idle in the first sub-queue can meet the resources required by the running of each task in the task waiting storage area can be judged by setting the first preset idle resource threshold and the second preset idle resource threshold, so that the tasks to be scheduled in the task waiting storage area can run in time, and the task processing efficiency is improved.

In a possible embodiment, the method may further include:

executing the head task of the first sub-queue;

under the condition that the execution of the queue head task fails, judging whether the execution times of the queue head task reach a preset time threshold value, if not, re-executing the queue head task, and if so, clearing the queue head task in the first sub-queue;

and clearing the head-of-line task in the first sub-queue under the condition that the head-of-line task is successfully executed.

The preset time threshold may be set according to the running speed of the server or information of the task to be scheduled. In practical application, when other tasks in the queue fail to execute, the execution may also be determined according to the preset number threshold, and when the threshold is reached, the task is cleared if the execution still fails.

By applying the method of the embodiment of the application, the task can be cleared when the task fails to be executed, so that the task processing interruption caused by the execution failure of the queue task is reduced, and the task processing efficiency is improved.

In a possible embodiment, the method may further include:

and establishing a corresponding tenant node on the basis of a preset root node, and distributing queue resources of the total queue resource amount to the established tenant node.

And aiming at any tenant node, the total amount of queue resources of all the sub-queues in the tenant node does not exceed the total amount of queue resources of the tenant node. Queue resources of the total queue resource amount allocated by the tenant node can be allocated according to a certain proportion, and resources smaller than the total queue resource amount can also be allocated according to the information of the queue task.

By applying the method of the embodiment of the application, the total resource amount of the tenant node can be ensured to support the running of the tenant queue by performing resource allocation on the tenant node, so that the resource isolation among tenants is realized, and each user queue in the tenant performs resource preemption.

In a possible embodiment, the method may further include:

acquiring a sub-queue creating request aiming at a second user, wherein the sub-queue creating request of the second user comprises a tenant node to which the second user belongs, a queue resource upper limit and a queue resource lower limit of the sub-queue of the second user, and the priority of the sub-queue of the second user;

In practical application, the method can be suitable for multiple users to perform queue operation, namely, a tenant node corresponding to a second user is created, so that resource isolation among tenants is realized, and the queue is suitable for an application scene of multiple users and multiple tenants.

In a possible implementation manner, the task processing system is provided with a Yarn component and a K8S component, and creation, deletion, and queue resource division of tenant nodes and sub-queues can be realized by the Yarn component or the K8S component.

In a second aspect of the embodiments of the present application, a task processing system is provided, where the task processing system is provided with a plurality of tenant nodes, each tenant node corresponds to a corresponding number of queue resources, each tenant node includes a plurality of sub-queues, the sub-queues correspond to users one to one, and the task processing system includes: a resource management component, the resource management component comprising as shown in FIG. 2:

the task running scheduling module 201 is configured to obtain a task to be scheduled, which is initiated by a first user, where the first user corresponds to a first sub-queue, and the first sub-queue belongs to a first tenant node;

the queue operation service module 202 is configured to select a queue resource to be scheduled from queue resources of a second sub-queue when a current queue resource of the first sub-queue is insufficient to support a task to be scheduled and the first tenant node has the second sub-queue meeting a preset resource scheduling condition; changing queue resources to be scheduled into queue resources of a first sub-queue;

the task operation scheduling module 201 is further configured to add the task to be scheduled to the first sub-queue.

By applying the task processing system, when the current queue resource of the first queue is not enough to support the task to be scheduled, the selected queue resource in the second sub-queue is changed into the queue resource of the first sub-queue for completing the scheduling of the task to be scheduled, so that the utilization rate of the queue resource can be improved, and the rationality of queue resource utilization is improved.

the queue operation service module is specifically used for judging that each sub-queue except the first sub-queue in the first tenant node is a second sub-queue meeting preset resource scheduling conditions under the condition that the priority of the sub-queue is lower than that of the first sub-queue and the current queue resource of the sub-queue is greater than the lower limit of the resource of the sub-queue; selecting a specified number of queue resources to be scheduled from the queue resources of the second sub-queue, wherein the specified number satisfies that the queue resources of the second sub-queue are not less than the resource lower limit of the second sub-queue after the queue resources to be scheduled are scheduled by the second sub-queue; and the specified quantity meets the condition that the queue resources of the first sub-queue do not exceed the resource upper limit of the first sub-queue after the queue resources to be scheduled are scheduled to be the queue resources of the first sub-queue.

In one possible implementation, the task execution scheduling module is further configured to: and under the condition that the queue resources of the first sub-queue are insufficient to continue adding the tasks to be scheduled, storing the tasks to be scheduled which are not added into the task waiting storage area.

In one possible implementation, the task execution scheduling module is further configured to: and under the condition that the queue resources of the first sub-queue are not enough to load all the tasks to be scheduled and the priority of all the tasks to be scheduled is higher than that of the tail task of the first sub-queue, the tail task of the first sub-queue is transferred to a task waiting storage area until the queue resources of the first sub-queue are enough to load all the tasks to be scheduled or the tasks to be scheduled with the priority lower than that of the tail task exist.

In one possible implementation, the task execution scheduling module is further configured to: when the queue resource of the first sub-queue reaches a first preset idle resource threshold value, selecting tasks in sequence from high to low in priority from each task of the first sub-queue of the task waiting storage area to add to the first sub-queue, and until the task of the first sub-queue does not exist in the task waiting storage area or the idle queue resource of the first sub-queue is smaller than a second preset idle resource threshold value, wherein the first preset idle resource threshold value is larger than the second preset idle resource threshold value.

In one possible implementation, the queue operation service module is further configured to: acquiring a tenant node creating instruction, wherein the tenant node creating instruction comprises the total queue resource amount of the tenant node; and establishing a corresponding tenant node on the basis of a preset root node, and distributing queue resources of the total queue resource amount for the established tenant node, wherein the total amount of the queue resources of all the sub-queues in the tenant node does not exceed the total queue resource amount of the tenant node for any tenant node.

In one possible implementation, the queue operation service module is further configured to: acquiring a sub-queue creating request aiming at a second user, wherein the sub-queue creating request of the second user comprises a tenant node to which the second user belongs, a queue resource upper limit and a queue resource lower limit of the sub-queue of the second user, and the priority of the sub-queue of the second user; and according to the sub-queue creation request of the second user, creating a sub-queue with corresponding priority on the basis of the tenant node to which the second user belongs, and allocating queue resources for the created sub-queue, wherein the allocated queue resources do not exceed the upper limit of the queue resources of the sub-queue of the second user and are not lower than the lower limit of the queue resources of the sub-queue of the second user.

In one possible implementation, the task processing system further includes: a Yarn module and a K8S module; and the queue operation service module realizes creation, deletion and queue resource division of tenant nodes and sub-queues through the Yarn component or the K8S component.

By applying the task processing system of the embodiment of the application, the priority of the tasks can be judged globally, and the tasks which are executed preferentially can be determined according to the resource amount of the queue, so that the queue resources can be reasonably utilized when the queue tasks are ensured to run.

The following example is provided to specifically describe the task processing method and task processing system of the present application,

firstly, a hierarchical queue is created by using K8S;

as shown in fig. 3, a hierarchical queue takes Root in the graph as a Root node, and a user node Resource acquires total resources of tenant nodes according to a certain proportion, for example, tenant 1 and tenant 2 nodes acquire 30% and 70% of resources of the Root node respectively, and then the user node acquires resources of respective corresponding tenant nodes, for example, user 1-1 node acquires 40% of resources of corresponding tenant 1 node, user 1-2 node acquires 60% of resources of corresponding tenant 1 node, user 2-1 node acquires 60% of resources of corresponding tenant 2 node, and user 2-2 node acquires 40% of resources of corresponding tenant 2 node. A user can send operation requests such as creating, deleting and updating the queue through the K8S client, and the queue running on the K8S can monitor the user operation requests through the management control center to operate the queue, such as adding events, deleting events and updating events.

Secondly, a resource management center is established by using the YARN, as shown in fig. 4, wherein the resource management component includes a queue resource information management service module, a queue operation service module, a tenant user queue relationship service module and a task operation scheduling service module.

The queue resource information management service module can be used for displaying the current information view of each sub-queue, counting the resource information of each sub-queue and triggering and updating the queue resource when the task to be scheduled is completed; the queue operation service module can be used for processing a first user operation queue request and updating resource information of each sub-queue; the tenant user queue relationship service module can be used for binding a tenant and a queue relationship and providing tenant and first user unbinding queue operation, wherein the first user corresponds to a first sub-queue, the first sub-queue belongs to a first tenant node, and an unbound queue can be placed into a default tenant node by default; the task operation scheduling module can be used for selecting a task scheduling state according to the priority of a task to be scheduled and the amount of free resources in a current queue after receiving a user operation request, the task with high priority is scheduled first, the task with low priority can be scheduled after the queue resources sufficiently support operation, then the task with low priority is bound to different examples according to different task types, and the task with low priority and short operation time can be driven away if necessary, namely the task with low priority and short operation time is transferred to a task waiting storage area. The YARN and the K8S can schedule the queue task according to the priority level of the task corresponding to the user node, the queue resource, the capacity of the queue resource and namespace (namespace).

As shown in fig. 5, in the information view, it can be seen that, queue 1-1 and queue 1-2 are sub-queues of queue 1, total resources of queue 1 are jointly allocated, and the weight and priority of each queue can be seen in the view, for example, queue 1-1 is a high-level queue, and queue 1-2 is a low-level queue; and the resource usage upper and lower limits of the queue resource, such as the central processing unit and the memory of the queue task. And when the task to be scheduled is completed, triggering the queue operation service module 202 to update the queue resources. In practical applications, the operation queue request may include operations such as creating a queue, deleting a queue, and updating a queue.

Finally, scheduling the task to be executed according to the resource management component, as shown in fig. 6, binding the queue and running the queue task when the queue is waiting, releasing the resource to update the queue resource and then ending the queue task when the task runs successfully, and releasing the queue task resource to end the task to check if the task runs unsuccessfully. If necessary, the tasks with low priority can be evicted and transferred to the task waiting storage area, and the resources to be queued are enough to support the tasks to be scheduled to run and then added into the queue to resume running.

A third aspect of the embodiments of the present application provides a task processing device, which is applied to a task scheduling system, where the task scheduling system is provided with a plurality of tenant nodes, each of the tenant nodes corresponds to a corresponding number of queue resources, each of the tenant nodes includes a plurality of sub-queues, and the sub-queues correspond to users one to one, and the device includes, as shown in fig. 7:

a to-be-scheduled task obtaining module 701, configured to obtain a to-be-scheduled task initiated by a first user, where the first user corresponds to a first sub-queue, and the first sub-queue belongs to a first tenant node;

a queue resource selecting module 702, configured to select a queue resource to be scheduled from queue resources of a second sub-queue when a current queue resource of the first sub-queue is insufficient to support the task to be scheduled and the first tenant node has the second sub-queue meeting a preset resource scheduling condition;

a queue resource changing module 703, configured to change the queue resource to be scheduled into a queue resource of the first sub-queue;

a task to be scheduled adding module 704, configured to add the task to be scheduled to the first sub-queue.

By applying the device of the embodiment, when the current queue resource of the first queue is not enough to support the task to be scheduled, the selected queue resource in the second sub-queue is changed into the queue resource of the first sub-queue for completing the scheduling of the task to be scheduled, so that the utilization rate of the queue resource can be improved, and the rationality of the utilization of the queue resource can be increased.

In a possible implementation manner, the number of the tasks to be scheduled is multiple, and the task to be scheduled adding module is specifically configured to add the tasks to be scheduled to the first sub-queue in sequence according to a sequence from a high priority to a low priority of each task to be scheduled until the tasks to be scheduled are added completely or queue resources of the first sub-queue are insufficient to continue adding the tasks to be scheduled.

In a possible embodiment, the apparatus further comprises:

and the task to be scheduled storage module is used for storing the tasks to be scheduled which are not added into the task waiting storage area under the condition that the queue resources of the first sub-queue are not enough to continue to add the tasks to be scheduled.

In a possible embodiment, the apparatus further comprises:

In one possible embodiment, the apparatus further comprises:

In a possible embodiment, the apparatus further comprises:

the system comprises a creation instruction acquisition module, a creation instruction acquisition module and a processing module, wherein the creation instruction acquisition module is used for acquiring a tenant node creation instruction, and the tenant node creation instruction comprises the total queue resource amount of the tenant node;

In a possible embodiment, the apparatus further comprises:

In a possible implementation manner, a Yarn component and a K8S component are arranged in the task processing system, and creation, deletion and queue resource division of tenant nodes and sub-queues are realized through the Yarn component or the K8S component.

In the technical scheme of the application, the operations of obtaining, storing, using, processing, transmitting, providing, disclosing and the like of the personal information of the related user, all with the authorization of the user.

An embodiment of the present application further provides an electronic device, as shown in fig. 8, including:

a memory 801 for storing a computer program;

the processor 802 is configured to implement the following steps when executing the program stored in the memory 801:

acquiring a task to be scheduled initiated by a first user, wherein the first user corresponds to a first sub-queue, and the first sub-queue belongs to a first tenant node;

under the conditions that the current queue resources of the first sub-queue are not enough to support the tasks to be scheduled and a second sub-queue meeting preset resource scheduling conditions exists in the first tenant node, selecting queue resources to be scheduled from the queue resources of the second sub-queue;

changing the queue resources to be scheduled into the queue resources of the first sub-queue;

and adding the task to be scheduled to the first sub-queue.

The electronic device may further include a communication bus and/or a communication interface, and the processor 802, the communication interface, and the memory 801 communicate with each other through the communication bus.

The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the electronic equipment and other equipment.

The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.

The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.

In yet another embodiment provided by the present application, a computer-readable storage medium is further provided, in which a computer program is stored, and the computer program realizes the steps of any one of the above task processing methods when executed by a processor.

In yet another embodiment provided by the present application, there is also provided a computer program product containing instructions, which when run on a computer, cause the computer to perform any one of the task processing methods of the above embodiments.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid State Disk (SSD)), among others.

It should be noted that, in this document, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising a … …" does not exclude the presence of another identical element in a process, method, article, or apparatus that comprises the element.

All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on differences from other embodiments. In particular, as for the apparatus embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The above description is only for the preferred embodiment of the present application and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application are included in the scope of protection of the present application.

Claims

1. A task processing method is characterized in that the task processing method is applied to a task processing system, a plurality of tenant nodes are arranged in the task processing system, each tenant node corresponds to a corresponding number of queue resources, the tenant nodes comprise a plurality of sub-queues, and the sub-queues correspond to users one by one, and the method comprises the following steps:

the method comprises the steps of obtaining a task to be scheduled, which is initiated by a first user, wherein the first user corresponds to a first sub-queue, and the first sub-queue belongs to a first tenant node;

and adding the task to be scheduled to the first sub-queue.

2. The method of claim 1, wherein the current queue resources of the first sub-queue do not exceed the upper resource limit of the first sub-queue; for each sub-queue except the first sub-queue in the first tenant node, under the condition that the priority of the sub-queue is lower than the priority of the first sub-queue and the current queue resource of the sub-queue is larger than the resource lower limit of the sub-queue, the sub-queue is judged to be a second sub-queue meeting a preset resource scheduling condition;

3. The method according to claim 1, wherein the number of the tasks to be scheduled is multiple, and the adding the tasks to be scheduled to the first sub-queue comprises:

4. The method of claim 3, further comprising:

5. The method of claim 1, wherein prior to the step of adding the task to be scheduled to the first sub-queue, the method further comprises:

6. The method according to claim 4 or 5, characterized in that the method further comprises:

7. The method of claim 1, further comprising:

8. The method of claim 1, further comprising:

9. The method according to claim 7 or 8, wherein a Yarn component and a K8S component are provided in the task processing system, and creation, deletion, and queue resource division of the tenant node and the sub-queue are realized by the Yarn component or the K8S component.

10. A task processing system is characterized in that a plurality of tenant nodes are arranged in the task processing system, each tenant node corresponds to a corresponding number of queue resources, each tenant node comprises a plurality of sub-queues, the sub-queues correspond to users one by one, and the task processing system comprises: a resource management component, the resource management component comprising:

the task operation scheduling module is further configured to add the task to be scheduled to the first sub-queue.

11. The system of claim 10, wherein the resource management component further comprises:

12. A task processing device is applied to a task processing system, a plurality of tenant nodes are arranged in the task processing system, each tenant node corresponds to a corresponding number of queue resources, the tenant nodes comprise a plurality of sub-queues, the sub-queues correspond to users one by one, and the device comprises:

the scheduling method comprises the steps that a to-be-scheduled task obtaining module is used for obtaining a to-be-scheduled task initiated by a first user, wherein the first user corresponds to a first sub-queue, and the first sub-queue belongs to a first tenant node;

13. An electronic device, comprising:

a memory for storing a computer program;

a processor for implementing the method of any one of claims 1 to 9 when executing a program stored in a memory.

14. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method of any one of claims 1 to 9.