WO2021190088A1

WO2021190088A1 - Task distribution method and apparatus, and task execution system and server

Info

Publication number: WO2021190088A1
Application number: PCT/CN2021/071014
Authority: WO
Inventors: 刘向东
Original assignee: 北京金山云网络技术有限公司
Priority date: 2020-03-26
Filing date: 2021-01-11
Publication date: 2021-09-30
Also published as: CN111459666A

Abstract

A task distribution method and apparatus, and a task execution system and a server, relating to the field of cloud computing. The method comprises: first obtaining a task to be distributed of a target tenant, the target tenant corresponding to a first distribution node (S202); and putting said task into a preset queue to be distributed, so that the first distribution node or distribution nodes except the first distribution node distribute said task to a task executor (S204). According to the method, each distribution node manages the corresponding target tenant; tasks to be distributed of corresponding target tenants can be put into a queue to be distributed; all the distribution nodes can distribute all said tasks in said queue; compared with the mode that a distribution node distributes the tasks of all tenants, the mode has the advantages that the pressure of the distribution tasks can be dispersed to all the distribution nodes, even if the task quantity is large, the pressure dispersed to all the distribution nodes is small, and therefore stable operation of the task scheduling system is guaranteed, and the distribution efficiency of the tasks is improved.

Description

Task dispatching method, device, task execution system and server

This disclosure claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 202010226220.1, and the invention title is "task dispatch method, device, task execution system and server" on March 26, 2020, the entire content of which is incorporated by reference In this disclosure.

Technical field

The present disclosure relates to the technical field of task scheduling, in particular to a task dispatch method, device, task execution system and server.

Background technique

In related technologies, the task scheduling system usually dispatches tasks to execution nodes through dispatching nodes, and only one dispatching node can dispatch tasks at the same time. This method can avoid task dispatch conflicts among multiple dispatching nodes, and the operation is simple. When the amount of tasks increases, the pressure of the dispatch node at a certain point in time suddenly increases, and it is easy to reach the performance bottleneck of the dispatch node, resulting in the failure of the entire task scheduling system, thereby affecting the efficiency of task dispatch.

Summary of the invention

The purpose of the embodiments of the present disclosure is to provide a task dispatching method, device, task execution system and server, so as to improve the efficiency of task dispatching.

In the first aspect, the embodiments of the present disclosure provide a task dispatching method, which is applied to a first dispatching node; the method includes: obtaining tasks to be dispatched by a target tenant; wherein the target tenant corresponds to the first dispatching node; The task to be dispatched is placed in a preset queue to be dispatched for the first dispatching node or dispatching nodes other than the first dispatching node to dispatch the task to be dispatched to the task executor.

In one embodiment, before the step of obtaining the task to be dispatched by the target tenant, the method further includes: obtaining the target tenant from a preset tenant pool; wherein the tenant pool includes at least one tenant, and each tenant is Obtained by the first dispatching node or a dispatching node other than the first dispatching node.

In one embodiment, after the above step of placing the task to be dispatched into the preset queue for dispatch, the method further includes: putting the target tenant into the tenant pool for the first dispatching node or other than the first dispatching node Dispatching nodes other than those obtain the target tenant from the tenant pool.

In one embodiment, the target tenant is preset with a job ready queue; the job ready queue is used to store the tasks to be dispatched by the target tenant; the above step of obtaining the tasks to be dispatched by the target tenant includes: obtaining the target from the job ready queue Tenant's tasks to be dispatched.

In an embodiment, the tasks to be dispatched stored in the job ready queue carry the priority of the tasks to be dispatched; wherein, the priority is set by a priority configuration instruction in the Redis service.

In an embodiment, the above-mentioned priority configuration command includes a zset command or a list command setting.

In one embodiment, after the above step of placing the task to be dispatched into the preset queue for dispatch, the method further includes: judging whether the task to be dispatched meets the preset dispatch conditions; if the dispatch conditions are not met, it will be dispatched The task is returned to the target tenant.

In one embodiment, the above-mentioned preset dispatch conditions include: the resource occupation of the tasks to be dispatched is less than or equal to the remaining available resources of the system, and the number of tasks to be dispatched currently being dispatched is less than or equal to the remaining resources that can be dispatched at the same time in the system. The number of tasks to be dispatched.

In one embodiment, the target tenant is preset with a dispatch record queue; the dispatch record queue is used to store the dispatched tasks of the target tenant and the resource occupancy of the dispatched tasks; the above method further includes: if the dispatch conditions are met, Put the task to be dispatched into the dispatch record queue of the target tenant.

In one embodiment, the above method further includes: if a task is obtained from the queue to be dispatched, putting the obtained task into a preset dispatched queue, so that the task executor obtains the task from the dispatched queue, and Perform the acquired task.

In a second aspect, embodiments of the present disclosure provide a task dispatching device, which is set as a first dispatching node; the device includes: a task acquisition module, which is set to acquire tasks to be dispatched by a target tenant; wherein, the target tenant and the first dispatch node Corresponding to the dispatch node; the task dispatch module is set to place the task to be dispatched into the preset queue for dispatch, so that the first dispatch node or dispatch nodes other than the first dispatch node can dispatch the task to be dispatched to the task executor .

In one embodiment, the above-mentioned device further includes a target tenant determination module, which is configured to: obtain target tenants from a preset tenant pool; wherein, the tenant pool includes at least one tenant, and each tenant is assigned by the first dispatch node or the second tenant. Obtained by a dispatching node other than a dispatching node.

In an embodiment, the above-mentioned apparatus further includes a tenant playback module, configured to put the target tenant into the tenant pool, so that the first dispatch node or dispatch nodes other than the first dispatch node can obtain the target tenant from the tenant pool.

In one embodiment, the target tenant is preset with a job ready queue; the job ready queue is set to store the tasks to be dispatched by the target tenant; the task acquisition module is set to: acquire the tasks to be dispatched by the target tenant from the job ready queue .

In an embodiment, the above-mentioned priority configuration command includes a zset command or a list command.

In one embodiment, the above-mentioned device further includes a dispatch judging module configured to judge whether the task to be dispatched meets the preset dispatch conditions; if the dispatch conditions are not met, the task to be dispatched is returned to the target tenant.

In one embodiment, the above-mentioned preset dispatch conditions include: the resource occupation of the tasks to be dispatched is less than or equal to the remaining available resources of the system, and the number of tasks to be dispatched currently being dispatched is less than or equal to the remaining of the system that can be dispatched at the same time The number of tasks to be dispatched.

In one embodiment, the target tenant has a preset dispatch record queue; the dispatch record queue is set to store the dispatched tasks of the target tenant and the resource occupancy of dispatched tasks; the dispatch judgment module is also set as follows: Dispatch condition, put the task to be dispatched into the dispatch record queue of the target tenant.

In one embodiment, the above-mentioned device further includes a dispatched task obtaining module, which is configured to: if a task is obtained from the queue to be dispatched, the obtained task is put into a preset dispatched queue, so that the task executor can be removed from the dispatched queue. Obtain tasks in the dispatch queue and execute the obtained tasks.

In a third aspect, embodiments of the present disclosure provide a task execution system, which includes multiple task executors and multiple dispatch nodes; each dispatch node is in communication connection with multiple task executors; and the dispatch node is set to: obtain a target Tasks to be dispatched by the tenant; among them, the target tenant corresponds to the dispatch node; put the tasks to be dispatched into the preset queue for dispatch, so that the dispatch node or dispatch nodes other than the dispatch node among multiple dispatch nodes will be dispatched The task is dispatched to the task executor; the task executor is set to execute the task to be dispatched.

In a fourth aspect, embodiments of the present disclosure provide a server including a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above task dispatch method.

In a fifth aspect, embodiments of the present disclosure provide a machine-readable storage medium that stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine can The execution of the instructions prompts the processor to implement the above-mentioned task dispatch method.

The embodiments of the present disclosure bring the following beneficial effects:

The embodiments of the present disclosure provide a task dispatch method, device, task execution system, and server. First, obtain the task to be dispatched by the target tenant; wherein the target tenant corresponds to the first dispatch node; and then put the task to be dispatched into the pre-dispatch node In the queue to be dispatched, the first dispatching node or dispatching nodes other than the first dispatching node may dispatch the task to be dispatched to the task executor. In the present invention, each dispatch node separately manages the corresponding target tenants, and can put the tasks to be dispatched of the corresponding target tenants into the dispatch queue. All dispatch nodes can dispatch all the tasks to be dispatched in the dispatch queue. The tasks to be dispatched by multiple tenants are distributed to each dispatching node to manage separately. Compared with the way that one dispatching node dispatches the tasks of all tenants, this method can spread the pressure of dispatching tasks to each dispatching node, even when the overall task volume is large , The pressure distributed to each dispatch node is also small, so as to ensure the smooth operation of the task scheduling system and improve the efficiency of task dispatch.

Description of the drawings

In order to more clearly describe the technical solutions of the embodiments of the present disclosure or related technologies, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments or related technologies. Obviously, the accompanying drawings in the following description are merely the present invention. For some of the embodiments, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work.

FIG. 1 is a schematic structural diagram of a task dispatch scenario in related technologies provided by the present disclosure;

Figure 2 is a flowchart of a task dispatch method provided by the present disclosure;

FIG. 3 is a flowchart of another task dispatching method provided by the present disclosure;

FIG. 4 is a flowchart of another task dispatching method provided by the present disclosure;

FIG. 5 is a schematic structural diagram of a task dispatching device provided by the present disclosure;

FIG. 6 is a schematic structural diagram of a task execution system provided by the present disclosure;

FIG. 7 is a schematic structural diagram of a server provided by the present disclosure.

Detailed ways

In order to make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments It is a part of the embodiments of the present disclosure, but not all of the embodiments. The components of the embodiments of the present disclosure generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations.

Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed embodiments of the present disclosure, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the embodiments of the present disclosure.

In the scheduling system, task dispatch is a very important function. Task dispatch is to select a suitable execution node to execute the task according to a certain strategy from a plurality of task execution nodes to be selected. In related technologies, task dispatch in a task scheduling system usually dispatches tasks to execution nodes through dispatch nodes, and only one master dispatch node can dispatch tasks at the same time. Figure 1 shows a task dispatch scenario in related technologies. For example, common task scheduling systems such as elastic-job and xxl-job implement dynamic master selection through distributed coordination components such as ZooKeeper, and select the master dispatch node that dispatches tasks at a certain time, and the master dispatch node performs Task dispatch. This method can avoid task dispatch conflicts among multiple dispatching nodes, and the operation is simple. However, when the amount of tasks increases, the pressure of the dispatching node at a certain point in time increases suddenly, which easily reaches the performance bottleneck of the dispatching node, resulting in the entire The task scheduling system fails, which affects the efficiency of task dispatch.

Based on this, the embodiments of the present disclosure provide a task dispatch method, device, task execution system, and server. The technology can be applied to a task dispatch scenario in a private cloud or a public cloud with a relatively large business scale. In order to facilitate the understanding of the embodiments of the present disclosure, a task dispatch method disclosed in the embodiments of the present disclosure is first introduced in detail. As shown in FIG. 2, the method is applied to the first dispatch node, and the method includes the following steps:

Step S202: Obtain the tasks to be dispatched by the target tenant; wherein, the target tenant corresponds to the first dispatch node.

The above-mentioned first dispatch node is usually any one of a plurality of dispatch nodes in the task scheduling system. The first dispatch node is usually a server, and may also be a service process on the server. Each dispatch node has its corresponding target tenant, where each dispatch node can correspond to one or more target tenants, each dispatch node corresponds to a different target tenant, and the same target tenant can only correspond to one dispatch node. The aforementioned target tenant usually refers to a customer who uses a company's cloud environment, and the customer can be a project team, work group, company, etc.

The above-mentioned first dispatching node may obtain the task to be dispatched of its corresponding target tenant, and the task to be dispatched may be tasks such as calculation and storage sent by the tenant through the terminal device. In an embodiment, the first dispatch node may obtain the task to be dispatched from the terminal device corresponding to the target tenant, and may also receive the task to be dispatched from the target tenant. Since each dispatch node is in charge of different target tenants, when multiple dispatch nodes obtain tasks to be dispatched at the same time, conflicts when different dispatch nodes obtain tasks to be dispatched from the same tenant at the same time can be avoided.

Step S204: Put the above-mentioned task to be dispatched into a preset queue for dispatch, so that the first dispatch node or a dispatch node other than the first dispatch node dispatches the task to be dispatched to the task executor.

The above-mentioned queue to be dispatched usually adopts a data structure corresponding to MQ (Message Queue) to store the tasks to be dispatched. The MQ message queue can be regarded as a container for storing transmission data, and the transmission data is also the task to be dispatched. The MQ message queue is usually an asynchronous inter-service communication method, and it is also an important component in a distributed system. It can solve the problems of application coupling, asynchronous messaging, traffic cutting, etc.; it also achieves high performance, high availability, scalability and Eventually consistent architecture, MQ message queue usually has RocketMQ, RabbitMQ, Kafka and other forms.

In one embodiment, all dispatch nodes subscribe to the dispatch queue, so that each dispatch node can dispatch any task to be dispatched in the dispatch queue. It can also be understood that the first dispatch node can not only dispatch the dispatch queue The tasks to be dispatched by the target tenants corresponding to the tasks in the dispatch queue can also be dispatched to the tasks to be dispatched by the target tenants corresponding to the dispatch nodes other than the first dispatch node.

Generally, subscribing to topics under MQ can be used to make all the dispatching nodes subscribe to the dispatch queue. The above-mentioned way of subscribing to the topic under MQ is usually that each dispatch node puts the obtained task to be dispatched into the message queue under the topic, and all the dispatch nodes subscribed to the topic can obtain any one from the message queue under the topic. Tasks to be dispatched. In an embodiment, all the dispatch nodes can subscribe to the same topic, so that all the dispatch nodes can dispatch all the tasks to be dispatched in the dispatch queue.

In the process of dispatch, if a task to be dispatched has been dispatched, the dispatched task can be marked as "dispatched" in the dispatch queue to prevent the task to be dispatched from being dispatched repeatedly, and this method can be used Realize multiple dispatch nodes to dispatch at the same time, thereby improving the dispatch efficiency of task dispatch.

The embodiment of the present disclosure provides a task dispatch method. Firstly, the target tenant's task to be dispatched is obtained; wherein the target tenant corresponds to the first dispatch node; and then the task to be dispatched is placed in a preset queue to be dispatched. For the first dispatching node or dispatching nodes other than the first dispatching node to dispatch the task to be dispatched to the task executor. In the embodiment of the present disclosure, each dispatch node manages the corresponding target tenants, and can put the tasks to be dispatched of the corresponding target tenants into the dispatch queue, and all dispatch nodes can dispatch all the tasks to be dispatched in the dispatch queue. This way The tasks to be dispatched by multiple tenants can be distributed to each dispatching node for management separately. Compared with the way that one dispatching node dispatches the tasks of all tenants, this method can spread the pressure of dispatching tasks to each dispatching node, even when the overall task volume is relatively high. When it is large, the pressure distributed to each dispatch node is also small, so as to ensure the smooth operation of the task scheduling system and improve the efficiency of task dispatch.

The embodiment of the present disclosure also provides another task dispatch method, which is implemented on the basis of the method in the above embodiment; the method focuses on the specific process of obtaining the task to be dispatched by the target tenant (implemented through the following step S304); As shown in Figure 3, the method includes the following steps:

Step S302: Obtain a target tenant from a preset tenant pool; wherein the target tenant has a preset job ready queue, and the job ready queue is used to store tasks to be dispatched by the target tenant; the tenant pool includes at least one tenant, each Ten tenants are acquired by the first dispatching node or a dispatching node other than the first dispatching node.

The above-mentioned preset tenant pool usually pools tenants through Redis (Remote Dictionary Server) to obtain the tenant pool. The Redis mentioned above is usually a key-value storage system, and Redis also supports sorting in different ways. All tenants are stored in the above tenant pool, and all dispatch nodes compete for tenant resources from the tenant pool, that is, each dispatch node obtains the corresponding target tenant from the tenant pool, and each dispatch node can obtain one or more Tenants, and each tenant can only be obtained by one dispatch node. For example, there are 6 tenants in the tenant pool, represented by T1, T2, T3, T4, T5, T6, and there are 3 dispatching nodes in the task scheduling system, then the target tenants corresponding to the first dispatching node can be T1 and T3. The target tenant corresponding to the second dispatching node may be T6, and the tenants corresponding to the third dispatching node are T2, T4, and T5.

The aforementioned target tenant is preset with a job ready queue, and the job ready queue is used to store tasks to be dispatched by the target tenant. Usually each tenant has a corresponding job ready queue. The job ready queue can use the data structure corresponding to Redis to save the tasks to be dispatched corresponding to the target tenant. This data structure relies on the advantages of operating data in Redis memory and provides a series of practicalities. The unique interface controls data, is easy to use, fast, and can realize unique data feature sorting and reading.

In an embodiment, the tasks to be dispatched stored in the job ready queue carry the priority of the tasks to be dispatched; wherein, the priority is set by a priority configuration instruction in the Redis service. The priority configuration command can be a zset command or a list command.

The priority of the tasks to be dispatched above may characterize the order in which the tasks to be dispatched are dispatched, and the tasks to be dispatched with higher priority are usually given priority. In an embodiment, the priority of the task to be dispatched can be set in the job ready queue according to the zset instruction or the list instruction in the Redis service.

When the zset instruction is used, the job ready queue can adopt the zset data structure, which is similar to the set, and adds the sequence attribute on the basis of the set. The order attribute can specify the priority when adding the queue to be dispatched. After each priority is specified, zset will automatically re-adjust the order of the tasks in the queue. It can also be understood that there are two lists in the job ready queue, one for the tasks to be dispatched , A column stores the priority corresponding to the task to be dispatched.

When the list instruction is used, the job ready queue can adopt a list data structure. The list data structure is usually a linked list structure. The job ready queue can include multiple lists of list data structures, and each list corresponds to a priority, for example, The priority of the task to be dispatched is divided into 5 levels. There are five lists of list data structure in the job ready queue, and the tasks to be dispatched of different priority are stored respectively.

Step S304: Obtain the tasks to be dispatched by the target tenant from the above-mentioned job ready queue.

When the dispatch node obtains the task to be dispatched from the job ready queue, it usually first obtains the task to be dispatched with high priority, so that the dispatch node can first dispatch the task to be dispatched with high priority, so that the task to be dispatched can be executed first. Enhance the experience of tenants.

Step S306: Put the above-mentioned task to be dispatched into a preset queue to be dispatched, so that the first dispatching node or dispatching nodes other than the first dispatching node can dispatch the task to be dispatched to the task executor.

In one embodiment, after the first dispatching node puts the task to be dispatched into the preset dispatch queue, the method further includes: putting the target tenant into the tenant pool for the first dispatching node or other than the first dispatching node. The dispatch node other than the node obtains the target tenant from the tenant pool.

The target tenant acquired by each dispatch node changes dynamically, that is, when the dispatch node puts the target tenant's to-be-dispatched tasks into the dispatch queue, the dispatch node will put the target tenant back into the tenant pool. When the dispatching node dispatches tasks next time, it can obtain tenants from the tenant pool again. The obtained tenants may be the target tenants mentioned above, or they may be other tenants besides the target tenants, so that each tenant can be dispatched reasonably and efficiently. Tenant's tasks to be dispatched.

The above task dispatch method firstly obtains the target tenant from the preset tenant pool; then obtains the task to be dispatched by the target tenant from the job ready queue corresponding to the target tenant; then puts the task to be dispatched into the preset queue to be dispatched , For the first dispatching node or dispatching nodes other than the first dispatching node to dispatch the task to be dispatched to the task executor. In this way, each dispatch node manages the corresponding target tenant separately, which avoids the conflict phenomenon when multiple dispatch nodes dispatch the tasks to be dispatched by one tenant at the same time. At the same time, all dispatch nodes can dispatch the tasks to be dispatched in the dispatch queue at the same time. Therefore, there is no sudden increase in the pressure of a single dispatch node in this method, which ensures the normal operation of the task scheduling system, thereby improving the efficiency of task dispatch.

The embodiment of the present disclosure also provides another task dispatching method, which is implemented on the basis of the method in the above-mentioned embodiment; the method focuses on describing the specific dispatching process after placing the above-mentioned task to be dispatched into the preset waiting queue; As shown in Figure 4, the method includes the following steps:

Step S402: Obtain the tasks to be dispatched by the target tenant; wherein, the target tenant corresponds to the first dispatch node.

Step S404: Put the above-mentioned task to be dispatched into a preset queue for dispatch.

In step S406, it is judged whether the above task to be dispatched meets the preset dispatch conditions; if the dispatch conditions are not met, step S408 is executed; if the dispatch conditions are met, step S410 is executed.

Usually, business logic is used to determine whether the task to be dispatched meets the preset dispatch conditions. The business logic defines the conditions that must be met to complete an action, that is, defines the dispatch conditions that need to be met when the task is dispatched. The above business logic is usually stored in a preset database. The database can use the string data type in the Redis database to store the business logic. Among them, string is usually a string, which is the simplest data type. The above-mentioned business logic may include the remaining available resources of the system, the remaining number of tasks to be dispatched that can be dispatched at the same time, etc.; the remaining available resources of the system may also be referred to as a logical resource group, which may include multiple types of remaining available resources, for example , The remaining amount of the Oracle connection, the remaining amount of memory, etc.; the remaining number of tasks to be dispatched in the system that can be dispatched at the same time can refer to the number of tasks to be dispatched that the system allows multiple dispatching nodes to dispatch at the same time.

In one embodiment, the business logic required for different tasks to be dispatched may be the same or different. For example, the number of Oracle connections required for task 1 to be dispatched is 10, and the number of Oracle connections required for task 2 to be dispatched is 20.

The aforementioned preset dispatch conditions include: the resource occupation of the tasks to be dispatched is less than or equal to the remaining available resources of the system, and the number of tasks to be dispatched currently being dispatched is less than or equal to the number of remaining tasks to be dispatched in the system that can be dispatched at the same time. number.

Only when the task to be dispatched meets the above dispatch conditions, can the task in the dispatch queue be dispatched. After the dispatch of the task to be dispatched is completed, it is necessary to update the business logic in the preset database according to the resource occupation of the task to be dispatched. For example, assuming that the remaining available resources of the system is 100, the remaining number of tasks to be dispatched in the system that can be dispatched at the same time is 20, and the resource occupancy of tasks to be dispatched is 40, the number of tasks being dispatched by the dispatching node at this time is 10. Then the dispatch node can dispatch the task to be dispatched. After the dispatch of the task to be dispatched is completed, the remaining available resource amount in the preset database is updated to 60.

Step S408: Return the task to be dispatched to the target tenant.

In one embodiment, if the task to be dispatched does not meet the dispatch conditions, the task to be dispatched is returned to the job ready queue corresponding to the target tenant, so as to wait for the dispatch node to take the task to be dispatched from the job ready queue and put it in In the queue to be dispatched.

Step S410: Put the aforementioned tasks to be dispatched into the dispatch record queue of the target tenant; the target tenant presets a dispatch record queue, and the dispatch record queue is used to store the dispatched tasks of the target tenant and the resource occupation of the dispatched tasks quantity.

The above-mentioned dispatch record queue usually adopts the Redis list data structure. Usually, each tenant has a dispatch record queue. The dispatch record queue can record the dispatched tasks of the target tenant and the resource occupation of the dispatched tasks. The total resource occupancy of dispatch tasks can also be compared with the remaining available resources in the business logic in the database to ensure the accuracy of the remaining available resources recorded in the data.

Step S412: Put the tasks to be dispatched in the aforementioned dispatch record queue into the dispatched queue, so that the first dispatch node or dispatch nodes other than the first dispatch node can dispatch the tasks to be dispatched to the task executor.

After the record queue to be dispatched is recorded, the dispatching node dispatches the recorded tasks to be dispatched to the dispatched queue so that all dispatching nodes can dispatch the dispatched tasks in the dispatched queue to the task executor. The above-mentioned dispatched queue may use the data structure corresponding to MQ to store the tasks to be dispatched.

In some embodiments, if a task is obtained from the queue to be dispatched, the obtained task is put into the preset dispatched queue, so that the task executor obtains the task from the dispatched queue and executes the obtained task . In one embodiment, the task executor can subscribe to tasks in the dispatched queue, so that the task executor can actively obtain the task to be executed from the dispatched queue and execute the acquired task to be executed.

The above task dispatch method first obtains the tasks to be dispatched by the target tenant; then puts the tasks to be dispatched into the preset dispatch queue, and then determines whether the dispatch tasks meet the preset dispatch conditions; if the dispatch conditions are not met , Return the task to be dispatched to the target tenant; if the dispatch conditions are met, put the task to be dispatched into the dispatch record queue of the target tenant, and then put the dispatched task in the dispatch record queue into the dispatched queue for The first dispatch node or dispatch nodes other than the first dispatch node dispatch the task to be dispatched to the task executor. In this way, multiple dispatch nodes can dispatch tasks in parallel, and in the case of a large amount of tasks, higher dispatch efficiency can still be achieved.

Corresponding to the embodiment of the task dispatching method described above, the embodiment of the present disclosure also provides a task dispatching device, which is set as the first dispatching node; as shown in FIG. 5, the device includes:

The task acquisition module 50 is configured to acquire the tasks to be dispatched by the target tenant; wherein, the target tenant corresponds to the first dispatch node.

The task dispatch module 51 is configured to put the tasks to be dispatched into a preset queue for dispatch, so that the first dispatch node or dispatch nodes other than the first dispatch node dispatch the tasks to be dispatched to the task executor.

The above task dispatching device first obtains the tasks to be dispatched by the target tenant; among them, the target tenant corresponds to the first dispatch node; and then puts the tasks to be dispatched into the preset waiting queue for the first dispatch node or other The dispatch node other than the first dispatch node dispatches the task to be dispatched to the task executor. In this method, each dispatch node manages the corresponding target tenants, and can put the tasks to be dispatched of the corresponding target tenants into the dispatch queue. All dispatch nodes can dispatch all the tasks to be dispatched in the dispatch queue. The tasks to be dispatched by multiple tenants are distributed to each dispatching node to manage separately. Compared with the way that one dispatching node dispatches all tenants' tasks, this method can spread the pressure of dispatching tasks to each dispatching node, even when the overall task volume is large , The pressure distributed to each dispatch node is also less, so as to ensure the smooth operation of the task scheduling system and improve the efficiency of task dispatch.

In one embodiment, the above-mentioned device further includes a target tenant determination module, which is configured to obtain target tenants from a preset tenant pool; wherein the tenant pool includes at least one tenant, and each tenant is removed by the first dispatch node or Obtained by a dispatch node other than the first dispatch node.

In one embodiment, the above-mentioned apparatus further includes a tenant playback module, configured to put the target tenant into the tenant pool, so that the first dispatching node or dispatching nodes other than the first dispatching node can obtain the target tenant from the tenant pool.

In one embodiment, the target tenant is preset with a job ready queue; the job ready queue is set to store the tasks to be dispatched by the target tenant; the task acquisition module 50 is set to: obtain the job ready queues to be dispatched from the target tenant Task.

The task to be dispatched stored in the above job ready queue carries the priority of the task to be dispatched; wherein, the priority is set by a priority configuration instruction in the Redis service.

The above-mentioned device also includes a dispatch judging module, which is configured to judge whether the task to be dispatched meets the preset dispatch conditions; if the dispatch condition is not met, the task to be dispatched is returned to the target tenant.

In one embodiment, the target tenant is preset with a dispatch record queue; the dispatch record queue is set to store the dispatched tasks of the target tenant and the resource occupancy of dispatched tasks; the dispatch judgment module is also set as follows: Dispatch condition, put the task to be dispatched into the dispatch record queue of the target tenant.

The implementation principles and technical effects of the task dispatching device provided in the embodiments of the present disclosure are the same as those of the foregoing method embodiments. For a brief description, for the parts not mentioned in the device embodiments, please refer to the corresponding content in the foregoing method embodiments.

Corresponding to the embodiment of the above-mentioned task dispatch method, the embodiment of the present disclosure also provides a task execution system. As shown in FIG. 6, the system includes a plurality of task executors 60 and a plurality of dispatch nodes 61; each dispatch node 61 It is connected to a plurality of task executors 60 in communication.

The above dispatch node 61 is set to: obtain the task to be dispatched by the target tenant; wherein the target tenant corresponds to the dispatch node; and put the task to be dispatched into the preset queue to be dispatched for the dispatch node or multiple dispatch nodes The dispatching nodes other than the dispatching node dispatch the task to be dispatched to the task executor 60.

The above-mentioned task executor 60 is configured to execute the task to be dispatched.

The above-mentioned task execution system includes multiple task executors and multiple dispatch nodes, wherein the dispatch node is set to obtain the tasks to be dispatched by the target tenant, and put the tasks to be dispatched into a preset queue for dispatch for dispatch The node or dispatching nodes other than the dispatching node among the dispatching nodes dispatch the task to be dispatched to the task executor; the task executor is set to execute the task to be dispatched. The dispatching node in the system is in charge of different tenants, so that multiple dispatching nodes can dispatch tasks at the same time, thereby improving the efficiency of task dispatching and execution; at the same time, the system can meet the needs of massive task dispatching, and the system's capabilities can be added by adding The new instance easily realizes the current expansion.

The embodiment of the present disclosure also provides a server. As shown in FIG. 7, the server includes a processor 101 and a memory 100. The memory 100 stores machine executable instructions that can be executed by the processor 101. The processor 101 executes the machine Executable instructions to implement the above task dispatch method.

Further, the server shown in FIG. 7 further includes a bus 102 and a communication interface 103, and the processor 101, the communication interface 103, and the memory 100 are connected through the bus 102.

The memory 100 may include a high-speed random access memory (RAM, Random Access Memory), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, a wide area network, a local network, a metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, an EISA bus, or the like. The bus can be divided into an address bus, a data bus, a control bus, and so on. For ease of presentation, only one bidirectional arrow is used in FIG. 7, but it does not mean that there is only one bus or one type of bus.

The processor 101 may be an integrated circuit chip with signal processing capability. In the implementation process, the steps of the foregoing method may be completed by an integrated logic circuit of hardware in the processor 101 or instructions in the form of software. The foregoing processor 101 may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU for short), a network processor (Network Processor, NP), etc.; it may also be a digital signal processor (Digital Signal Processor, DSP for short), etc. ), Application Specific Integrated Circuit (ASIC), Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps, and logical block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. The steps of the method disclosed in combination with the embodiments of the present disclosure may be directly embodied as being executed and completed by a hardware decoding processor, or executed and completed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100, and completes the steps of the method of the foregoing embodiment in combination with its hardware.

The embodiment of the present disclosure also provides a machine-readable storage medium, the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor The foregoing task dispatching method is implemented. For specific implementation, please refer to the method embodiment, which will not be repeated here.

Those skilled in the art can clearly understand that, for the convenience and conciseness of the description, the specific working process of the device and/or electronic device described above can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions recorded in the foregoing embodiments can still be modified, or some or all of the technical features can be equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present invention. Scope.

Industrial applicability

Based on the task dispatch method, device, task execution system, and server provided by the embodiments of the present disclosure, the tasks to be dispatched by multiple tenants can be distributed to each dispatch node for separate management. Compared with the way that one dispatch node dispatches the tasks of all tenants, this The method can disperse the pressure of dispatching tasks to each dispatching node, even when the overall task volume is large, the pressure distributed to each dispatching node is small, so as to ensure the smooth operation of the task scheduling system and improve the efficiency of task dispatch.

Claims

A task dispatch method, the method is applied to a first dispatch node; the method includes:

Acquiring tasks to be dispatched by the target tenant; wherein the target tenant corresponds to the first dispatch node;

Put the task to be dispatched into a preset queue for dispatch, so that the first dispatch node or a dispatch node other than the first dispatch node dispatches the task to be dispatched to a task executor.
The method according to claim 1, wherein before the step of obtaining the tasks to be dispatched by the target tenant, the method further comprises: obtaining the target tenant from a preset tenant pool; wherein the tenant pool includes At least one tenant, and each tenant is acquired by the first dispatching node or a dispatching node other than the first dispatching node.
The method according to claim 1, wherein after the step of placing the task to be dispatched into a preset queue for dispatch, the method further comprises:

Put the target tenant into the tenant pool, so that the first dispatch node or a dispatch node other than the first dispatch node obtains the target tenant from the tenant pool.
The method according to claim 1, wherein the target tenant is preset with a job ready queue; the job ready queue is used to store tasks to be dispatched by the target tenant;

The step of obtaining the task to be dispatched by the target tenant includes: obtaining the task to be dispatched by the target tenant from the job ready queue.
The method according to claim 4, wherein the task to be dispatched stored in the job ready queue carries the priority of the task to be dispatched; wherein the priority is set by a priority configuration instruction in the Redis service .
The method according to claim 5, wherein the priority configuration instruction includes a zset instruction or a list instruction.
The method according to claim 1, wherein after the step of placing the task to be dispatched into a preset queue for dispatch, the method further comprises:

Determine whether the task to be dispatched meets the preset dispatch conditions;

If the dispatch conditions are not met, the task to be dispatched is returned to the target tenant.
The method according to claim 7, wherein the preset dispatch condition comprises: the resource occupation of the task to be dispatched is less than or equal to the remaining available resources of the system, and the number of the task to be dispatched currently being dispatched The number is less than or equal to the number of remaining tasks to be dispatched in the system that can be dispatched at the same time.
The method according to claim 7, wherein the target tenant is preset with a dispatch record queue; the dispatch record queue is used to store the dispatched tasks of the target tenant and the resource occupancy of the dispatched tasks;

The method further includes: if the dispatch condition is met, placing the task to be dispatched into the dispatch record queue of the target tenant.
The method according to claim 1, wherein the method further comprises:

If a task is obtained from the to-be-dispatched queue, the obtained task is put into the preset dispatched queue, so that the task executor obtains the task from the dispatched queue and executes the obtained task.
A task dispatching device, the device is set as a first dispatching node; the device includes:

A task acquisition module, configured to acquire tasks to be dispatched from a target tenant; wherein the target tenant corresponds to the first dispatch node;

The task dispatch module is configured to put the task to be dispatched into a preset queue for dispatch, so that the first dispatch node or a dispatch node other than the first dispatch node dispatches the task to be dispatched to a task Actuator.
The device according to claim 11, wherein the device further comprises a target tenant determination module, configured to: obtain a target tenant from a preset tenant pool; wherein the tenant pool includes at least one tenant, each The tenant is acquired by the first dispatching node or a dispatching node other than the first dispatching node.
The device according to claim 11, wherein the device further comprises a tenant playback module configured to:

Put the target tenant into the tenant pool, so that the first dispatch node or a dispatch node other than the first dispatch node obtains the target tenant from the tenant pool.
The device according to claim 11, wherein the target tenant has a job ready queue preset; the job ready queue is set to store tasks to be dispatched by the target tenant;

The task obtaining module is configured to obtain the task to be dispatched by the target tenant from the job ready queue.
The device according to claim 14, wherein the task to be dispatched stored in the job ready queue carries the priority of the task to be dispatched; wherein the priority is set by a priority configuration instruction in the Redis service .
The device according to claim 15, wherein the priority configuration instruction includes a zset instruction or a list instruction.
The device according to claim 11, wherein the device further comprises a dispatch judgment module configured to:

Determine whether the task to be dispatched meets the preset dispatch conditions;

If the dispatch conditions are not met, the task to be dispatched is returned to the target tenant.
The device according to claim 17, wherein the preset dispatch condition comprises: the resource occupation of the task to be dispatched is less than or equal to the remaining available resources of the system, and the number of the task to be dispatched currently being dispatched The number is less than or equal to the number of remaining tasks to be dispatched in the system that can be dispatched at the same time.
The device according to claim 17, wherein the target tenant is preset with a dispatch record queue; the dispatch record queue is set to store the dispatched tasks of the target tenant and the resource occupancy of the dispatched tasks;

The dispatch judgment module is further configured to: if the dispatch condition is met, put the task to be dispatched into the dispatch record queue of the target tenant.
The device according to claim 11, wherein the device further comprises a dispatch task obtaining module, configured to:

If a task is obtained from the to-be-dispatched queue, the obtained task is put into the preset dispatched queue, so that the task executor obtains the task from the dispatched queue and executes the obtained task.
A task execution system, the system includes a plurality of task executors and a plurality of dispatch nodes; each of the dispatch nodes is in communication connection with the plurality of task executors;

The dispatch node is set to: obtain the task to be dispatched by the target tenant; wherein the target tenant corresponds to the dispatch node; and put the task to be dispatched into a preset queue to be dispatched for the dispatch A node or a dispatch node other than the dispatch node among the plurality of dispatch nodes dispatches the task to be dispatched to a task executor;

The task executor is set to execute the task to be dispatched.
A server comprising a processor and a memory, the memory storing machine executable instructions that can be executed by the processor, and the processor executing the machine executable instructions to implement any one of claims 1-10 The task dispatch method described above.
A machine-readable storage medium, the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement claim 1 -10 Any one of the task dispatching methods.