CN116501474B

CN116501474B - System, method and device for processing batch homogeneous tasks

Info

Publication number: CN116501474B
Application number: CN202310676088.8A
Authority: CN
Inventors: 杨非; 宋全恒; 杨清霖; 潘爱民
Original assignee: Zhejiang Lab
Current assignee: Zhejiang Lab
Priority date: 2023-06-08
Filing date: 2023-06-08
Publication date: 2023-09-22
Anticipated expiration: 2043-06-08
Also published as: CN116501474A

Abstract

The specification discloses a processing system, a method and a device for batch homogeneous tasks, wherein a scheduling node generates batch homogeneous tasks, the batch homogeneous tasks are issued to each computing cluster according to loads respectively corresponding to the computing clusters, and the batch homogeneous tasks are executed by each computing cluster. When the task quantity of the batch of homogenous tasks is too large, a plurality of computing clusters can be called to cooperatively process the same batch of homogenous tasks, so that the processing time of the batch of homogenous tasks is reduced, and the processing efficiency is ensured.

Description

System, method and device for processing batch homogeneous tasks

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a system, a method, and an apparatus for processing batch homogeneous tasks.

Background

At present, with the development of computer technology and the requirement of business fusion, a large number of homogeneous tasks need to be processed in a plurality of fields such as scientific research, finance, industrial manufacturing, digital treatment and the like. The homogenous tasks are tasks which have the same execution environment, the same business processes to be executed and are mutually independent.

In the prior art, a common batch homogeneous task processing method is realized based on a computing cluster. Specifically, the computing cluster includes a control center and a plurality of computing units. The control center can receive the batch of homogenous tasks, split the batch of homogenous tasks according to each computing unit in the computing cluster, and determine tasks to be executed respectively corresponding to each computing unit. And finally, for each computing unit, the control center calls the computing unit to execute the task to be executed corresponding to the computing unit, so as to complete the processing process of the batch of homogeneous tasks received by the control center.

However, when the number of received batch homogeneous tasks is large, the computing resources required for processing the batch homogeneous tasks are also large, which results in an increase in time required for processing the batch homogeneous tasks in the prior art, and further results in a decrease in processing efficiency.

Disclosure of Invention

The present disclosure provides a system, a method, and an apparatus for processing batch homogenous tasks, so as to partially solve the foregoing problems in the prior art.

The technical scheme adopted in the specification is as follows:

the present specification provides a system for processing batch homogenous tasks, the system comprising: the system comprises a scheduling node and a plurality of computing clusters, wherein the computing clusters comprise a plurality of computing units, and each computing unit provides the same batch processing service;

the scheduling node is used for responding to a task generation request and generating a plurality of batches of homogeneous tasks to be executed, wherein the batches of homogeneous tasks are tasks which have the same execution environment, the same service flow and mutual independence; determining tasks to be executed, which correspond to the computing clusters respectively, according to the load conditions and the batch homogeneous tasks, which correspond to the computing clusters respectively; issuing each task to be executed to a computing cluster corresponding to each task to be executed respectively;

The computing cluster is used for determining the load condition of the computing cluster according to the load conditions corresponding to each computing unit in the computing cluster, and sending the load condition of the computing cluster to the scheduling node; and receiving and executing the task to be executed, which is sent by the scheduling node, and obtaining an execution result of the task to be executed.

Optionally, the system further comprises: a storage node;

the storage node is used for receiving a task generation request sent by a user and storing the task generation request;

the scheduling node is used for acquiring a task generation request sent by the user from the storage node, and generating a plurality of batch homogeneous tasks to be executed according to task generation data carried in the task generation request.

Optionally, the scheduling node is configured to determine, according to the load conditions and the generated batch of homogeneous tasks corresponding to each computing cluster, the number of homogeneous tasks included in the tasks to be executed corresponding to each computing cluster; for each computing cluster, according to the number of homogeneous tasks contained in the tasks to be executed of the computing cluster, selecting the homogeneous tasks corresponding to the number from the undelivered homogeneous tasks as the tasks to be executed corresponding to the computing cluster.

Optionally, the system further comprises: a storage node;

the scheduling node is used for sending the identification of the computing cluster and the task to be executed of the computing cluster to the storage node;

the storage node is used for receiving the identification of the computing cluster and the task to be executed, which are sent by the scheduling node, determining the association relationship between the identification and the task to be executed, and storing the association relationship; receiving an acquisition instruction sent by a computing cluster, determining a task to be executed corresponding to the computing cluster according to an identification of the computing cluster carried in the acquisition instruction, and sending the task to be executed corresponding to the computing cluster.

The computing cluster is used for sending an acquisition instruction to the storage node and receiving a task to be executed sent by the storage node.

Optionally, the scheduling node is configured to determine load conditions corresponding to each computing unit included in each computing cluster, and determine, according to each load condition, a computing unit for executing the batch of homogenous tasks from each computing unit, as a designated unit; according to the load conditions respectively corresponding to the designated units and the batch homogeneous tasks, determining tasks to be executed respectively corresponding to the designated units, and according to the designated units respectively corresponding to the computing clusters, determining the tasks to be executed respectively corresponding to the computing clusters;

The computing cluster is used for receiving the task to be executed sent by the dispatching node, determining each appointed unit contained in the computing cluster, and dispatching each appointed unit to execute the task to be executed.

Optionally, for each homogeneous task in the batch of homogeneous tasks, the homogeneous task has a corresponding task execution state, and the task execution state includes one of non-start, executing success and executing failure;

the scheduling node is used for determining the task execution state of each homogeneous task in the batch of homogeneous tasks according to a preset time interval; and when the execution failure of the homogeneous task is determined according to the task execution state, updating the task execution state of the homogeneous task to be not started.

Optionally, the scheduling node is configured to determine, as a specified number, a number of computing units that execute the batch of homogenous tasks, and select, as a specified number, computing units that execute the batch of homogenous tasks from the computing units according to each load condition, where the number of specified units is the specified number.

Optionally, the scheduling node is configured to determine, from each computing unit, each designated unit for executing the batch of homogenous tasks, determine, for each homogenous task included in the batch of homogenous tasks, a number of computing units that need to execute the homogenous task, and if the number does not exceed a specific number, determine, from each designated unit, a designated unit that matches the homogenous task; and if the number exceeds the specified number, selecting the specified units from the specified units as the specified units for executing the homogeneous task.

Optionally, the scheduling node is configured to determine the number of computing units corresponding to a homogeneous task in the batch of homogeneous tasks, and determine, according to the load conditions respectively corresponding to each computing cluster, the batch of homogeneous tasks, and the number of computing units corresponding to the homogeneous task, the tasks to be executed respectively corresponding to each computing cluster, where the number of computing units corresponding to the homogeneous task is the number of computing units required for executing the homogeneous task.

The specification provides a method for processing batch homogeneous tasks, the method is applied to a scheduling node in a processing system of the batch homogeneous tasks, and the system comprises: the system comprises a scheduling node and a plurality of computing clusters, wherein the computing clusters comprise a plurality of computing units, and each computing unit provides the same batch processing service;

responding to a task generation request, and generating a plurality of batches of homogeneous tasks to be executed, wherein the batches of homogeneous tasks are tasks which have the same execution environment, the same service flow and mutual independence;

determining tasks to be executed, which correspond to the computing clusters respectively, according to the load conditions and the batch homogeneous tasks, which correspond to the computing clusters respectively;

And issuing each task to be executed to the computing clusters corresponding to each task to be executed respectively.

Optionally, the system comprises a storage node;

responding to a task generation request, generating a plurality of batches of homogeneous tasks to be executed, wherein the batch of homogeneous tasks specifically comprises the following steps:

sending a task acquisition request to the storage node;

receiving a task generation request returned by the storage node according to the task acquisition request, wherein the task generation request is sent by a user;

and determining task generation data carried in the task generation request, and generating a plurality of batches of homogeneous tasks to be executed according to the task generation data.

Optionally, determining the task to be executed corresponding to each computing cluster specifically includes:

determining the number of homogeneous tasks contained in the tasks to be executed, which correspond to the computing clusters respectively, according to the load conditions corresponding to the computing clusters and the generated batch of homogeneous tasks;

for each computing cluster, according to the number of homogeneous tasks contained in the tasks to be executed of the computing cluster, selecting the homogeneous tasks corresponding to the number from the undelivered homogeneous tasks as the tasks to be executed corresponding to the computing cluster.

determining the load conditions respectively corresponding to all the computing units contained in all the computing clusters, and determining the computing units for executing the batch homogeneous tasks from all the computing units according to the load conditions as a designated unit;

and determining tasks to be executed corresponding to the designated units according to the load conditions corresponding to the designated units and the batch homogeneous tasks, and determining the tasks to be executed corresponding to the calculation clusters according to the designated units corresponding to the calculation clusters.

the method further comprises the steps of:

determining the task execution state of each homogeneous task in the batch of homogeneous tasks according to a preset time interval;

and when the execution failure of the homogeneous task is determined according to the task execution state, updating the task execution state of the homogeneous task to be not started.

Optionally, from the computing units, determining a computing unit for executing the batch homogeneous task, as a specified unit, specifically includes:

and determining the number of computing units for executing the batch of homogenous tasks as a designated number, and selecting the computing units for executing the batch of homogenous tasks from the computing units according to each load condition as designated units, wherein the number of the designated units is the designated number.

determining each designated unit for executing the batch of homogenous tasks from each calculation unit, and determining the number of calculation units for executing each homogenous task contained in the batch of homogenous tasks;

if the number does not exceed the specific number, determining a designated unit matched with the homogeneous task from the designated units;

and if the number exceeds the specified number, selecting the specified units from the specified units as the specified units for executing the homogeneous task.

Optionally, each task to be executed is issued to a computing cluster corresponding to each task to be executed, which specifically includes:

The identification of the computing cluster and the task to be executed of the computing cluster are sent to the storage node;

determining a target cluster from the computing clusters;

according to the identification of the target cluster, a first instruction is sent to the storage node;

and receiving a task to be executed of the target cluster returned by the storage node, and issuing the task to be executed of the target cluster to the target cluster.

Optionally, determining, according to the load condition and the batch homogeneous task corresponding to each computing cluster, a task to be executed corresponding to each computing cluster, which specifically includes:

determining the number of computing units corresponding to the homogeneous tasks in the batch of homogeneous tasks;

determining tasks to be executed corresponding to each computing cluster according to the load conditions corresponding to each computing cluster, the batch of homogenous tasks and the number of computing units corresponding to the homogenous tasks;

the number of the computing units corresponding to the homogenous task is the number of the computing units required for executing the homogenous task.

The present specification provides a device for processing batch homogenous tasks, the device being applied to a scheduling node in a processing system of batch homogenous tasks, the batch homogenous task processing system comprising: the system comprises a scheduling node and a plurality of computing clusters, wherein the computing clusters comprise a plurality of computing units, and each computing unit provides the same batch processing task;

The request determining module is used for responding to the task generating request and generating a plurality of batches of homogeneous tasks to be executed, wherein the batches of homogeneous tasks are tasks which have the same execution environment, the same service flow and mutual independence;

the task determining module is used for determining tasks to be executed corresponding to the computing clusters according to the load conditions corresponding to the computing clusters and the batch homogeneous tasks;

and the issuing module is used for issuing each task to be executed to the computing clusters corresponding to each task to be executed respectively.

The present specification provides a computer readable storage medium storing a computer program which when executed by a processor implements a method of processing a batch of homogenous tasks as described above.

The present specification provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing a method of processing a batch of homogenous tasks as described above when executing the program.

The above-mentioned at least one technical scheme that this specification adopted can reach following beneficial effect:

and the dispatching node generates a batch of homogenous tasks, and issues the batch of homogenous tasks to each computing cluster according to loads corresponding to each computing cluster, and each computing cluster executes the batch of homogenous tasks. When the task quantity of the batch of homogenous tasks is too large, a plurality of computing clusters can be called to cooperatively process the same batch of homogenous tasks, so that the processing time of the batch of homogenous tasks is reduced, and the processing efficiency is ensured.

Drawings

The accompanying drawings, which are included to provide a further understanding of the specification, illustrate and explain the exemplary embodiments of the present specification and their description, are not intended to limit the specification unduly. In the drawings:

FIG. 1 is a schematic diagram of a batch homogeneous task processing system provided in the present specification;

FIG. 2 is a schematic diagram of a processing system provided herein;

FIG. 3 is a schematic flow diagram of a processing system executing batch processing tasks provided herein;

fig. 4 is a schematic scheduling flow diagram of a scheduling node provided in the present specification;

FIG. 5 is a schematic diagram of a scenario provided herein for determining a designated unit to perform a homogenous task;

FIG. 6 is a flow chart of a method for processing batch homogenous tasks provided in the present specification;

FIG. 7 is a schematic diagram of a device for processing batch homogenous tasks provided in the present specification;

fig. 8 is a schematic view of the electronic device corresponding to fig. 6 provided in the present specification.

Detailed Description

For the purposes of making the objects, technical solutions and advantages of the present specification more apparent, the technical solutions of the present specification will be clearly and completely described below with reference to specific embodiments of the present specification and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present specification. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are intended to be within the scope of the present disclosure.

The following describes in detail the technical solutions provided by the embodiments of the present specification with reference to the accompanying drawings.

Fig. 1 is a schematic structural diagram of a batch homogeneous task processing system provided in the present specification, where the batch homogeneous task processing system includes: a scheduling node and a number of computing clusters. The computing cluster comprises a plurality of computing units, and each computing unit provides the same batch processing service. That is, the processing system for batch homogenous tasks provided herein is essentially a system that schedules multiple computing clusters to process batch homogenous tasks. The computing units provide the same batch processing service, and each homogeneous task in the batch homogeneous tasks can be executed by any computing unit according to the configuration information of the batch homogeneous tasks.

From a hardware perspective, in one or more embodiments provided herein, a computing unit may also be referred to as a computing node. For each node in the system, the node may be one or more servers, or may be one or more intelligent devices.

From a software perspective, in one or more embodiments provided herein, each node in the system may be code running on a server, and the functions of each node are implemented by running the code of each node. For the computing clusters in the system, each computing cluster may provide the same functional batch service to the outside by executing the same code. In addition, in the present specification, codes of different nodes are independent from each other, and each node can communicate or data transmission with other nodes through a preset data interface.

From a hardware perspective, in one or more embodiments provided herein, each node runs on a different server.

In one or more embodiments provided herein, the processing system for the batch of homogenous tasks is composed of a single scheduling node and multiple computing clusters, and the batch of homogenous tasks is illustrated as a database query task or a model use task. The data to be queried is generally compared with each data stored in the database, so that the process of comparing the data to be queried with each data stored in the database can be a database query task, and the database query tasks generated between the data to be queried and each data stored in the database have the same execution environment, the same business flow and are mutually independent, thus being capable of being used as batch homogeneous tasks.

Likewise, after the model is brought online, the model is typically used to perform the business. In general, a plurality of pieces of data to be executed may be input into a model, and the model inputs execution results corresponding to the pieces of data to be executed to execute a service. Therefore, the process of determining the execution result of one piece of data to be executed through the model can be one model use task, the process of determining the execution result of a plurality of pieces of data to be executed through the model can be a plurality of model use tasks, and the execution environments among the model use tasks are the same, the business processes are the same and independent from each other, so that the model use tasks can be used as batch homogeneous tasks.

Of course, the database query task and the model use task are only examples of batch homogeneous tasks, and specifically, task types of the batch homogeneous tasks can be set according to needs, so that the conditions that execution environments are the same, business processes are the same and mutually independent are satisfied, which is not limited in the specification.

A scheduling node in the processing system of the batch of homogenous tasks may receive a task generation request. The task generating request can be sent by a user or can be automatically generated by a server for storing task generating data according to preset task generating conditions. The preset task generating condition may be at least one of completion of execution of the last batch of homogenous tasks, arrival of the current time at a preset time point, and monitoring of a specified operation performed by the user.

The task generation request can carry task generation data, so that the scheduling node can analyze the task generation request, determine the task generation data carried in the task generation request, and generate a plurality of batches of homogeneous tasks to be executed according to the task generation data. The task generating data may include task names of batch homogeneous tasks corresponding to the task generating data, task types corresponding to the batch homogeneous tasks, function identifiers of functions required to be called by executing the tasks, and the like.

In one or more embodiments provided in the present disclosure, after determining a batch of homogenous tasks, the scheduling node may split the batch of homogenous tasks into tasks to be executed corresponding to each computing cluster, where each computing cluster completes the execution process of the batch of homogenous tasks. For a computing cluster, the computing resources that the computing cluster can provide can be determined based on the load situation of the computing cluster. The scheduling node can split the batch of homogeneous tasks according to the load conditions respectively corresponding to the computing clusters.

Specifically, for each computing cluster, the scheduling node may send a load acquisition instruction to the computing cluster.

After the computing cluster acquires the load acquisition instruction, the load conditions corresponding to the computing units in the computing cluster can be determined, and the load conditions of the computing cluster can be determined according to the load conditions corresponding to the computing units. Then, after determining the load condition of the computing cluster, the computing cluster may return the load condition of the computing cluster according to the load acquisition instruction. The load condition may be the available computing resources in the computing cluster, or may be the ratio of the available computing resources in the computing cluster to the total computing resources.

The scheduling node may then receive the load conditions sent by the computing cluster. The scheduling node may determine the respective loading conditions for each computing cluster.

Then, the scheduling node can split the batch of homogeneous tasks according to the determined load conditions respectively corresponding to the computing clusters to obtain tasks to be executed respectively corresponding to the computing clusters. For each computing cluster, the task to be executed corresponding to the computing cluster is the homogeneous task to be executed by the computing cluster in the batch of homogeneous tasks.

The scheduling node may issue, for each computing cluster, the task to be executed of the computing cluster to the computing cluster after determining the task to be executed corresponding to each computing cluster.

Then, the computing cluster can receive the task to be executed issued by the scheduling node and execute the task to be executed to obtain an execution result of the task to be executed.

The computing cluster comprises a plurality of computing units. The computing cluster can split the tasks to be executed corresponding to the computing cluster according to the load conditions corresponding to the computing units in the computing cluster, determine the tasks to be executed corresponding to the computing units, and execute the tasks to be executed corresponding to the computing units according to the computing units.

In addition, in this specification, for one of the batch of homogeneous tasks, the homogeneous task may be performed by a single computing node or may be performed by multiple computing nodes. In the case where more computing resources are required for the homogeneous task, the computing resources provided by the plurality of computing nodes may satisfy the computing resources required for performing the homogeneous task. Thus, when splitting the batch of homogeneous tasks, it is necessary to consider not only the load situation of each computing cluster, but also the computing resources required to execute one of the batch of homogeneous tasks, which are provided by several computing units. I.e. a homogenous task has to be performed by several computing units.

Thus, the scheduling node may determine the number of computing units corresponding to one of the batch of homogenous tasks after generating the batch of homogenous tasks. The number of computing units corresponding to the homogeneous task may be determined according to an estimated ratio of computing resources required for executing the homogeneous task to computing resources that may be provided by each computing unit, or may be preset in a task generation request, and the number of computing units corresponding to one homogeneous task may be determined while a scheduling node generates batch homogeneous tasks according to the task generation request.

Then, the server can split the batch of homogeneous tasks into tasks to be executed corresponding to each computing cluster according to the load condition of each computing cluster, the number of the homogeneous tasks in the batch of homogeneous tasks and the number of computing units corresponding to each homogeneous task.

Based on the processing system of the batch homogeneous tasks shown in fig. 1, the scheduling node generates the batch homogeneous tasks, and issues the batch homogeneous tasks to each computing cluster according to loads respectively corresponding to each computing cluster, and each computing cluster executes the batch homogeneous tasks. When the task quantity of the batch of homogenous tasks is too large, a plurality of computing clusters can be called to cooperatively process the same batch of homogenous tasks, so that the processing time of the batch of homogenous tasks is reduced, and the processing efficiency is ensured.

Further, for the system in this specification, in order to guarantee high utilization of the system, the scheduling node in the system typically needs to process multiple batches of homogenous tasks. For the scheduling node, if more data is stored in the scheduling node, more or less computing resources in the scheduling node are occupied, thereby reducing the processing efficiency of the system. In order to avoid this, a storage node may also be provided in the system.

In the process of task processing by the system, the processing system may receive task generation requests sent by a plurality of users at the same time. The scheduling node may store the task processing request sent by the user in the storage node in case that the storage node is provided in the system, and acquire the task generating request from the storage node when the task processing request sent by the user satisfies the processing condition.

Thus, when a task generation request is received, the scheduling node may determine whether itself is in the process of executing a batch of homogenous tasks. If so, the scheduling node may forward the task generation request to the storage node.

The storage node may receive a task generation request sent by the scheduling node.

When the scheduling node determines that the aforementioned task generation request stored in the storage node satisfies the processing condition, the scheduling node may send a request acquisition instruction to the storage node. The processing condition may be that available computing resources in the computing cluster are more than a preset resource threshold, batch homogeneous tasks may be processed for the computing cluster, a preset processing time may be reached, available computing resources in the computing cluster are enough to execute tasks corresponding to the task generation request, and the like. The processing conditions may be specifically set as needed, and this is not limited in this specification.

The storage node may determine a target generation request from among the task generation requests stored by itself according to the request acquisition request, and send the target generation request to the scheduling node.

The scheduling node may generate a batch of homogenous tasks based on the received target generation request. The target generation request may be any one of the task generation requests randomly selected from the task generation requests by the storage node, or may be a task generation request with the earliest transmission time selected from the task generation requests by the storage node according to the transmission time of the task generation requests.

Of course, the scheduling node may also pull the task generation request from the storage node after determining that the execution of the batch of homogenous tasks by itself is complete. The data transmission mode between the scheduling node and the storage node can be set according to the requirement, and the specification does not limit the data transmission mode.

Further, in this specification, each task generation request may have a corresponding priority, and when the scheduling node receives the task generation request, the scheduling node may further determine the priority of the task generation request being executed by itself as the first priority, and the priority of the received task generation request as the second priority. And if the first priority is higher than the second priority, continuing to execute the task generation request corresponding to the first priority, and if the second priority is higher than the first priority, executing the task generation request corresponding to the second priority. The priority may be carried in the task generation request or may be determined according to the generation time of each task generation request, and specifically how to determine the priority may be set according to needs, which is not limited in this specification.

Or, continuing to execute the task generation request corresponding to the first priority, storing the received task generation request and the second priority corresponding to the task generation request in a storage node, and determining that the corresponding task generation request needs to be performed from the storage node according to the priorities corresponding to the task generation requests respectively after generating batch homogeneous tasks according to the task generation request corresponding to the first priority. Specifically, how the scheduling node processes each task generation request based on the priority, or how the scheduling node reasonably schedules each task generation request, which can be set according to the needs, and the specification does not limit the task generation request.

Based on the same thought, the specification provides a schematic structural diagram of a batch homogeneous task processing system. As shown in fig. 2.

Fig. 2 is a schematic structural diagram of a processing system provided in the present specification. In the figure, the system comprises a scheduling node, three computing clusters and a storage node. The scheduling node is used for scheduling a plurality of computing clusters to execute batch homogeneous tasks, the computing clusters are used for executing the batch homogeneous tasks, and the storage node is used for storing execution results. The user may send a task generation request to the scheduling node. The scheduling node can receive the task generation request, generate batch homogeneous tasks according to the task generation request when determining that the task generation request can be executed, and call a plurality of computing clusters to process the batch homogeneous tasks. When the scheduling node determines that the task generation request cannot be executed, the task generation request can be stored in a storage node, when the task generation request can be executed, the task generation request is acquired from the storage node, then batch homogeneous tasks are generated according to the task generation request, and a plurality of computing clusters are called to process the batch homogeneous tasks.

Based on the same thought, the specification provides a schematic structural diagram of a batch homogeneous task processing system. As shown in fig. 3.

FIG. 3 is a flow chart of a processing system for performing batch processing tasks provided herein. In the figure, the system comprises a scheduling node, three computing clusters and a storage node. The scheduling node can acquire a task generation request sent by a user from the storage node, analyze the task generation request to obtain task generation data, determine batch homogeneous tasks according to the task generation data, split the batch homogeneous tasks into tasks to be executed respectively corresponding to the computing clusters according to the load conditions respectively corresponding to the computing clusters, and issue the tasks.

In addition, for each computing cluster, when determining the task to be executed corresponding to the computing cluster, the scheduling node may determine the serial numbers corresponding to each task in the batch of homogeneous tasks generated by the scheduling node. And determining the number of the homogeneous task contained in the task to be executed, which corresponds to each computing cluster, as each designated number according to the load corresponding to each computing cluster and the number corresponding to each task in the batch of homogeneous tasks, and then issuing the homogeneous task of each designated number to each computing cluster.

Taking 3000 tasks in the batch of homogeneous tasks, and the serial numbers corresponding to the homogeneous tasks contained in the batch of homogeneous tasks are 1-3000 as an example, it is assumed that two computing clusters exist in the system: and if the load condition of the computing cluster A is: the unused computing resources are 200MB, and the load situation of computing cluster B is: the unused computing resources are 100MB. The scheduling node may determine, according to the load conditions of the computing cluster a and the computing cluster B and the numbers corresponding to the tasks included in the batch of homogeneous tasks, a designated number corresponding to the computing cluster a: 1-2000, and the assigned number corresponding to the computing cluster B: 2001-3000.

Of course, in the above manner of determining the tasks to be executed corresponding to each computing cluster based on the numbers, the numbers corresponding to each homogeneous task in the batch of homogeneous tasks are also determined, and the step of determining the numbers has the same requirement on computing resources, so that computing resources in the scheduling nodes are occupied, and task processing efficiency is reduced. In order to avoid the above situation, the scheduling node may further determine, for each computing cluster, a number of homogeneous tasks included in the tasks to be executed corresponding to the computing cluster, and determine, based on the number, the tasks to be executed of the computing cluster.

Specifically, the scheduling node may determine the load conditions corresponding to each computing cluster, and determine, according to the load conditions corresponding to each computing cluster and the generated batch of homogenous tasks, the number of homogenous tasks included in the tasks to be executed corresponding to each computing cluster.

Then, the scheduling node can determine each undelivered homogeneous task, and for each computing cluster, according to the number of homogeneous tasks contained in the tasks to be executed corresponding to the computing cluster, select, from each homogeneous task undelivered by the scheduling node, the homogeneous task corresponding to the number as the task to be executed corresponding to the computing cluster.

The scheduling node may issue, for each computing cluster, a task to be executed corresponding to the computing cluster.

In this specification, as previously described, the processing system may process multiple batches of homogenous tasks simultaneously. For a computing cluster, only one batch of homogenous tasks can be executed in the same period of time by one computing cluster, but for a scheduling node, one scheduling node can respond to different task generation requests sent by a plurality of users at the same time and issue each batch of homogenous tasks to different computing clusters. In this case, if the tasks to be executed and the related data thereof are stored in the scheduling node after determining the tasks to be executed respectively corresponding to the computing clusters, it is unavoidable that the computing resources in the scheduling node are occupied, resulting in a situation that the processing efficiency of the processing system is low. In order to avoid the occurrence of the above situation, the scheduling node may further store tasks to be executed corresponding to each computing cluster in the storage node.

Specifically, the scheduling node may send, for each computing cluster, an identification of the computing cluster and a task to be performed by the computing cluster to a storage node in the system.

The storage node can receive the identification of the computing cluster and the task to be executed, which are sent by the scheduling node, determine the association relationship between the identification and the task to be executed, store the association relationship between the identification and the task to be executed, and store the received task to be executed.

After the storage node stores the content, the scheduling node may determine a target cluster from the computing clusters. The target cluster may be a computing cluster in which the previous batch of homogeneous tasks is processed, or may be a computing cluster in which the idle computing resources reach a preset resource threshold. In particular, how the target cluster is determined may be set according to needs, which is not limited in this specification.

The scheduling node may send a first instruction to the storage node according to the identification of the target cluster after determining the target cluster. The first instruction is used for determining a task to be executed of the target cluster.

The storage node may receive the first instruction, determine a task to be executed of the target cluster according to the identifier of the target cluster carried in the first instruction, and return the task to be executed of the target cluster to the scheduling node.

The scheduling node may receive the task to be executed of the target cluster sent by the storage node, and issue the task to be executed of the target cluster to the target cluster.

The target cluster may receive the task to be executed issued by the storage node, and store the execution result of the task to be executed to the storage node.

It should be noted that, the storage node may not only store the task generation request sent by the user, the tasks to be executed corresponding to each computing cluster, and the execution results corresponding to each task to be executed, but also store the code or function required to be executed for executing each batch of homogenous tasks, that is, the processing script of the batch of homogenous tasks. The processing script may be a code that provides an execution environment for the homogenous task, or may be a code that is executed to obtain an execution result of the homogenous task. When the scheduling node generates a batch of homogeneous tasks according to the task generation request, the processing script of the batch of homogeneous tasks can be determined from the processing scripts stored in the storage node, and the tasks to be executed respectively corresponding to the computing clusters are determined according to the processing script, the load condition of each computing cluster, the number of homogeneous tasks in the batch of homogeneous tasks, the number of computing nodes corresponding to one homogeneous task, and the like. In particular, how the computing cluster processes the task to be executed, how the content carried in the task to be executed, how the task to be executed is generated, and the purpose of the processing script stored in the storage node may be set according to the needs, which is not limited in this specification.

Further, in this specification, a situation that a certain computing cluster is down in the task execution process may also occur, so as to avoid the occurrence of the foregoing situation, in this specification, the scheduling node may further monitor each computing cluster.

Specifically, the scheduling node may send, for each computing cluster, a load acquisition request to the computing cluster according to a preset first time interval.

The computing cluster may determine its own load condition according to the load acquisition request and return its own load condition according to the load acquisition request.

The scheduling node may receive the load conditions sent by each computing cluster, and monitor each computing cluster according to the load conditions.

Furthermore, when a certain computing cluster is down in the task execution process, the scheduling node needs to allocate each task executed by the computing cluster that is down to other computing clusters in time for processing. Therefore, in this specification, the scheduling node may also detect execution progress of each task included in the batch of homogenous tasks, which corresponds to each task.

Specifically, for each computing cluster, the scheduling node may send a progress acquisition request to the computing cluster according to a preset second time interval.

The computing cluster can determine the execution progress of the task to be processed according to the progress acquisition request, and return the execution progress of the task to be processed to the scheduling node according to the progress acquisition request. The execution progress of the task to be processed may be each homogeneous task that is executed by the computing cluster and may be each homogeneous task that is not executed by the computing cluster, and in particular, how the execution progress is determined and how the execution progress is characterized may be set according to needs, which is not limited in this specification.

In addition, in this specification, a case may occur in which execution of a certain homogeneous task fails. The goal of the processing system is to process all of the homogeneous tasks contained in the batch of homogeneous tasks successfully. Thus, the scheduling node may also determine, for each homogeneous task in the batch of homogeneous tasks, the execution status of that homogeneous task, i.e., not started, executing successfully, executing failed, etc. The scheduling node may then determine that the execution state is a failed execution homogenous task from among the homogenous tasks, and update the state of the failed execution homogenous task to not started. The computing unit for executing the homogenous task may then re-execute the homogenous task according to the state of the homogenous task.

Based on the same thought, the specification provides a scheduling flow diagram of a scheduling node. As shown in fig. 4.

Fig. 4 is a schematic scheduling flow diagram of a scheduling node provided in the present specification. The scheduling node can generate data according to the tasks to generate batch homogeneous tasks. After the batch of homogeneous tasks is generated, determining each homogeneous task which is not issued in the batch of homogeneous tasks stored by the user. And further determining the designated clusters according to the load conditions corresponding to the computing clusters. Wherein the designated cluster is a computing cluster for performing the batch of homogenous tasks.

Then, the scheduling node may determine a task to be executed of each designated cluster from the undelivered homogenous tasks, and issue the task to be executed of the designated cluster to each designated cluster.

Finally, the scheduling node may determine an execution state of each homogeneous task in the tasks to be executed of the designated cluster.

When the homogeneous task is successfully executed, the scheduling node can release the resources occupied by the homogeneous task. When the homogeneous task is not successfully executed, if the homogeneous task fails to be executed, the scheduling node can update the state of the homogeneous task to be not started, and if the homogeneous task is not successfully executed and fails to be executed, i.e. the homogeneous task is in a non-starting state or an executing state, the scheduling node can skip the homogeneous task and continuously judge whether the next homogeneous task is successfully executed. When the state of the homogeneous task is updated to be not started by the scheduling node, the computing node executing the homogeneous task can be continuously called to re-execute the homogeneous task, and the homogeneous task can be added to the undelivered homogeneous task in the batch of homogeneous tasks to be continuously issued.

Of course, if a certain homogeneous task fails to execute more times, there may be an error in the data in the homogeneous task. Therefore, the scheduling node can record the number of times of execution failure of the homogeneous task for each homogeneous task, and when the number of times reaches a preset number of times threshold, the scheduling node sends prompt information according to the identification of the homogeneous task, wherein the prompt information is used for prompting that the number of times of execution failure of the homogeneous task reaches the preset number of times threshold.

In addition, in actual operation, each batch of homogenous tasks may be performed by a computing unit in a computing cluster, one computing unit may perform only one batch of homogenous tasks within the same period, and for one computing cluster, the computing cluster may be used to perform multiple batches of homogenous tasks within the same period. If the scheduling node stores the tasks to be executed corresponding to each computing unit, the scheduling node may have a higher requirement on computing resources, and if the scheduling node has fewer computing resources, a situation that the scheduling node is down may occur. Therefore, in order to avoid the occurrence of the above situation, the scheduling node may further determine tasks to be executed corresponding to each computing node, and store the tasks to be executed corresponding to each computing unit in the storage node.

Specifically, for each computing node, the scheduling node may determine a load condition of the computing node, and match each computing node with each homogeneous task according to the load condition of the computing node and the number of computing nodes corresponding to the homogeneous tasks in each batch of homogeneous tasks generated by the scheduling node, so as to determine the homogeneous tasks and the computing nodes that are matched with each other. Wherein for each matching homoplasmic task and computing node, the number of homoplasmic tasks may be one, the number of computing nodes may be one or more, that is, one computing node can process only one homoplasmic task in the same period, and one homoplasmic task can be processed by a plurality of computing nodes.

Thus, for each computing node, the scheduling node may determine the computing node's task to be performed based on the homogenous tasks that match the computing node.

The scheduling node may send, for each computing node, the identity of the computing node and the task to be performed by the computing node to a storage node in the system after determining the task to be performed by the computing node.

The storage node can receive the identification of the computing node and the task to be executed, which are sent by the scheduling node, determine the association relationship between the identification and the task to be executed, store the association relationship between the identification and the task to be executed, and store the received task to be executed.

After the storage node stores the content, the scheduling node may determine the target node from the computing nodes. The target node may be a computing node that has completed the previous batch of homogeneous tasks among the computing nodes, or may be a computing node in which the idle computing resources reach a preset resource threshold. In particular, how the target node determines that the target node can be set according to needs, which is not limited in this specification.

The scheduling node may send a second instruction to the storage node based on the identity of the target node after determining the target node. Wherein the second instruction is for determining a task to be performed by the target node.

The storage node may receive the second instruction, determine a task to be executed of the target node according to the identifier of the target node carried in the second instruction, and return the task to be executed of the target node to the scheduling node.

The scheduling node may receive the task to be executed of the target node sent by the storage node, and issue the task to be executed of the target node to the target node.

The target node in the computing cluster may receive the task to be executed issued by the storage node, and store the execution result of the task to be executed in the storage node.

In this specification, for each computing cluster, the computing cluster typically includes a plurality of computing units, and each computing unit may process a homogenous task that is not exactly the same. Thus, when monitoring the progress of execution of each homogenous task, the scheduling node may send a progress acquisition request directly to each computing unit for that computing unit.

The computing unit may determine, according to the progress acquisition request, an execution progress of the task to be executed, where the execution progress may be a progress of each of the homogeneous tasks that have been processed by the computing unit, or may be a progress of each of the homogeneous tasks that has not been processed.

The scheduling node may receive execution progress of each computing unit respectively corresponding thereto. Based on the above manner, when there is a situation that the computing unit or the computing cluster is down, the scheduling node may determine that the down computing unit or the computing cluster does not execute the completed homogeneous tasks as specific tasks. And determining a computing cluster for executing the specific task from the computing clusters contained in the system, and issuing the specific task to the computing cluster for executing the specific task.

The computing cluster performing the particular task may receive the particular task and execute the particular task.

Further, for a computing unit or a computing cluster, if the load of the computing unit or the computing cluster is too high, the computing unit or the computing cluster may be down. Therefore, in order to avoid the occurrence of the above situation, when determining the tasks to be executed corresponding to each computing cluster, the scheduling node may determine only the tasks to be executed corresponding to the computing clusters for executing the batch of homogenous tasks.

Specifically, after determining the load conditions corresponding to the computing clusters, the scheduling node may determine, according to a preset load threshold, each computing cluster whose load condition does not exceed the load threshold, as a computing cluster for executing the batch of homogenous tasks. Wherein, the load threshold may be: more than 80% of the computing resources have been used, more than 90% of the computing resources have been used, etc. Each computing cluster for which the load condition does not exceed the load threshold is used to characterize a computing cluster for which there are more computing resources remaining.

Then, the scheduling node can determine the tasks to be executed corresponding to the computing clusters for executing the batch of homogenous tasks according to the load conditions corresponding to the computing clusters for executing the batch of homogenous tasks.

Furthermore, the scheduling node may determine, for each computing unit, a computing unit for executing the batch of homogenous tasks according to a load condition of the computing unit, and further determine, based on the determined computing units for executing the batch of homogenous tasks, tasks to be executed corresponding to each computing cluster, respectively.

Specifically, the scheduling node may determine a load condition corresponding to each computing unit included in each computing cluster.

The scheduling node may then determine, from among the computing units, a computing unit for performing the batch of homogenous tasks as a designated unit, based on each load situation. Wherein the determined specified units have more available computing resources. Or, less computing resources have been used in the specified unit.

Therefore, the scheduling node can determine the tasks to be executed corresponding to the designated units according to the load conditions corresponding to the designated units and the batch of homogeneous tasks. And then the tasks to be executed corresponding to the designated units are issued to the designated units for execution, so that the processing process of batch homogeneous tasks is completed.

In addition, in this specification, the data amount of each batch of homogenous tasks is different, and thus, the number of computing units or computing clusters for executing the batch of homogenous tasks is also different for each batch of homogenous tasks. Thus, the scheduling node may also determine the number of computing units performing the batch of homogenous tasks as the specified number, that is, the specified number of computing units may be used to perform the batch of homogenous tasks. Meanwhile, in order to avoid the occurrence of downtime of the computing units, the scheduling node can also select the computing unit for executing the batch of homogeneous tasks from the computing units as a designated unit according to the load condition of each computing unit. Wherein the number of the specified units is the specified number.

Further, for each homogeneous task included in the batch of homogeneous tasks, if the homogeneous task is more complex, multiple computing units may be required to perform the homogeneous task, and if the homogeneous task is simpler, only one computing unit or a part of computing resources in one computing unit may be required to perform the homogeneous task. Thus, the scheduling node may also determine, for each homogeneous task, a designated unit for performing that homogeneous task.

Specifically, the scheduling node may determine, for each homogeneous task included in the batch of homogeneous tasks, a number of computing units that need to execute the homogeneous task.

If the number does not exceed the specified number, the scheduling node may determine from among the computing units each of the specified units for performing batch homogeneity and determine from among the specified units the specified unit that matches the homogeneity task. That is, from among the specified units available to perform the batch of homogenous tasks, the specified unit for performing the homogenous task is selected. The specific number may be 1, 2, etc., and the specific number of the specific number may be set as needed, which is not limited in this specification.

If the number exceeds a specified number, the scheduling node may select the number of computing units from among the computing units as the specified unit to perform the homogenous task. Of course, the scheduling node may also select the number of designated units as designated units for performing the homogenous task.

Further, a computing cluster typically includes a large number of computing units, and for convenience of management, the computing units are typically managed in partitions in the cluster. That is, a computing cluster contains several partitions, each containing one or more computing nodes. When the number of computing units needed to perform the homogenous task exceeds a specified number, the scheduling node also needs to determine the partition containing the specified number of units as the partition performing the homogenous task.

Specifically, for each partition, the scheduling node may determine the number of specified units contained in the partition.

The scheduling node may then determine whether the number of specified units contained in the partition exceeds the number of computing units that specify a homogenous task.

If so, the scheduling node may determine that the partition is a partition for executing the homogenous task, and issue the homogenous task to each designated unit included in the partition.

If not, the scheduling node may determine that the partition is not available for executing the homogenous task.

Based on the same thought, the present specification provides a schematic view of a scenario for determining a designated unit for performing a homogenous task. As shown in fig. 5.

FIG. 5 is a schematic view of a scenario provided herein in which a dispatch node may determine a designated unit to execute a homogenous task from among computing points and, for each homogenous task, determine the number of computing units to execute the homogenous task.

The scheduling node may then determine whether the number is less than a preset number threshold. Wherein the number threshold may be 1.

If so, a threshold number of computational units may be characterized as sufficient to perform the homogenous task. The scheduling node may match the homogenous task with each of the designated units, determine the designated unit that matches the homogenous task, and issue the homogenous task to the designated unit.

If not, the calculation unit, which may characterize the number threshold, is insufficient to perform the homogenous task. The scheduling node may then determine the partition for performing the homogenous task.

The scheduling node may determine, for each partition, a number of designated units contained in the partition, and when the number of designated units contained in the partition is not less than a number of computing units performing the homogenous task, determine that the partition is a partition for performing the homogenous task, and issue the homogenous task to each designated unit contained in the partition.

When the partition contains a number of specified units that is less than the number of computing units that perform the homogenous task, the scheduling node may determine that the partition is not a partition that performs the homogenous task.

Based on the same thought, the specification provides a flow diagram of a method for processing batch homogenous tasks. As shown in fig. 6, wherein:

s100: and generating a plurality of batches of homogeneous tasks to be executed in response to the task generation request, wherein the batches of homogeneous tasks are tasks which have the same execution environment, the same service flow and mutual independence.

In one or more embodiments provided herein, the method is applied to a scheduling node in a processing system for batch homogenous tasks, the system comprising: the system comprises a scheduling node and a plurality of computing clusters, wherein the computing clusters comprise a plurality of computing units, and each computing unit provides the same batch processing service.

The scheduling node may respond to the task generation request and determine a batch of homogenous tasks based on the task generation request.

S102: and determining the tasks to be executed corresponding to each computing cluster according to the load conditions corresponding to each computing cluster and the batch homogeneous tasks.

S104: and issuing each task to be executed to the computing clusters corresponding to each task to be executed respectively.

In one or more embodiments provided in the present specification, specific real-time steps of the method for processing a batch of homogenous tasks may refer to descriptions of scheduling nodes in the above system, which is not described herein.

Based on the processing method of the batch homogeneous task shown in fig. 6, the batch homogeneous task is generated according to the task generation request, and the batch homogeneous task is issued to each computing cluster according to the loads respectively corresponding to each computing cluster, and the batch homogeneous task is executed by each computing cluster. When the task quantity of the batch of homogenous tasks is too large, a plurality of computing clusters can be called to cooperatively process the same batch of homogenous tasks, so that the processing time of the batch of homogenous tasks is reduced, and the processing efficiency is ensured.

In addition, the system always comprises a storage node, the storage node can be used for storing a task generation request sent by a user, and then the scheduling node can send a task acquisition request to the storage node and receive the task generation request returned by the storage node according to the task acquisition request. Wherein the task generation request is sent for the user. The scheduling node can then determine task generation data carried in the task generation request and generate a batch of homogenous tasks to be executed according to the task generation data.

Further, the scheduling node may determine, according to the load condition and the generated batch of homogenous tasks corresponding to each computing cluster, the number of homogenous tasks included in the tasks to be executed corresponding to each computing cluster, and then select, for each computing cluster, from the respective homogenous tasks that are not issued, the homogenous task corresponding to the number as the task to be executed corresponding to the computing cluster according to the number of homogenous tasks included in the tasks to be executed of the computing cluster.

Further, the scheduling node may determine load conditions corresponding to each computing unit included in each computing cluster, determine, from each computing unit, a computing unit for executing the batch of homogeneous tasks according to each load condition, as an appointed unit, determine, according to load conditions corresponding to each appointed unit and the batch of homogeneous tasks, tasks to be executed corresponding to each appointed unit, and determine, according to appointed units corresponding to each computing cluster, tasks to be executed corresponding to each computing cluster.

In addition, for each homogeneous task in the batch of homogeneous tasks, the homogeneous task has its corresponding task execution state, the task execution state including one of inactive, executing success, and executing failure. Then, the server may determine, for each homogeneous task in the batch of homogeneous tasks, a task execution state of the homogeneous task according to a preset time interval, and update the task execution state of the homogeneous task to be not started when determining that the homogeneous task fails to execute according to the task execution state.

Further, the scheduling node may determine the number of computing units performing the batch of homogeneous tasks as a specified number, and select, from among the computing units, the computing units for performing the batch of homogeneous tasks as a specified number according to each load condition, the specified number of the computing units.

Further, the scheduling node may determine, from among the computing units, each designated unit for executing the batch of homogenous tasks, and determine, for each homogenous task included in the batch of homogenous tasks, a number of computing units that need to execute the homogenous task.

If the number does not exceed the specific number, the scheduling node may determine a designated unit matching the homogeneous task from the designated units.

If the number exceeds the specified number, the number of specified units is selected from the specified units as the specified units for executing the homogenous task.

It should be noted that, the method for processing the batch of homogenous tasks is applied to the scheduling unit in the system for processing the batch of homogenous tasks, so the specific execution process of the method for processing the batch of homogenous tasks can be referred to the description of the scheduling unit in the system for processing the batch of homogenous tasks, which is not repeated herein.

Based on the same thought, the present disclosure also provides a device for processing batch homogenous tasks, as shown in fig. 7.

Fig. 7 is a schematic structural diagram of a device for processing batch homogenous tasks provided in the present specification, where the device is applied to a scheduling node in a batch homogenous task processing system, and the batch homogenous task processing system includes: the system comprises a scheduling node and a plurality of computing clusters, wherein the computing clusters comprise a plurality of computing units, and each computing unit provides the same batch processing task; wherein:

the request determining module 200 is configured to generate, in response to a task generating request, a plurality of batch homogeneous tasks to be executed, where the batch homogeneous tasks are tasks that have the same execution environment, the same service flow, and are independent from each other.

The task determining module 202 is configured to determine, according to the load conditions and the batch homogeneous tasks respectively corresponding to the computing clusters, tasks to be executed respectively corresponding to the computing clusters.

And the issuing module 204 is configured to issue each task to be executed to a computing cluster corresponding to each task to be executed.

Optionally, the system includes a storage node, a request determining module 200, configured to send a task acquisition request to the storage node, receive a task generation request returned by the storage node according to the task acquisition request, where the task generation request is sent by a user, determine task generation data carried in the task generation request, and generate a plurality of batch homogeneous tasks to be executed according to the task generation data.

Optionally, the task determining module 202 is configured to determine, according to the load condition and the generated batch of homogenous tasks corresponding to each computing cluster, the number of homogenous tasks included in the tasks to be executed corresponding to each computing cluster, and for each computing cluster, select, according to the number of homogenous tasks included in the tasks to be executed of the computing cluster, from the respective homogenous tasks that are not issued, a homogenous task corresponding to the number as a task to be executed corresponding to the computing cluster.

Optionally, the task determining module 202 is configured to determine load conditions corresponding to each computing unit included in each computing cluster, determine, according to each load condition, computing units for executing the batch of homogeneous tasks from each computing unit, as designated units, determine, according to load conditions corresponding to each designated unit and the batch of homogeneous tasks, tasks to be executed corresponding to each designated unit, and determine, according to designated units corresponding to each computing cluster, tasks to be executed corresponding to each computing cluster.

Optionally, for each homogeneous task in the batch of homogeneous tasks, the homogeneous task has a corresponding task execution state, where the task execution state includes one of non-startup, in-execution, execution success and execution failure, and the task determining module 202 is configured to determine, for each homogeneous task in the batch of homogeneous tasks, a task execution state of the homogeneous task according to a preset time interval, and update the task execution state of the homogeneous task to non-startup when it is determined that the execution of the homogeneous task fails according to the task execution state.

Optionally, the task determining module 202 is configured to determine, as a specified number, a number of computing units that perform the batch of homogenous tasks, and select, as a specified number, computing units that perform the batch of homogenous tasks from the computing units according to each load situation, where the number of specified units is the specified number.

Optionally, the task determining module 202 is configured to determine, from among the computing units, each of the designated units for executing the batch of homogenous tasks, determine, for each homogenous task included in the batch of homogenous tasks, a number of computing units that need to execute the homogenous task, determine, from among the designated units, a designated unit that matches the homogenous task if the number does not exceed a specific number, and select, from among the designated units, the designated unit of the number as the designated unit that executes the homogenous task if the number exceeds the designated number.

Optionally, the issuing module 204 is configured to send the identifier of the computing cluster and the task to be executed of the computing cluster to the storage node, determine a target cluster from the computing clusters, send a first instruction to the storage node according to the identifier of the target cluster, receive the task to be executed of the target cluster returned by the storage node, and issue the task to be executed of the target cluster to the target cluster.

Optionally, the task determining module 202 is configured to determine the number of computing units corresponding to a homogeneous task in the batch of homogeneous tasks, and determine, according to the load conditions respectively corresponding to each computing cluster, the batch of homogeneous tasks, and the number of computing units corresponding to the homogeneous task, the tasks respectively corresponding to each computing cluster to be executed, where the number of computing units corresponding to the homogeneous task is the number of computing units required for executing the homogeneous task.

The present specification also provides a computer readable storage medium storing a computer program operable to perform a method of processing a batch of homogenous tasks as provided in fig. 1 above.

The present specification also provides a schematic structural diagram of the electronic device shown in fig. 8. At the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile storage, as illustrated in fig. 8, although other hardware required by other services may be included. The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs to implement a method for processing batch homogeneous tasks as described above with respect to fig. 6. Of course, other implementations, such as logic devices or combinations of hardware and software, are not excluded from the present description, that is, the execution subject of the following processing flows is not limited to each logic unit, but may be hardware or logic devices.

In the 90 s of the 20 th century, improvements to one technology could clearly be distinguished as improvements in hardware (e.g., improvements to circuit structures such as diodes, transistors, switches, etc.) or software (improvements to the process flow). However, with the development of technology, many improvements of the current method flows can be regarded as direct improvements of hardware circuit structures. Designers almost always obtain corresponding hardware circuit structures by programming improved method flows into hardware circuits. Therefore, an improvement of a method flow cannot be said to be realized by a hardware entity module. For example, a programmable logic device (Programmable Logic Device, PLD) (e.g., field programmable gate array (Field Programmable Gate Array, FPGA)) is an integrated circuit whose logic function is determined by the programming of the device by a user. A designer programs to "integrate" a digital system onto a PLD without requiring the chip manufacturer to design and fabricate application-specific integrated circuit chips. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, such programming is mostly implemented by using "logic compiler" software, which is similar to the software compiler used in program development and writing, and the original code before the compiling is also written in a specific programming language, which is called hardware description language (Hardware Description Language, HDL), but not just one of the hdds, but a plurality of kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), lava, lola, myHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog are currently most commonly used. It will also be apparent to those skilled in the art that a hardware circuit implementing the logic method flow can be readily obtained by merely slightly programming the method flow into an integrated circuit using several of the hardware description languages described above.

The controller may be implemented in any suitable manner, for example, the controller may take the form of, for example, a microprocessor or processor and a computer readable medium storing computer readable program code (e.g., software or firmware) executable by the (micro) processor, logic gates, switches, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable logic controllers, and embedded microcontrollers, examples of which include, but are not limited to, the following microcontrollers: ARC 625D, atmel AT91SAM, microchip PIC18F26K20, and Silicone Labs C8051F320, the memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also appreciate that, in addition to implementing the controller in a pure computer readable program code, it is well possible to implement the same functionality by logically programming the method steps such that the controller is in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Such a controller may thus be regarded as a kind of hardware component, and means for performing various functions included therein may also be regarded as structures within the hardware component. Or even means for achieving the various functions may be regarded as either software modules implementing the methods or structures within hardware components.

The system, apparatus, module or unit set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. One typical implementation is a computer. In particular, the computer may be, for example, a personal computer, a laptop computer, a cellular telephone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

For convenience of description, the above devices are described as being functionally divided into various units, respectively. Of course, the functions of each element may be implemented in one or more software and/or hardware elements when implemented in the present specification.

It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the present specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present description is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the specification. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable lesion detection device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable lesion detection device, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable lesion detection device to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

It will be appreciated by those skilled in the art that embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, the present specification may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The description may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, as relevant to see a section of the description of method embodiments.

The foregoing is merely exemplary of the present disclosure and is not intended to limit the disclosure. Various modifications and alterations to this specification will become apparent to those skilled in the art. Any modifications, equivalent substitutions, improvements, or the like, which are within the spirit and principles of the present description, are intended to be included within the scope of the claims of the present description.

Claims

1. A system for processing a batch of homogenous tasks, the system comprising: the system comprises a scheduling node and a plurality of computing clusters, wherein the computing clusters comprise a plurality of computing units, and each computing unit provides the same batch processing service; for each computing cluster, the computing cluster comprising a number of partitions, each partition comprising one or more computing units;

the scheduling node is used for receiving task generation requests sent by a plurality of users, generating requests for each task, and responding to the task generation requests to generate a plurality of batches of homogeneous tasks to be executed, wherein the batches of homogeneous tasks are tasks with the same execution environment, the same service flow and mutual independence; determining the load conditions corresponding to the computing units contained in each computing cluster, and determining the computing units used for executing the batch homogeneous task from the computing units according to the load conditions as a designated unit; determining the number of computing units required for executing the homogeneous task for each homogeneous task contained in the batch of homogeneous tasks, determining the number of designated units contained in each partition for each partition, and taking the partition as the partition for executing the homogeneous task and taking the homogeneous task as a task to be executed of each designated unit in the partition if the number of computing units for executing the homogeneous task does not exceed the number of designated units contained in the partition; determining tasks to be executed, which correspond to each computing cluster respectively, according to the designated units, which correspond to each computing cluster respectively; issuing each task to be executed to a computing cluster corresponding to each task to be executed respectively;

The computing cluster is used for determining the load condition of the computing cluster according to the load conditions corresponding to each computing unit in the computing cluster, and sending the load condition of the computing cluster to the scheduling node; and receiving the task to be executed sent by the dispatching node, determining each appointed unit contained in the computing cluster, and dispatching each appointed unit to execute the task to be executed to obtain an execution result of the task to be executed.

2. The system of claim 1, wherein the system further comprises: a storage node;

3. The system of claim 1, wherein the scheduling node is configured to determine, according to the load conditions and the generated batch of homogeneous tasks respectively corresponding to the computing clusters, the number of homogeneous tasks included in the tasks to be executed respectively corresponding to the computing clusters; for each computing cluster, according to the number of homogeneous tasks contained in the tasks to be executed of the computing cluster, selecting the homogeneous tasks corresponding to the number from the undelivered homogeneous tasks as the tasks to be executed corresponding to the computing cluster.

4. The system of claim 3, wherein the system further comprises: a storage node;

the scheduling node is used for sending the identification of the computing cluster and the task to be executed of the computing cluster to the storage node; determining a target cluster from the computing clusters, and sending a first instruction to the storage node according to the identification of the target cluster; receiving a task to be executed of the target cluster returned by the storage node, and issuing the task to be executed of the target cluster to the target cluster;

the storage node is used for receiving the identification of the computing cluster and the task to be executed, which are sent by the scheduling node, determining the association relationship between the identification and the task to be executed, and storing the association relationship between the identification and the task to be executed; receiving a first instruction, determining a task to be executed of the target cluster according to the identification of the target cluster carried in the first instruction, and returning;

and the computing cluster is used for receiving the task to be executed issued by the dispatching node and executing the task.

5. The system of claim 1, wherein for each homogeneous task of the batch of homogeneous tasks, the homogeneous task has its corresponding task execution state, the task execution state comprising one of not started, executing success, and executing failure;

6. The system of claim 1, wherein the scheduling node is configured to determine a number of computing units that perform the batch of homogenous tasks as a specified number, and select computing units from each computing unit that perform the batch of homogenous tasks as specified units according to each load condition, the specified number of units being the specified number.

7. The system of claim 1, wherein the scheduling node is configured to determine, for each partition, a number of specified units included in the partition, and if the number of computing units performing the homogenous task exceeds the number of specified units included in the partition, treat the partition as a partition that does not perform the homogenous task.

8. The system of claim 1, wherein the scheduling node is configured to determine a number of computing units corresponding to a homogeneous task in the batch of homogeneous tasks, and determine, according to a load condition respectively corresponding to each computing cluster, the batch of homogeneous tasks, and the number of computing units corresponding to the homogeneous task, a task to be executed respectively corresponding to each computing cluster, where the number of computing units corresponding to the homogeneous task is a number of computing units required for executing the homogeneous task.

9. A method for processing batch homogenous tasks, wherein the method is applied to a scheduling node in a processing system of batch homogenous tasks, the system comprising: the system comprises a scheduling node and a plurality of computing clusters, wherein the computing clusters comprise a plurality of computing units, and each computing unit provides the same batch processing service; for each computing cluster, the computing cluster comprising a number of partitions, each partition comprising one or more computing nodes;

receiving task generation requests sent by a plurality of users, generating a request for each task, and responding to the task generation requests to generate a plurality of batches of homogeneous tasks to be executed, wherein the batches of homogeneous tasks are tasks with the same execution environment, the same service flow and mutual independence;

determining the load conditions corresponding to the computing units contained in each computing cluster, and determining the computing units used for executing the batch homogeneous task from the computing units according to the load conditions as a designated unit;

determining, for each homogeneous task included in the batch of homogeneous tasks, a number of computing units required to perform the homogeneous task, and, for each partition, a number of designated units included in the partition;

If the number of the computing units for executing the homogeneous task does not exceed the number of the designated units contained in the partition, taking the partition as the partition for executing the homogeneous task, and taking the homogeneous task as a task to be executed of each designated unit in the partition;

determining tasks to be executed, which correspond to each computing cluster respectively, according to the designated units, which correspond to each computing cluster respectively;

10. The method of claim 9, wherein the system comprises a storage node;

receiving task generation requests sent by a plurality of users, generating requests for each task, and generating a plurality of batches of homogeneous tasks to be executed in response to the task generation requests, wherein the tasks comprise:

sending a task acquisition request to the storage node;

and determining task generation data carried in the task generation request aiming at each received task generation request, and generating a plurality of batches of homogeneous tasks to be executed according to the task generation data.

11. The method of claim 9, wherein determining the tasks to be performed corresponding to the computing clusters respectively comprises:

12. The method of claim 9, wherein for each homogeneous task in the batch of homogeneous tasks, the homogeneous task has its corresponding task execution state, the task execution state comprising one of not started, executing success, and executing failure;

the method further comprises the steps of:

13. The method according to claim 9, wherein determining a computing unit for executing the batch of homogenous tasks from among the computing units as a designated unit comprises:

14. The method of claim 9, wherein the method further comprises:

for each partition, determining the number of specified units contained in the partition, and if the number of computing units for executing the homogenous task exceeds the number of specified units contained in the partition, taking the partition as a partition for not executing the homogenous task.

15. The method of claim 9, wherein the system further comprises: a storage node;

the method further comprises the steps of:

determining a target cluster from the computing clusters;

16. The method of claim 9, wherein the method further comprises:

17. A device for processing batch homogenous tasks, wherein the device is applied to a scheduling node in a batch homogenous task processing system, the batch homogenous task processing system comprising: the system comprises a scheduling node and a plurality of computing clusters, wherein the computing clusters comprise a plurality of computing units, and each computing unit provides the same batch processing task; for each computing cluster, the computing cluster comprising a number of partitions, each partition comprising one or more computing nodes;

The request determining module is used for receiving task generating requests sent by a plurality of users, generating requests for each task, and responding to the task generating requests to generate a plurality of batches of homogeneous tasks to be executed, wherein the batches of homogeneous tasks are tasks which have the same execution environment, the same service flow and mutually independent;

the task determining module is used for determining the load conditions respectively corresponding to all computing units contained in all computing clusters, and determining the computing units for executing the batch homogeneous tasks from all computing units according to the load conditions as a designated unit; determining, for each homogeneous task included in the batch of homogeneous tasks, a number of computing units required to perform the homogeneous task, and, for each partition, a number of designated units included in the partition; if the number of the computing units for executing the homogeneous task does not exceed the number of the designated units contained in the partition, taking the partition as the partition for executing the homogeneous task, and taking the homogeneous task as a task to be executed of each designated unit in the partition;

the issuing module is used for determining tasks to be executed corresponding to each computing cluster according to the designated units corresponding to each computing cluster; and issuing each task to be executed to the computing clusters corresponding to each task to be executed respectively.

18. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the method of any of the preceding claims 9-16.

19. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of any of the preceding claims 9-16 when executing the program.