WO2022078347A1

WO2022078347A1 - Task scheduling method and apparatus, electronic device, and storage medium

Info

Publication number: WO2022078347A1
Application number: PCT/CN2021/123361
Authority: WO
Inventors: 刘根生
Original assignee: 深圳壹账通智能科技有限公司
Priority date: 2020-10-13
Filing date: 2021-10-12
Publication date: 2022-04-21
Also published as: CN112181662B; CN112181662A

Abstract

The present application relates to the technical field of data processing, and provides a task scheduling method and apparatus, an electronic device, and a storage medium. The method comprises: upon receiving a task to be processed, acquiring the respective degrees of task parallelism of all schedulable servers and performing computation, so as to obtain the total degree of task parallelism; partitioning the task according to the total degree of task parallelism, so as to obtain a partition queue, wherein the partition queue consists of multiple task partitions having equal amounts of data; scheduling the task partitions in the partition queue according to the degree of task parallelism of each of the schedulable servers, such that the task partitions are distributed to the schedulable servers; and upon detecting, from all the schedulable servers, an abnormal server having experienced an abnormality, performing, according to the partition queue, a task transfer operation with respect to a task partition of the abnormal server. The present application can improve the efficiency of task processing. In addition, the partition queue can be stored in a blockchain.

Description

Task scheduling method, device, electronic device and storage medium

This application claims the priority of the Chinese patent application filed on October 13, 2020 with the application number 202011092091.8 and the invention title is "task scheduling method, device, electronic device and storage medium", the entire contents of which are incorporated by reference in this application.

technical field

The present application relates to the technical field of data processing, and in particular, to a task scheduling method, apparatus, electronic device and storage medium.

Background technique

At present, in the process of processing a large amount of data, in order to improve efficiency, a task can be divided into multiple task shards and processed by multiple servers at the same time, but the inventor realized that the performance of different servers may be inconsistent, and the processing task The efficiency of the server is inconsistent, and the division of task shards is not uniform. When some servers complete the task, other servers are still processing the task, which leads to the waste of some resources and the task processing efficiency is not high.

Therefore, how to perform task scheduling to improve the efficiency of task processing is a technical problem that needs to be solved.

SUMMARY OF THE INVENTION

In view of the above content, it is necessary to provide a task scheduling method, which can improve the efficiency of task processing.

A first aspect of the present application provides a task scheduling method, the task scheduling method includes:

When a pending task is received, obtain and count the task parallelism of all schedulable servers, and obtain the total parallelism of the task;

According to the total degree of parallelism of the task, the task to be processed is fragmented to obtain a fragmentation queue, wherein the fragmentation queue is composed of a plurality of task fragments with a uniform amount of data;

According to the task parallelism of each of the schedulable servers, the task shards in the sharding queue are scheduled to each of the schedulable servers;

When it is detected that there is an abnormal server with an abnormality among all the schedulable servers, task transfer is performed on the task fragment of the abnormal server according to the fragmentation queue.

A second aspect of the present application provides an electronic device, the electronic device includes a processor and a memory, the processor is configured to execute computer-readable instructions stored in the memory to implement the following steps:

A third aspect of the present application provides a computer-readable storage medium on which at least one computer-readable instruction is stored, and the at least one computer-readable instruction is executed by a processor to implement the following steps:

According to the total degree of parallelism of the task, the task to be processed is subjected to fragmentation processing to obtain a fragmentation queue, wherein the fragmentation queue is composed of multiple task fragments with an even amount of data;

A fourth aspect of the present application provides a task scheduling apparatus, and the task scheduling apparatus includes:

The acquisition module is used to acquire and count the task parallelism of all schedulable servers when the pending task is received, and obtain the total parallelism of the task;

a processing module, configured to perform fragmentation processing on the to-be-processed task according to the total parallelism of the task to obtain a fragmentation queue, wherein the fragmentation queue is composed of a plurality of task fragments with a uniform amount of data;

a scheduling module, configured to schedule the task shards in the sharding queue to each schedulable server according to the task parallelism of each schedulable server;

The transfer module is configured to perform task transfer on the task fragment of the abnormal server according to the fragment queue when it is detected that there is an abnormal server in all the schedulable servers.

It can be seen from the above technical solutions that in this application, a task can be divided into multiple task slices with a uniform amount of data according to the total task parallelism, and the task slices processed by each schedulable server have their own task parallelism. Consistent, that is, each schedulable server can simultaneously complete its own task fragmentation, so that the resources of different servers can be fully utilized, and the efficiency of task processing is improved. At the same time, if an exception occurs on the server that is executing the task, the task transfer is performed according to the task queue, that is, any server obtains the slice array by registering the task in the task queue and executes it to ensure the normal execution of the task.

Description of drawings

FIG. 1 is a flowchart of a preferred embodiment of the task scheduling method of the present application.

FIG. 2 is a functional block diagram of a preferred embodiment of the task scheduling apparatus of the present application.

FIG. 3 is a schematic structural diagram of an electronic device implementing a preferred embodiment of the task scheduling method of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field to which this application belongs. The terms used herein in the specification of the application are for the purpose of describing specific embodiments only, and are not intended to limit the application.

The task scheduling method of the embodiment of the present application is applied to an electronic device, and can also be applied to a hardware environment composed of an electronic device and a server connected to the electronic device through a network, and is jointly executed by the server and the electronic device. The network includes, but is not limited to: a wide area network, a metropolitan area network or a local area network.

The server may refer to a computer system that can provide services to other devices (eg, electronic devices) in the network. If a personal computer can provide a file transfer protocol (File Transfer Protocol, referred to as FTP) service, it can also be called a server. In a narrow sense, a server refers to some high-performance computers that can provide services to the outside world through a network. Compared with ordinary personal computers, servers have higher requirements in terms of stability, security, and performance. , chipset, memory, disk system, network and other hardware are different from ordinary personal computers.

The electronic device is a device that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field programmable gates Arrays (FPGA), digital processors (DSP), embedded devices, etc. The electronic equipment may also include network equipment and/or user equipment. Wherein, the network device includes but is not limited to a single network device, a server group composed of multiple network devices, or a cloud composed of a large number of hosts or network devices based on cloud computing, wherein cloud computing is distributed computing A super virtual computer consisting of a group of loosely coupled sets of computers. The user equipment includes but is not limited to any electronic product that can interact with the user through a keyboard, a mouse, a remote control, a touchpad or a voice-activated device, for example, a personal computer, a tablet computer, a smart phone, a personal digital Assistant PDA etc.

Please refer to FIG. 1. FIG. 1 is a flowchart of a preferred embodiment of a task scheduling method disclosed in the present application. Wherein, according to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted. Wherein, the executing subject of the task scheduling method may be an electronic device.

S11. When the task to be processed is received, obtain and count the task parallelism of all schedulable servers, and obtain the total parallelism of the task.

The degree of task parallelism refers to the number of tasks that can be processed at the same time.

The total task parallelism may refer to the sum of the task parallelisms of all schedulable servers, that is, the total number of tasks that can be simultaneously processed by all schedulable servers.

The schedulable server may refer to a currently available server.

In this optional implementation manner, the status information of all servers in the registration list can be obtained regularly by using a preset server (such as a server with a task scheduling center system) using an Internet packet explorer (Packet Internet Groper, ping). , if a response from a server can be received normally, it can be determined that the server is online and is a schedulable server that can be used; if a response from a server cannot be received, it can be determined that the server is offline, yes Unavailable server. Each schedulable server may send its own task parallelism to the preset server, and the preset server may collect statistics on the task parallelism of all schedulable servers to obtain the total task parallelism.

The Internet packet explorer is a program used to test the amount of network connections. It can send a request message to a specific destination host through a service command Ping to determine whether it can successfully exchange data with another host. package to determine whether the other host is running normally, whether the network is smooth, etc.

S12. According to the total parallelism of the tasks, perform sharding processing on the to-be-processed task to obtain a sharding queue, wherein the sharding queue is composed of a plurality of task shards with an even amount of data.

It should be emphasized that, in order to further ensure the privacy and security of the above-mentioned fragmented queue, the above-mentioned fragmented queue can also be stored in a node of a blockchain.

In this embodiment of the present application, the task to be processed may be fragmented to obtain a fragmented queue. For example, there are 1 million pieces of data that need to be processed for the task to be processed. After fragmentation processing, 10 task scores are obtained. Each task shard has 100,000 pieces of data, and the shard queue composed of these task shards can be expressed as: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9].

Specifically, according to the total parallelism of the tasks, performing fragmentation processing on the to-be-processed task, and obtaining a fragmented queue includes:

Divide the task to be processed, and obtain a plurality of task fragments whose number is consistent with the total parallelism of the task;

The fragmentation queue is generated according to the plurality of task fragments.

In this optional implementation manner, the tasks to be processed may be segmented according to the total parallelism of the tasks. Assuming that the total parallelism of the tasks is M, the tasks to be processed may be processed into M pieces Task fragmentation, and generate the fragmentation queue [0,1,2,...,M-1], the data volume of each task fragment is consistent with the data volume of any other task fragment.

S13. Schedule the task shards in the sharding queue to each schedulable server according to the task parallelism of each schedulable server.

Specifically, according to the task parallelism of each of the schedulable servers, scheduling the task shards in the sharding queue to each of the schedulable servers includes:

For each schedulable server, according to the arrangement order of the task shards in the shard queue, determine from the task shards a number of target task shards that are consistent with the task parallelism of the schedulable server ;

The target task is sharded and scheduled to the schedulable server.

In this optional implementation manner, for each of the schedulable servers, the number of tasks of the schedulable server may be sequentially obtained from the shard queue according to the arrangement order of the task shards in the shard queue. Target task shards with the same degree of parallelism, schedule the target task shards to the schedulable server, for example: assuming that the task parallelism of the schedulable server is 2, and the shard queue is [0,1 ,2,3,4], according to the sorting order from left to right, the obtained task shards are "0" and "1", that is, [0,1], at this time, the sharding queue becomes [ 2, 3, 4], assuming that the task parallelism of the next schedulable server is 1, the task shard obtained by the next schedulable server is "2", namely [2].

Optionally, the target task fragment can be stored in the task fragment scheduling record corresponding to the fragment queue in the form of an array, such as "A: [0,1]", which can be used to indicate that the server A last The obtained task shards are "0" and "1". If server A is scheduled with task shards "5" and "6", the "A: [0,1]" in the task shard scheduling record will be changed. Overwritten by "A:[5,6]".

As an optional implementation manner, after the task sharding in the sharding queue is scheduled to each schedulable server according to the task parallelism of each schedulable server, and the When detecting that there is an abnormal server with an abnormality in all the schedulable servers, according to the fragmentation queue, before performing task transfer on the task fragment of the abnormal server, the method further includes:

obtaining the second task fragmentation scheduling record of the fragmentation queue;

For each schedulable server, determine whether there is scheduling information corresponding to the schedulable server in the second task segment scheduling record, where the scheduling information is the task segment scheduled to the schedulable server. film information;

If there is no scheduling information corresponding to the schedulable server in the second task fragmentation scheduling record, write the task fragmentation information scheduled to the schedulable server into the second task fragmentation scheduling record ;or

If there is scheduling information corresponding to the schedulable server in the second task fragmentation scheduling record, the scheduling information is replaced with the task fragmentation information scheduled to the schedulable server.

In this optional implementation manner, the information of the task fragmentation last obtained by each server may be recorded to obtain the task fragmentation scheduling record. When the task fragment is scheduled to the schedulable server, the task fragment scheduling record needs to be updated. The task fragment scheduling record only saves the task fragment information obtained by the schedulable server for the last time, and the amount of information is small. , which can be read quickly. In this embodiment, the task fragmentation scheduling record at the current moment of the fragmentation queue is obtained, that is, the second task fragmentation scheduling record. For each schedulable server, this embodiment first determines the second task fragmentation scheduling record. Whether there is scheduling information corresponding to the schedulable server in the task fragmentation scheduling record, if there is no scheduling information corresponding to the schedulable server in the second task fragmentation scheduling record, it will be scheduled to the schedulable server. The task fragmentation information in the server is written into the second task fragmentation scheduling record; if there is scheduling information corresponding to the schedulable server in the second task fragmentation scheduling record, replace the scheduling information with Task fragmentation information scheduled to the schedulable server.

S14. When it is detected that there is an abnormal server with an abnormality among all the schedulable servers, according to the fragment queue, perform task transfer on the task fragment of the abnormal server.

In the embodiment of the present application, all the schedulable servers can be monitored, and when an abnormal server with an exception is monitored, the task transfer can be performed on the task fragment of the abnormal server according to the fragment queue in time, Improve system stability.

Specifically, according to the fragmentation queue, performing task transfer on the task fragmentation of the abnormal server includes:

obtaining the first task fragmentation scheduling record of the fragmentation queue;

According to the first task fragmentation scheduling record, determine the task fragmentation scheduled to the abnormal server;

The task shards of the abnormal server are added to the shard queue, so as to reschedule the task shards of the abnormal server.

In this optional implementation manner, when it is detected that there is an abnormal server in all the schedulable servers that has an exception, the task fragment scheduling record of the fragment queue at the current moment may be obtained, that is, the first task fragment Slice scheduling record, obtain the information of the last task slice obtained by the abnormal server from the first task slice scheduling record to determine the task slice scheduled to the abnormal server, and then assign the abnormal server to the The last obtained task shard is added to the shard queue, for example, the task shards of the abnormal server are "3" and "4", assuming that the shard queue at this time is empty, that is, the task of the shard queue All shards have been scheduled. After adding the task shard of the exception server to the shard queue, the shard queue can be expressed as [3, 4]. At this time, if the task registry exists The server in the idle state can schedule the task shards in the shard queue according to the task parallelism of the server in the idle state. For example, if the task parallelism of the server in the idle state is 2, then Both task shards "3" and "4" can be scheduled to this idle server.

As an optional implementation manner, when it is detected that there is an abnormal server with an abnormality in all the schedulable servers, according to the fragmentation queue, after the task fragmentation of the abnormal server is transferred, the The method also includes:

obtaining the third task fragmentation scheduling record of the fragmentation queue;

The scheduling information of the abnormal server is deleted from the scheduling record of the third task segment.

In this optional implementation manner, after the task transfer is performed according to the fragmentation queue, the task fragmentation scheduling record at the current moment in the fragmentation queue may be maintained, that is, the third task fragmentation scheduling may be performed. The record is maintained, the scheduling information of the abnormal server is deleted from the third task fragmentation scheduling record, and redundant data is deleted in time to improve system performance.

As an optional implementation manner, after the task shards in the sharding queue are scheduled to each schedulable server according to the task parallelism of each schedulable server, the method further include:

When detecting that there are target servers with completed tasks in all the schedulable servers, determine whether there are remaining task fragments in the fragmentation queue;

If there are remaining task fragments in the fragmentation queue, then according to the task parallelism of the target server, determine the task fragment to be scheduled from the remaining task fragments;

The to-be-scheduled task segment is scheduled to the target server.

In this optional implementation manner, when it is detected that among all the schedulable servers there is a target server that has completed tasks or when a new idle target server is started and becomes a schedulable server, it is determined that the target server can Continue to receive tasks. If there are remaining task shards in the shard queue, you can determine the to-be-scheduled task shards from the remaining task shards according to the task parallelism of the target server, and assign the to-be-scheduled task shards. Scheduling task segments are scheduled to the target server.

In the method flow described in Figure 1, a task can be divided into multiple task slices with uniform data volume according to the total task parallelism, and the task slices processed by each schedulable server are consistent with its own task parallelism. That is, each schedulable server can simultaneously complete its own task fragmentation, so that the resources of different servers can be fully utilized, and the efficiency of task processing is improved. At the same time, if an exception occurs on the server that is executing the task, the task transfer is performed according to the task queue, that is, any server obtains the slice array by registering the task in the task queue and executes it to ensure the normal execution of the task.

FIG. 2 is a functional block diagram of a preferred embodiment of a task scheduling apparatus disclosed in the present application.

Referring to FIG. 2 , the task scheduling apparatus 20 can run in an electronic device. The task scheduling apparatus 20 may include a plurality of functional modules composed of program code segments. The program codes of each program segment in the task scheduling apparatus 20 may be stored in a memory and executed by at least one processor to perform some or all of the steps in the task scheduling method described in FIG. 1 .

In this embodiment, the task scheduling apparatus 20 may be divided into multiple functional modules according to the functions performed by the task scheduling apparatus 20 . The functional modules may include: an acquisition module 201 , a processing module 202 , a scheduling module 203 and a transfer module 204 . A module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can perform fixed functions, and are stored in a memory.

The obtaining module 201 is configured to obtain and count the task parallelism of all schedulable servers when the task to be processed is received, and obtain the total parallelism of the task.

The schedulable server may refer to a currently available server.

The processing module 202 is configured to perform fragmentation processing on the to-be-processed task according to the total parallelism of the tasks to obtain a fragmentation queue, wherein the fragmentation queue is composed of a plurality of task fragments with a uniform amount of data.

The scheduling module 203 is configured to schedule the task shards in the sharding queue to each schedulable server according to the task parallelism of each schedulable server.

The transfer module 204 is configured to perform task transfer on the task shards of the abnormal server according to the shard queue when it is detected that there is an abnormal server with an exception among all the schedulable servers.

As an optional implementation manner, according to the fragmentation queue, the transfer module 204 performs task transfer on the task fragmentation of the abnormal server specifically as follows:

As an optional implementation manner, the task scheduling apparatus 20 further includes:

The first judgment module is used for the scheduling module 203 to schedule the tasks in the sharding queue to each schedulable server according to the task parallelism of each schedulable server. When there are target servers with completed tasks in all the schedulable servers, determine whether there are remaining task fragments in the fragmentation queue;

A determination module, for if there are remaining task fragments in the fragmentation queue, then according to the task parallelism of the target server, determine the task fragmentation to be scheduled from the remaining task fragments;

The scheduling module 203 is further configured to schedule the to-be-scheduled task segments to the target server.

As an optional implementation manner, the scheduling module 203 schedules the task shards in the sharding queue to each schedulable server according to the task parallelism of each schedulable server Specifically:

The target task is sharded and scheduled to the schedulable server.

As an optional implementation manner, the processing module 202 performs fragmentation processing on the to-be-processed task according to the total parallelism of the tasks, and the method for obtaining a fragmented queue is as follows:

Divide the task to be processed, and obtain a plurality of task shards whose number is consistent with the total parallelism of the task;

As an optional implementation manner, the obtaining module 201 is further configured to, according to the task parallelism of each schedulable server, the scheduling module 203 schedule the task shards in the sharding queue to each After each of the schedulable servers, and when the transfer module 204 detects that there is an abnormal server with an exception in all the schedulable servers, the transfer module 204 performs the task fragmentation on the abnormal server according to the fragmentation queue. Before the task transfer, obtain the second task fragmentation scheduling record of the fragmentation queue;

The task scheduling device 20 further includes:

A second judging module, configured to judge, for each schedulable server, whether there is scheduling information corresponding to the schedulable server in the second task fragmentation scheduling record, wherein the scheduling information is the scheduling information to the schedulable server. Describe the task fragmentation information of the schedulable server;

A writing module, configured to write the task fragmentation information scheduled to the schedulable server into the second task fragmentation scheduling record if the scheduling information corresponding to the schedulable server does not exist in the second task fragmentation scheduling record In the task shard scheduling record; or

The scheduling module 203 is further configured to replace the scheduling information with the task segment scheduled to the schedulable server if there is scheduling information corresponding to the schedulable server in the second task slice scheduling record. slice information.

As an optional implementation manner, the acquisition module 201 is further configured to, when the transfer module 204 detects that there is an abnormal server with an exception in all the schedulable servers, according to the fragmentation queue, to After the task fragmentation of the abnormal server performs task transfer, obtain the third task fragmentation scheduling record of the fragmentation queue;

The task scheduling device 20 further includes:

A deletion module, configured to delete the scheduling information of the abnormal server from the third task fragmentation scheduling record.

In the task scheduling device 20 described in FIG. 2 , a task can be divided into a plurality of task slices with an even amount of data according to the total task parallelism, and the task slices processed by each schedulable server and its own task parallelism Consistent, that is, each schedulable server can simultaneously complete its own task fragmentation, so that the resources of different servers can be fully utilized, and the efficiency of task processing is improved. At the same time, if an exception occurs on the server that is executing the task, the task transfer is performed according to the task queue, that is, any server obtains the slice array by registering the task in the task queue and executes it to ensure the normal execution of the task.

As shown in FIG. 3 , FIG. 3 is a schematic structural diagram of an electronic device implementing a preferred embodiment of the task scheduling method of the present application. The electronic device 3 includes a memory 31 , at least one processor 32 , a computer program 33 stored in the memory 31 and executable on the at least one processor 32 , and at least one communication bus 34 .

Those skilled in the art can understand that the schematic diagram shown in FIG. 3 is only an example of the electronic device 3, and does not constitute a limitation on the electronic device 3, and may include more or less components than those shown, or combinations thereof Certain components, or different components, for example, the electronic device 3 may also include input and output devices, network access devices, and the like.

The electronic device 3 also includes, but is not limited to, any electronic product that can interact with the user through a keyboard, a mouse, a remote control, a touchpad or a voice-activated device, for example, a personal computer, a tablet computer, a smart phone, Personal Digital Assistant (PDA), game console, Internet Protocol Television (IPTV), smart wearable devices, etc. The network where the electronic device 3 is located includes but is not limited to the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (Virtual Private Network, VPN) and the like.

The at least one processor 32 may be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC) ), Field-Programmable Gate Array (FPGA) or other programmable logic devices, transistor logic devices, discrete hardware components, etc. The processor 32 can be a microprocessor or the processor 32 can also be any conventional processor, etc. The processor 32 is the control center of the electronic device 3, and uses various interfaces and lines to connect the entire electronic device 3 of each part.

The memory 31 can be used to store the computer program 33 and/or modules/units, and the processor 32 executes or executes the computer programs and/or modules/units stored in the memory 31 and calls the computer programs and/or modules/units stored in the memory 31. 31 to realize various functions of the electronic device 3 . The memory 31 may mainly include a stored program area and a stored data area, wherein the stored program area may store an operating system, an application program (such as a sound playback function, an image playback function, etc.) required for at least one function, and the like; the storage data area may Data and the like created according to the use of the electronic device 3 are stored. In addition, the memory 31 may include volatile and non-volatile memory, such as hard disk, internal memory, plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, flash memory card ( Flash Card), at least one disk storage device, flash memory device, etc.

Exemplarily, the computer program 33 may be divided into one or more modules/units, and the one or more modules/units are stored in the memory 31 and executed by the processor 32 to complete the this application. The one or more modules/units may be a series of computer-readable instruction segments capable of performing specific functions, and the computer-readable instruction segments are used to describe the execution process of the computer-readable instructions in the electronic device 3 . For example, the computer program 33 can be divided into an acquisition module 201 , a processing module 202 , a scheduling module 203 , and a transfer module 204 .

1, the memory 31 in the electronic device 3 stores multiple computer-readable instructions to implement a task scheduling method, and the processor 32 can execute the multiple computer-readable instructions to implement:

Specifically, for the specific implementation method of the above-mentioned instruction by the processor 32, reference may be made to the description of the relevant steps in the corresponding embodiment of FIG. 1, and details are not described herein.

In the electronic device 3 depicted in FIG. 3 , a task can be divided into multiple task slices with a uniform amount of data according to the total task parallelism, and the task slice processed by each schedulable server is consistent with its own task parallelism , that is, each schedulable server can simultaneously complete the obtained task fragmentation, so that the resources of different servers can be fully utilized, and the efficiency of task processing is improved. At the same time, if an exception occurs on the server that is executing the task, the task transfer is performed according to the task queue, that is, any server obtains the slice array by registering the task in the task queue and executes it to ensure the normal execution of the task.

If the modules/units integrated in the electronic device 3 are implemented in the form of software functional units and are sold or used as independent products, they can be stored in a computer-readable storage medium, and the computer-readable storage medium may be a non-volatile storage medium. A volatile storage medium can also be a volatile storage medium. Based on this understanding, the present application can implement all or part of the processes in the methods of the above embodiments, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer When the program is executed by the processor, the steps of the foregoing method embodiments can be implemented. Wherein, the computer program code may be in the form of source code, object code, executable file or some intermediate form, and so on. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory) , Random Access Memory (RAM, Random Access Memory), etc.

In the several embodiments provided in this application, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. For example, the apparatus embodiments described above are only illustrative. For example, the division of the modules is only a logical function division, and there may be other division manners in actual implementation.

The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or can be implemented in the form of hardware plus software function modules.

It will be apparent to those skilled in the art that the present application is not limited to the details of the above-described exemplary embodiments, but that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Accordingly, the embodiments are to be regarded in all respects as illustrative and not restrictive, and the scope of the application is to be defined by the appended claims rather than the foregoing description, which is therefore intended to fall within the scope of the claims. All changes within the meaning and scope of the equivalents of , are included in this application. Any reference signs in the claims shall not be construed as limiting the involved claim. Furthermore, it is clear that the word "comprising" does not exclude other units or steps and the singular does not exclude the plural. Several units or means recited in the system claims can also be realized by one unit or means by means of software or hardware. Second-class terms are used to denote names and do not denote any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit them. Although the present application has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present application can be Modifications or equivalent substitutions can be made without departing from the spirit and scope of the technical solutions of the present application.

Claims

A task scheduling method, wherein the task scheduling method comprises:

When a pending task is received, obtain and count the task parallelism of all schedulable servers, and obtain the total parallelism of the task;

According to the total degree of parallelism of the task, the task to be processed is fragmented to obtain a fragmentation queue, wherein the fragmentation queue is composed of a plurality of task fragments with a uniform amount of data;

According to the task parallelism of each of the schedulable servers, the task shards in the sharding queue are scheduled to each of the schedulable servers;

When it is detected that there is an abnormal server with an abnormality among all the schedulable servers, task transfer is performed on the task fragment of the abnormal server according to the fragmentation queue.
The task scheduling method according to claim 1, wherein, according to the fragmentation queue, performing task transfer on the task fragmentation of the abnormal server comprises:

obtaining the first task fragmentation scheduling record of the fragmentation queue;

According to the first task fragmentation scheduling record, determine the task fragmentation scheduled to the abnormal server;

The task shards of the abnormal server are added to the shard queue, so as to reschedule the task shards of the abnormal server.
The task scheduling method according to claim 1, wherein after the task sharding in the sharding queue is scheduled to each schedulable server according to the task parallelism of each schedulable server , the task scheduling method further includes:

When detecting that there are target servers with completed tasks in all the schedulable servers, determine whether there are remaining task fragments in the fragmentation queue;

If there are remaining task fragments in the fragmentation queue, then according to the task parallelism of the target server, determine the task fragment to be scheduled from the remaining task fragments;

The to-be-scheduled task segment is scheduled to the target server.
The task scheduling method according to claim 1, wherein, according to the task parallelism of each of the schedulable servers, scheduling the task shards in the sharding queue to each of the schedulable servers comprises: :

For each schedulable server, according to the arrangement order of the task shards in the shard queue, determine from the task shards a number of target task shards that are consistent with the task parallelism of the schedulable server ;

The target task is sharded and scheduled to the schedulable server.
The task scheduling method according to any one of claims 1 to 4, wherein, according to the total parallelism of the tasks, performing fragmentation processing on the to-be-processed task, and obtaining a fragmented queue comprises:

Divide the task to be processed, and obtain a plurality of task fragments whose number is consistent with the total parallelism of the task;

The fragmentation queue is generated according to the plurality of task fragments.
The task scheduling method according to any one of claims 1 to 4, wherein, according to the task parallelism of each schedulable server, the task sharding in the sharding queue is scheduled to each After the schedulable server, and when it is detected that there is an abnormal server with an abnormality in all the schedulable servers, according to the fragmentation queue, before the task transfer is performed on the task fragment of the abnormal server, the The task scheduling method further includes:

obtaining the second task fragmentation scheduling record of the fragmentation queue;

For each schedulable server, determine whether there is scheduling information corresponding to the schedulable server in the second task segment scheduling record, where the scheduling information is the task segment scheduled to the schedulable server. film information;

If there is no scheduling information corresponding to the schedulable server in the second task fragmentation scheduling record, write the task fragmentation information scheduled to the schedulable server into the second task fragmentation scheduling record ;or

If there is scheduling information corresponding to the schedulable server in the second task fragmentation scheduling record, the scheduling information is replaced with the task fragmentation information scheduled to the schedulable server.
The task scheduling method according to any one of claims 1 to 4, wherein when it is detected that there is an abnormal server in all the schedulable servers that has an abnormality, according to the fragmentation queue, the abnormality After the task fragmentation of the server performs the task transfer, the task scheduling method further includes:

obtaining the third task fragmentation scheduling record of the fragmentation queue;

The scheduling information of the abnormal server is deleted from the scheduling record of the third task segment.
A task scheduling device, wherein the task scheduling device comprises:

The acquisition module is used to acquire and count the task parallelism of all schedulable servers when the pending task is received, and obtain the total parallelism of the task;

a processing module, configured to perform fragmentation processing on the to-be-processed task according to the total parallelism of the task to obtain a fragmentation queue, wherein the fragmentation queue is composed of a plurality of task fragments with a uniform amount of data;

a scheduling module, configured to schedule the task shards in the sharding queue to each schedulable server according to the task parallelism of each schedulable server;

The transfer module is configured to perform task transfer on the task fragment of the abnormal server according to the fragment queue when it is detected that there is an abnormal server in all the schedulable servers.
An electronic device, wherein the electronic device includes a processor and a memory, and the processor is configured to execute at least one computer-readable instruction stored in the memory to implement the following steps:

When a pending task is received, obtain and count the task parallelism of all schedulable servers, and obtain the total parallelism of the task;

According to the total degree of parallelism of the task, the task to be processed is fragmented to obtain a fragmentation queue, wherein the fragmentation queue is composed of a plurality of task fragments with a uniform amount of data;

According to the task parallelism of each of the schedulable servers, the task shards in the sharding queue are scheduled to each of the schedulable servers;

When it is detected that there is an abnormal server with an abnormality among all the schedulable servers, task transfer is performed on the task fragment of the abnormal server according to the fragmentation queue.
The electronic device according to claim 9, wherein when the task transfer is performed on the task shard of the exception server according to the shard queue, the processor executes the at least one computer-readable instruction to Implement the following steps:

obtaining the first task fragmentation scheduling record of the fragmentation queue;

According to the first task fragmentation scheduling record, determine the task fragmentation scheduled to the abnormal server;

The task shards of the abnormal server are added to the shard queue, so as to reschedule the task shards of the abnormal server.
The electronic device according to claim 9, wherein after the task sharding in the sharding queue is scheduled to each schedulable server according to the task parallelism of each schedulable server , the processor also executes the at least one computer-readable instruction to implement the following steps:

When detecting that there are target servers with completed tasks in all the schedulable servers, determine whether there are remaining task fragments in the fragmentation queue;

If there are remaining task fragments in the fragmentation queue, then according to the task parallelism of the target server, determine the task fragment to be scheduled from the remaining task fragments;

The to-be-scheduled task segment is scheduled to the target server.
The electronic device according to claim 9, wherein when the task shards in the sharding queue are scheduled to each of the schedulable servers according to the task parallelism of each of the schedulable servers , the processor executes the at least one computer-readable instruction to implement the following steps:

For each schedulable server, according to the arrangement order of the task shards in the shard queue, determine from the task shards a number of target task shards that are consistent with the task parallelism of the schedulable server ;

The target task is sharded and scheduled to the schedulable server.
The electronic device according to any one of claims 9 to 12, wherein, when the to-be-processed task is sharded according to the total parallelism of the tasks to obtain a sharded queue, the processor The at least one computer-readable instruction is executed to implement the following steps:

Divide the task to be processed, and obtain a plurality of task fragments whose number is consistent with the total parallelism of the task;

The fragmentation queue is generated according to the plurality of task fragments.
The electronic device according to any one of claims 9 to 12, wherein, in the task parallelism of each of the schedulable servers, the task shards in the sharding queue are scheduled to each of the sharded servers. After the schedulable server, and when it is detected that there is an abnormal server with an abnormality in all the schedulable servers, according to the fragmentation queue, before the task transfer of the task fragment of the abnormal server is performed, the The processor also executes the at least one computer-readable instruction to implement the following steps:

obtaining the second task fragmentation scheduling record of the fragmentation queue;

For each schedulable server, determine whether there is scheduling information corresponding to the schedulable server in the second task segment scheduling record, where the scheduling information is the task segment scheduled to the schedulable server. film information;

If there is no scheduling information corresponding to the schedulable server in the second task fragmentation scheduling record, write the task fragmentation information scheduled to the schedulable server into the second task fragmentation scheduling record ;or

If there is scheduling information corresponding to the schedulable server in the second task fragmentation scheduling record, the scheduling information is replaced with the task fragmentation information scheduled to the schedulable server.
A computer-readable storage medium, wherein the computer-readable storage medium stores at least one computer-readable instruction, and the at least one computer-readable instruction implements the following steps when executed by a processor:

When a pending task is received, obtain and count the task parallelism of all schedulable servers, and obtain the total parallelism of the task;

According to the total degree of parallelism of the task, the task to be processed is fragmented to obtain a fragmentation queue, wherein the fragmentation queue is composed of a plurality of task fragments with a uniform amount of data;

According to the task parallelism of each of the schedulable servers, the task shards in the sharding queue are scheduled to each of the schedulable servers;

When it is detected that there is an abnormal server with an abnormality among all the schedulable servers, task transfer is performed on the task fragment of the abnormal server according to the fragmentation queue.
The storage medium according to claim 15, wherein when the task transfer is performed on the task shard of the exception server according to the shard queue, the at least one computer-readable instruction is executed by the processor to implement The following steps:

obtaining the first task fragmentation scheduling record of the fragmentation queue;

According to the first task fragmentation scheduling record, determine the task fragmentation scheduled to the abnormal server;

The task shards of the abnormal server are added to the shard queue, so as to reschedule the task shards of the abnormal server.
The storage medium according to claim 15, wherein after the task sharding in the sharding queue is scheduled to each of the schedulable servers according to the task parallelism of each schedulable server , the at least one computer-readable instruction is executed by the processor to further implement the following steps:

When detecting that there are target servers with completed tasks in all the schedulable servers, determine whether there are remaining task fragments in the fragmentation queue;

If there are remaining task fragments in the fragmentation queue, then according to the task parallelism of the target server, determine the task fragment to be scheduled from the remaining task fragments;

The to-be-scheduled task segment is scheduled to the target server.
The storage medium according to claim 15, wherein when the task shards in the sharding queue are scheduled to each of the schedulable servers according to the task parallelism of each of the schedulable servers , the at least one computer-readable instruction is executed by the processor to implement the following steps:

For each schedulable server, according to the arrangement order of the task shards in the shard queue, determine from the task shards a number of target task shards that are consistent with the task parallelism of the schedulable server ;

The target task is sharded and scheduled to the schedulable server.
The storage medium according to any one of claims 15 to 18, wherein when the to-be-processed task is sharded according to the total parallelism of the tasks to obtain a sharded queue, the at least one The computer readable instructions, when executed by a processor, perform the following steps:

Divide the task to be processed, and obtain a plurality of task shards whose number is consistent with the total parallelism of the task;

The fragmentation queue is generated according to the plurality of task fragments.
The storage medium according to any one of claims 15 to 18, wherein in the task parallelism of each of the schedulable servers, the task shards in the sharding queue are scheduled to each of the sharded servers. After the schedulable server, and when it is detected that there is an abnormal server with an abnormality in all the schedulable servers, according to the fragmentation queue, before the task transfer is performed on the task fragment of the abnormal server, the The at least one computer-readable instruction is executed by the processor to further implement the following steps:

obtaining the second task fragmentation scheduling record of the fragmentation queue;

For each schedulable server, determine whether there is scheduling information corresponding to the schedulable server in the second task segment scheduling record, where the scheduling information is the task segment scheduled to the schedulable server. film information;

If there is no scheduling information corresponding to the schedulable server in the second task fragmentation scheduling record, write the task fragmentation information scheduled to the schedulable server into the second task fragmentation scheduling record ;or

If there is scheduling information corresponding to the schedulable server in the second task fragmentation scheduling record, the scheduling information is replaced with the task fragmentation information scheduled to the schedulable server.