CN112559143A - Task scheduling method and system and computing device - Google Patents
Task scheduling method and system and computing device Download PDFInfo
- Publication number
- CN112559143A CN112559143A CN202011414208.XA CN202011414208A CN112559143A CN 112559143 A CN112559143 A CN 112559143A CN 202011414208 A CN202011414208 A CN 202011414208A CN 112559143 A CN112559143 A CN 112559143A
- Authority
- CN
- China
- Prior art keywords
- task
- executed
- execution
- tasks
- scheduling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a task scheduling method, which is executed in a computing device, wherein the computing device is connected with a plurality of executing devices, and the method comprises the following steps: receiving a task execution request, wherein the task execution request comprises a task execution table, and the task execution table comprises a plurality of tasks to be executed; judging whether the upstream tasks corresponding to each task to be executed in the task execution table are executed successfully or not; if the upstream tasks of the tasks to be executed are determined to be successfully executed, triggering the tasks to be executed, and determining target execution equipment for executing the tasks to be executed; and sending the task to be executed to the target execution equipment so that the target execution equipment can execute the task. The invention also discloses a corresponding task scheduling system and a corresponding computing device. The task scheduling method and the task scheduling system can flexibly and efficiently schedule and execute a large number of tasks.
Description
Technical Field
The invention relates to the technical field of computers and internet, in particular to a task scheduling method, a task scheduling system and computing equipment.
Background
In actual service development, some timing tasks are often needed to solve problems, and service developers usually schedule tasks by using Crontab, SpringCron, and QuartZ. However, in a scenario where the number of tasks and the amount of data are large, for example, in a large data scenario, the dependency relationship between the tasks is complex, and the management of the tasks is very confused, which not only seriously affects the execution efficiency of the tasks, but also causes a series of problems: the life cycle of the tasks cannot be uniformly coordinated and managed, the tasks are difficult to arrange when dependence exists between the tasks, the tasks cannot be grouped or assigned to execute the tasks, and the like.
In the prior art, one scheme is to schedule a task based on QuartZ, which is an open source project and is a set of framework for scheduling an open source scheduling task completely written by Java, and a trigger is used to set a task timing operation rule and control the operation time of the task. The disadvantage of this solution is that the life cycle of the tasks cannot be uniformly managed and the orchestration of the tasks cannot be performed.
Yet another solution is to perform task scheduling based on Azkaban, a batch workflow task scheduler open by Linkedin, for running a set of work and processes in a workflow in a particular order. Azkaban defines a KV file format to establish dependencies between tasks and provides a Web user interface for easy operation to maintain and track workflows. The main drawback of this solution is that distributed deployment is not possible and the task orchestration has to be done on a stream basis.
For this reason, a task scheduling method is needed to solve the problems in the above technical solutions.
Disclosure of Invention
To this end, the present invention provides a task scheduling method and a task scheduling system in an attempt to solve or at least alleviate the above-existing problems.
According to an aspect of the present invention, there is provided a task scheduling method, executed in a computing device, the computing device being connected to a plurality of execution devices, the method comprising: receiving a task execution request, wherein the task execution request comprises a task execution table, and the task execution table comprises a plurality of tasks to be executed; judging whether the upstream tasks corresponding to each task to be executed in the task execution table are executed successfully or not; if the upstream tasks of the tasks to be executed are determined to be successfully executed, triggering the tasks to be executed, and determining target execution equipment for executing the tasks to be executed; and sending the task to be executed to the target execution equipment so that the target execution equipment can execute the task.
Optionally, in the task scheduling method according to the present invention, the step of determining a target execution device for executing the task to be executed includes: acquiring an execution device list comprising a plurality of pieces of execution device information, wherein the execution device list comprises device information respectively corresponding to the plurality of execution devices; the target execution device is determined based on device information of the plurality of execution devices.
Optionally, in the task scheduling method according to the present invention, the device information includes one or more of a heartbeat, a memory, a CPU, and a number of tasks currently executed by the execution device.
Optionally, in the task scheduling method according to the present invention, the step of determining whether the upstream tasks of the to-be-executed task in the task execution table are executed successfully includes: registering dependency monitoring for each task to be executed to monitor each task to be executed in the task execution table and judge whether each task to be executed is executed successfully; when determining that one task is successfully executed, triggering other dependent monitoring to judge whether the successfully executed task is an upstream task of the task to be executed corresponding to the other dependent monitoring; if so, traversing each upstream task corresponding to the task to be executed in the task execution table to determine whether each upstream task is successfully executed.
Optionally, in the task scheduling method according to the present invention, after sending the task to be executed to the target execution device, the method further includes: judging whether the task is executed successfully; and if the execution is successful, triggering other dependent monitoring to judge whether the task which is successfully executed is an upstream task of the task to be executed corresponding to the other dependent monitoring.
Optionally, in the task scheduling method according to the present invention, the step of determining whether the task is successfully executed further includes: and if the task fails to be executed, sending an alarm message of the task failure to the client.
Optionally, in the task scheduling method according to the present invention, a plurality of tasks to be executed in the task execution table are arranged in a tree structure, and each task to be executed corresponds to one node in the tree structure.
Optionally, in the task scheduling method according to the present invention, a node corresponding to each task to be executed in the tree structure is adapted to be determined by an execution cycle and an execution time of the task to be executed.
Optionally, in the task scheduling method according to the present invention, the computing device is connected to a scheduling platform, and the scheduling platform is adapted to generate a task execution table based on a plurality of tasks to be executed, and send a task execution request to the computing device based on the task execution table.
According to an aspect of the present invention, there is provided a task scheduling system, including a scheduling cluster, the scheduling cluster including: a scheduling server adapted to perform the task scheduling method as described above; and each executing device is connected with the scheduling server and is suitable for executing the tasks to be executed distributed by the scheduling server.
Optionally, in the task scheduling system according to the present invention, further comprising: and the scheduling platform is connected with the scheduling server and is suitable for generating a task execution table based on a plurality of tasks to be executed and sending a task execution request to the scheduling server based on the task execution table.
According to an aspect of the present invention, there is provided a computing device comprising: at least one processor; and a memory storing program instructions, wherein the program instructions are configured to be executed by the at least one processor, the program instructions comprising instructions for performing the task scheduling method as described above.
According to an aspect of the present invention, there is provided a readable storage medium storing program instructions which, when read and executed by a computing device, cause the computing device to perform the method as described above.
According to the technical scheme, the invention provides a task scheduling method and a task scheduling system. In the task execution table with the tree structure, a dependency relationship exists between each task and an upstream task, the execution state of each task can be monitored by registering dependency monitoring for each task, and the triggering execution of each task depends on the upstream task having the dependency relationship with the task. Therefore, the tasks have dependence triggering relations and run relatively independently, overall management and ordered scheduling of a plurality of tasks to be executed are achieved, the change of each task does not affect the execution of other tasks, a large number of timing tasks can be scheduled and executed flexibly and efficiently, and the data synchronization of the distributed data system is more efficient.
In addition, according to the task scheduling method provided by the invention, when the task to be executed is triggered, the task to be executed is allocated to the current most matched target execution equipment for execution, so that the task execution efficiency can be improved, and the existing execution equipment resources can be reasonably utilized.
Drawings
To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings, which are indicative of various ways in which the principles disclosed herein may be practiced, and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description read in conjunction with the accompanying drawings. Throughout this disclosure, like reference numerals generally refer to like parts or elements.
FIG. 1 shows a schematic diagram of a task scheduling system 100 according to one embodiment of the present invention;
FIG. 2 shows a schematic diagram of a computing device 200, according to one embodiment of the invention; and
FIG. 3 shows a flow diagram of a task scheduling method 300 according to one embodiment of the invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
FIG. 1 shows a schematic diagram of a task scheduling system 100 according to one embodiment of the present invention.
As shown in fig. 1, the system 100 may include a dispatch cluster 130 and a dispatch platform 110, the dispatch platform 110 is communicatively connected to the dispatch cluster 130, the dispatch cluster 130 includes a dispatch server 120 and a plurality of execution devices 125 adapted to execute tasks, the dispatch server 120 and the plurality of execution devices 125 may be distributively deployed at different geographic locations, and the dispatch server 120 is communicatively connected to the plurality of execution devices 125, for example, through a wired or wireless network connection. Here, the invention is not limited to the specific number or deployment of the execution devices 125 in the cluster 130.
The scheduling platform 110 may be implemented as a personal computer including desktop and notebook configurations, and may also be implemented as a mobile device such as a cell phone, tablet, notebook, television box, wearable device, and the like. The operator may perform task maintenance on the scheduling platform 110, for example, perform operation, maintenance and scheduling on a plurality of tasks to be executed, and send a task execution request for the plurality of tasks to be executed on the scheduling platform 110. The dispatch platform 110 may send a task execution request including a plurality of tasks to be executed to the dispatch cluster 130 in response to an operator request.
The scheduling cluster 130 as a whole may be used to provide data change management services for the distributed data system to achieve synchronous updating of data between the various data storage devices in the distributed data system. For example, when data in the distributed data system has a change demand, synchronous update of the data between the data storage devices can be realized by creating a task based on the data change demand at the scheduling platform 110 and executing the task by the scheduling cluster 130.
Among other things, the dispatch server 120 may be used to provide task dispatch services, i.e., to coordinate the execution of multiple tasks, such as assigning each task to be executed to an appropriate execution device 125 in the cluster 130.
In one embodiment, the scheduling server 120 and each execution device 125 may be implemented as a Master-Slave structure, wherein the scheduling server 120 may be implemented as a Master server, and each execution device 125 is implemented as a Slave server. The dispatch server 120 and the various execution devices 125 may communicate messages based on the framework Netty of the TCP protocol.
In one embodiment, a distributed data system includes a plurality of data storage devices deployed in a distributed manner. Each execution device 125 of dispatch server 120 in dispatch cluster 130 is coupled to a corresponding data storage device. In this way, the scheduling cluster 130 can maintain and manage the change of the data in the distributed data system, and when the data in the distributed data system needs to be changed, the task is created, and the scheduling cluster 130 is used to perform the task in a coordinated manner, so that the synchronous update of the data among the data storage devices in the distributed data system can be realized.
In one embodiment, the dispatch platform 110 is communicatively coupled to the dispatch server 120, such as by a wired or wireless network connection, although the invention is not limited thereto. The dispatch platform 110 and dispatch server 120 may also be implemented as a single computing device, with the dispatch platform 110 residing at the front end of the computing device.
The scheduling server 120 may receive a task execution request sent by the scheduling platform 110, where the task execution request includes a task execution table, and the task execution table includes a plurality of tasks to be executed. It should be noted that the scheduling platform 110 may generate the task execution table based on the plurality of tasks to be executed, and when the scheduling platform 110 triggers the task execution request, the scheduling platform 110 sends the task execution request to the scheduling server 120 based on the task execution table to request the scheduling server 120 to execute the plurality of tasks to be executed in the task execution table.
In an embodiment, the tasks to be executed are timed tasks, an operator may schedule a plurality of tasks to be executed on the scheduling platform 110 based on an execution cycle and an execution time of each task to be executed, and the scheduling platform 110 generates a task execution table with a tree structure based on the plurality of tasks to be executed and the execution cycle and the execution time corresponding to each task to be executed. That is to say, a plurality of tasks to be executed in the task execution table are arranged in a tree structure, and each task to be executed corresponds to one node in the tree structure. Here, the corresponding node of each task to be executed in the tree structure is determined by the execution period and execution time of the task to be executed.
It should be noted that, in the task execution table with the tree structure, there is a dependency relationship between tasks to be executed based on respective corresponding node positions. Specifically, the task to be executed of each node may correspond to one or more upstream tasks, and the task to be executed of each node and the corresponding upstream task have a dependency relationship when triggered to execute.
It should be noted that each node in the tree structure may correspond to a parent node and one or more child nodes, and the task to be executed corresponding to the parent node of each node and the parent node of the parent node is an upstream task corresponding to the node. That is to say, the upstream task corresponding to the task to be executed by the node includes: and (4) from the root node to the tasks to be executed corresponding to all the nodes on the branch passed by the node.
In one embodiment, the scheduling server 120 is adapted to perform the task scheduling method 300 of the present invention. The task scheduling method 300 of the present invention will be described in detail below.
In an embodiment according to the present invention, the scheduling server 120 may control the execution of the tasks based on the dependency relationship between the tasks to be executed in the task execution table, so as to implement the ordered scheduling of the tasks.
Specifically, after receiving the task execution request from the scheduling platform 110, the scheduling server 120 determines whether the upstream tasks corresponding to each to-be-executed task in the task execution table are executed successfully. The scheduling server 120 triggers the execution of the task to be executed only when it is determined that all the upstream tasks corresponding to (any one of) the task to be executed in the task execution table are successfully executed. After the task to be executed is triggered, a target execution device in the cluster 130 for executing the task to be executed is determined, and the task to be executed is allocated to the target execution device for execution.
In one embodiment, the execution state of each task to be executed in the task execution table is monitored by registering dependent snooping for each task to be executed in the task execution table, and whether each task to be executed is executed successfully is judged. When any task is determined to be successfully executed, triggering each other dependent monitor to judge: whether the task that is successfully executed is an upstream task of the corresponding task to be executed (i.e. the other tasks that rely on snooping itself). If the task is the upstream task, traversing each upstream task corresponding to the task to be executed (namely the task monitored by the dependent monitor) in the task execution table so as to judge whether each upstream task is executed successfully.
It should be noted that the dependent snooping is snooping registered to the task to be executed based on a dependency relationship between the task to be executed and the corresponding one or more upstream tasks. Triggering execution of a task based on dependency snooping depends on an upstream task having a dependency relationship with the task. That is, based on registering dependent snooping for the to-be-executed task, whether the to-be-executed task can trigger execution depends on whether each upstream task is successfully executed.
According to the task scheduling system, the task execution table with the tree structure is created, and the scheduling server schedules the execution of a plurality of tasks to be executed based on the task execution table, so that the overall management of all the tasks to be executed is realized. In the task execution table with the tree structure, a dependency relationship exists between each task and an upstream task, the execution state of each task can be monitored by registering dependency monitoring for each task, and the triggering execution of each task depends on the upstream task having the dependency relationship with the task. Therefore, the tasks have dependence triggering relations and run relatively independently, overall management and ordered scheduling of a plurality of tasks to be executed are achieved, the change of each task does not affect the execution of other tasks, a large number of timing tasks can be scheduled and executed flexibly and efficiently, and the data synchronization of the distributed data system is more efficient. In addition, the tasks to be executed are reasonably distributed for the multiple executing devices which are distributed, so that the high availability of the system can be ensured, the load balance is realized, and the efficient and stable operation of the multiple tasks is facilitated.
According to an embodiment of the invention, the various components of the task scheduling system 100 described above may communicate over one or more networks, such as a Local Area Network (LAN) or a Wide Area Network (WAN), such as the Internet. The dispatch platform 110, dispatch server 120, and each execution device 125 may be implemented by a computing device 200 as described below.
FIG. 2 shows a schematic diagram of a computing device 200, according to one embodiment of the invention.
As shown in FIG. 2, in a basic configuration 202, a computing device 200 typically includes a system memory 206 and one or more processors 204. A memory bus 208 may be used for communication between the processor 204 and the system memory 206.
Depending on the desired configuration, the processor 204 may be any type of processing, including but not limited to: a microprocessor (μ P), a microcontroller (μ C), a Digital Signal Processor (DSP), or any combination thereof. The processor 204 may include one or more levels of cache, such as a level one cache 210 and a level two cache 212, a processor core 214, and registers 216. Example processor cores 214 may include Arithmetic Logic Units (ALUs), Floating Point Units (FPUs), digital signal processing cores (DSP cores), or any combination thereof. The example memory controller 218 may be used with the processor 204, or in some implementations the memory controller 218 may be an internal part of the processor 204.
Depending on the desired configuration, system memory 206 may be any type of memory, including but not limited to: volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 106 may include an operating system 220, one or more applications 222, and program data 224. The application 222 is actually a plurality of program instructions that direct the processor 204 to perform corresponding operations. In some embodiments, application 222 may be arranged to cause processor 204 to operate with program data 224 on an operating system.
Computing device 200 may also include an interface bus 240 that facilitates communication from various interface devices (e.g., output devices 242, peripheral interfaces 244, and communication devices 246) to the basic configuration 202 via the bus/interface controller 230. The example output device 242 includes a graphics processing unit 248 and an audio processing unit 250. They may be configured to facilitate communication with various external devices, such as a display or speakers, via one or more a/V ports 252. Example peripheral interfaces 244 can include a serial interface controller 254 and a parallel interface controller 256, which can be configured to facilitate communications with external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device) or other peripherals (e.g., printer, scanner, etc.) via one or more I/O ports 258. An example communication device 246 may include a network controller 260, which may be arranged to facilitate communications with one or more other computing devices 262 over a network communication link via one or more communication ports 264.
A network communication link may be one example of a communication medium. Communication media may typically be embodied by computer readable instructions, data structures, program modules, and may include any information delivery media, such as carrier waves or other transport mechanisms, in a modulated data signal. A "modulated data signal" may be a signal that has one or more of its data set or its changes made in such a manner as to encode information in the signal. By way of non-limiting example, communication media may include wired media such as a wired network or private-wired network, and various wireless media such as acoustic, Radio Frequency (RF), microwave, Infrared (IR), or other wireless media. The term computer readable media as used herein may include both storage media and communication media.
In an embodiment according to the invention, the computing device 200 is configured to perform a task scheduling method according to the invention. Included in the application 222 of the computing device 200 are a plurality of program instructions for performing the task scheduling method 300 of the present invention that may direct the processor 204 to perform the task scheduling method 300 of the present invention. The computing device 200 can implement the scheduled execution of multiple tasks by performing the task scheduling method 300 of the present invention.
FIG. 3 shows a flow diagram of a task scheduling method 300 according to one embodiment of the invention. The method 300 is suitable for execution in a computing device, such as the computing device 200 described above.
In an embodiment consistent with the invention, computing device 200 is coupled to dispatch platform 110 and may receive a task execution request sent by dispatch platform 110. The computing device 200 is communicatively coupled to a plurality of execution devices 125, respectively, where the computing device 200 and the plurality of execution devices 125 together form a dispatch cluster 130. Where the computing device 200 may assign tasks to be performed to the appropriate execution devices 125 for execution to coordinate the execution of the multiple tasks.
It should be noted that the task to be executed is, for example, a task created based on the data update requirement of the distributed data system at the scheduling platform 110, and synchronous data update between the data storage devices in the distributed data system can be realized by submitting a plurality of tasks to be executed to the computing device 200 and allocating the tasks to be executed to the appropriate execution device 125 by the computing device 200 for execution.
As shown in FIG. 3, the task scheduling method 300 begins at step S310.
In step S310, a task execution request sent by the scheduling platform 110 is received, where the task execution request includes a task execution table, and a plurality of tasks to be executed in the task execution table. Here, it should be noted that the scheduling platform 110 may generate the task execution table based on the plurality of tasks to be executed, and when the scheduling platform 110 triggers the task execution request, the scheduling platform 110 sends the task execution request to the scheduling server 120 based on the task execution table in response to the request of the operator, so as to request the scheduling server 120 to execute the plurality of tasks to be executed in the task execution table.
According to an embodiment, the tasks to be executed are timed tasks, the operator can schedule a plurality of tasks to be executed on the scheduling platform 110 based on the execution period and the execution time of each task to be executed, and the scheduling platform 110 generates a task execution table with a tree structure based on the plurality of tasks to be executed and the execution period and the execution time corresponding to each task to be executed. That is to say, a plurality of tasks to be executed in the task execution table are arranged in a tree structure, and each task to be executed corresponds to one node in the tree structure. Here, the corresponding node of each task to be executed in the tree structure is determined by the execution period and execution time of the task to be executed.
According to the embodiment of the invention, in the task execution table with the tree structure, a dependency relationship exists between tasks to be executed based on respective corresponding node positions, specifically, the tasks to be executed of each node can respectively correspond to one or more upstream tasks, and the tasks to be executed of each node and the corresponding upstream tasks have a dependency relationship when triggered to execute.
It should be noted that each node in the tree structure may correspond to a parent node and one or more child nodes, and the task to be executed corresponding to the parent node of each node and the parent node of the parent node is an upstream task corresponding to the node. That is to say, the upstream task corresponding to the task to be executed by the node includes: and (4) from the root node to the tasks to be executed corresponding to all the nodes on the branch passed by the node.
In an embodiment according to the present invention, the computing device 200 may control the execution of the tasks based on the dependency relationship existing between the tasks to be executed in the task execution table, so as to implement the ordered scheduling of the tasks.
In step S320, the computing device 200 determines whether all the upstream tasks corresponding to each task to be executed in the task execution table are successfully executed. Here, when it is determined that all the upstream tasks of the to-be-executed tasks in the task execution table are successfully executed, the execution of the to-be-executed task is triggered.
Subsequently, in step S330, if it is determined that all the upstream tasks of any one of the tasks to be executed in the task execution table are successfully executed, the task to be executed is triggered, and a target execution device in the cluster 130 for executing the task to be executed is determined, so that the task to be executed is allocated to the target execution device for execution.
Specifically, the computing device 200 determines the target execution device based on the device information of the plurality of execution devices by acquiring an execution device list including a plurality of pieces of execution device information, the execution device list including device information corresponding to each execution device, respectively.
According to one embodiment, the device information of the execution devices in the execution device list includes information such as heartbeat, memory, CPU, number of tasks currently executed, and the like, but the present invention is not limited to the specific kind of the listed device information. Here, the computing device 200 can obtain the device information of all the executing devices in the cluster 130 by obtaining the executing device list, and the device information may reflect the current operating state and the task executing capability of the corresponding executing device, in other words, the computing device 200 may determine the operating state and the task executing capability of each executing device according to the device information. In this way, the computing device 200 may select a target execution device from the plurality of execution devices that best matches the current task to be executed by evaluating the current task to be executed and the ability to execute the task for each execution device.
It should be understood that when the task to be executed is triggered, the task to be executed is allocated to the currently most matched target execution device for execution, so that not only can the execution efficiency of the task be improved, but also the existing execution device resources can be reasonably utilized, load balancing is realized, and efficient and stable operation of a plurality of tasks is facilitated.
Finally, in step S340, the task to be executed is sent to the determined target execution device, so that the target execution device executes the task. It should be appreciated that when a task to be executed is sent to a target execution device, the task is initiated for execution.
According to an embodiment, whether all the upstream tasks corresponding to each task to be executed in the task execution table are successfully executed may be determined according to the following method:
registering dependent monitoring for each task to be executed in the task execution table so as to monitor the execution state of each task to be executed in the task execution table and judge whether each task to be executed is executed successfully.
When any task is determined to be successfully executed, triggering each other dependent monitor to judge: whether the task that is successfully executed is an upstream task of the corresponding task to be executed (i.e. the other tasks that rely on snooping itself). If the task is the upstream task, traversing each upstream task corresponding to the task to be executed (namely the task monitored by the dependent monitor) in the task execution table, and judging whether each upstream task is executed successfully.
Here, it should be noted that the dependent snooping is snooping registered to the task to be executed based on a dependency relationship between the task to be executed and the corresponding one or more upstream tasks. The execution of a dependent snoop triggering task depends on an upstream task that has a dependency relationship with the task. That is, based on registering dependent snooping for the to-be-executed task, whether the to-be-executed task can trigger execution depends on whether each upstream task is successfully executed.
According to one embodiment, after the task to be executed is sent to the target execution device in step S330 (task start), it is determined whether the task is successfully executed by the target execution device based on the dependent monitoring registered for the task. If the execution is successful, every other dependent snoop is triggered to determine: whether the task that is successfully executed is an upstream task of the corresponding task to be executed (i.e. the other tasks that rely on snooping itself).
And if the task to be executed fails to execute, sending an alarm message of the task execution failure to the scheduling platform 110 so that an operator of the scheduling platform 110 can timely process the task which fails to execute.
Therefore, according to the task scheduling method, the task execution table with the tree structure is created, and the computing device schedules the execution of a plurality of tasks to be executed based on the task execution table, so that the overall management of all the tasks to be executed is realized. In the task execution table with the tree structure, a dependency relationship exists between each task and an upstream task, the execution state of each task can be monitored by registering dependency monitoring for each task, and the triggering execution of each task depends on the upstream task having the dependency relationship with the task. Therefore, the tasks have dependence triggering relations and run relatively independently, overall management and ordered scheduling of a plurality of tasks to be executed are achieved, the change of each task does not affect the execution of other tasks, a large number of timing tasks can be scheduled and executed flexibly and efficiently, and the data synchronization of the distributed data system is more efficient.
A8, the task scheduling method according to A7, wherein the node corresponding to each task to be executed in the tree structure is suitable for being determined by the execution period and the execution time of the task to be executed.
A9, the task scheduling method according to any one of A1-A8, wherein the computing device is connected with a scheduling platform, and the scheduling platform is adapted to generate a task execution table based on a plurality of tasks to be executed, and send a task execution request to the computing device based on the task execution table.
B11, the task scheduling system according to B10, further comprising: and the scheduling platform is connected with the scheduling server and is suitable for generating a task execution table based on a plurality of tasks to be executed and sending a task execution request to the scheduling server based on the task execution table.
The various techniques described herein may be implemented in connection with hardware or software or, alternatively, with a combination of both. Thus, the methods and apparatus of the present invention, or certain aspects or portions thereof, may take the form of program code (i.e., instructions) embodied in tangible media, such as removable hard drives, U.S. disks, floppy disks, CD-ROMs, or any other machine-readable storage medium, wherein, when the program is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing the invention.
In the case of program code execution on programmable computers, the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device. Wherein the memory is configured to store program code; the processor is configured to execute the multilingual spam-text recognition method of the present invention according to instructions in said program code stored in the memory.
By way of example, and not limitation, readable media may comprise readable storage media and communication media. Readable storage media store information such as computer readable instructions, data structures, program modules or other data. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. Combinations of any of the above are also included within the scope of readable media.
In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems may also be used with examples of this invention. The required structure for constructing such a system will be apparent from the description above. Moreover, the present invention is not directed to any particular programming language. It is appreciated that a variety of programming languages may be used to implement the teachings of the present invention as described herein, and any descriptions of specific languages are provided above to disclose the best mode of the invention.
In the description provided herein, numerous specific details are set forth. It is understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.
Similarly, it should be appreciated that in the foregoing description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the disclosed method should not be interpreted as reflecting an intention that: that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of this invention.
Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in a device as described in this embodiment or alternatively may be located in one or more devices different from the devices in this example. The modules in the foregoing examples may be combined into one module or may be further divided into multiple sub-modules.
Those skilled in the art will appreciate that the modules in the device in an embodiment may be adaptively changed and disposed in one or more devices different from the embodiment. The modules or units or components of the embodiments may be combined into one module or unit or component, and furthermore they may be divided into a plurality of sub-modules or sub-units or sub-components. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and all of the processes or elements of any method or apparatus so disclosed, may be combined in any combination, except combinations where at least some of such features and/or processes or elements are mutually exclusive. Each feature disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.
Furthermore, those skilled in the art will appreciate that while some embodiments described herein include some features included in other embodiments, rather than other features, combinations of features of different embodiments are meant to be within the scope of the invention and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.
Furthermore, some of the described embodiments are described herein as a method or combination of method elements that can be performed by a processor of a computer system or by other means of performing the described functions. A processor having the necessary instructions for carrying out the method or method elements thus forms a means for carrying out the method or method elements. Further, the elements of the apparatus embodiments described herein are examples of the following apparatus: the apparatus is used to implement the functions performed by the elements for the purpose of carrying out the invention.
As used herein, unless otherwise specified the use of the ordinal adjectives "first", "second", "third", etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this description, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as described herein. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter. Accordingly, many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the appended claims. The present invention has been disclosed in an illustrative rather than a restrictive sense, and the scope of the present invention is defined by the appended claims.
Claims (10)
1. A method of task scheduling, performed in a computing device, the computing device being connected to a plurality of execution devices, the method comprising:
receiving a task execution request, wherein the task execution request comprises a task execution table, and the task execution table comprises a plurality of tasks to be executed;
judging whether the upstream tasks corresponding to each task to be executed in the task execution table are executed successfully or not;
if the upstream tasks of the tasks to be executed are determined to be successfully executed, triggering the tasks to be executed, and determining target execution equipment for executing the tasks to be executed; and
and sending the task to be executed to the target execution equipment so that the target execution equipment can execute the task.
2. The task scheduling method according to claim 1, wherein the step of determining a target execution device for executing the task to be executed comprises:
acquiring an execution device list comprising a plurality of pieces of execution device information, wherein the execution device list comprises device information respectively corresponding to the plurality of execution devices;
the target execution device is determined based on device information of the plurality of execution devices.
3. The task scheduling method according to claim 2, wherein the device information includes one or more of a heartbeat, a memory, a CPU, and a number of tasks currently executed of the execution device.
4. The task scheduling method according to any one of claims 1 to 3, wherein the step of determining whether the upstream tasks of the to-be-executed task in the task execution table are all executed successfully comprises:
registering dependency monitoring for each task to be executed to monitor each task to be executed in the task execution table and judge whether each task to be executed is executed successfully;
when determining that one task is successfully executed, triggering other dependent monitoring to judge whether the successfully executed task is an upstream task of the task to be executed corresponding to the other dependent monitoring;
if so, traversing each upstream task corresponding to the task to be executed in the task execution table to determine whether each upstream task is successfully executed.
5. The task scheduling method according to claim 4, wherein after the task to be executed is sent to the target execution device, the method further comprises the steps of:
judging whether the task is executed successfully;
and if the execution is successful, triggering other dependent monitoring to judge whether the task which is successfully executed is an upstream task of the task to be executed corresponding to the other dependent monitoring.
6. The task scheduling method of claim 5, wherein the step of determining whether the task was successfully executed further comprises:
and if the task fails to be executed, sending an alarm message of the task failure to the client.
7. The task scheduling method according to any one of claims 1 to 6, wherein the plurality of tasks to be executed in the task execution table are arranged in a tree structure, and each task to be executed corresponds to one node in the tree structure.
8. A task scheduling system comprising a scheduling cluster, the scheduling cluster comprising:
a scheduling server adapted to perform the task scheduling method of any one of claims 1-7; and
and each executing device is connected with the scheduling server and is suitable for executing the tasks to be executed distributed by the scheduling server.
9. A computing device, comprising:
at least one processor; and
a memory storing program instructions configured for execution by the at least one processor, the program instructions comprising instructions for performing the method of any of claims 1-7.
10. A readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011414208.XA CN112559143A (en) | 2020-12-04 | 2020-12-04 | Task scheduling method and system and computing device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011414208.XA CN112559143A (en) | 2020-12-04 | 2020-12-04 | Task scheduling method and system and computing device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112559143A true CN112559143A (en) | 2021-03-26 |
Family
ID=75058919
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011414208.XA Pending CN112559143A (en) | 2020-12-04 | 2020-12-04 | Task scheduling method and system and computing device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112559143A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115292021A (en) * | 2022-09-28 | 2022-11-04 | 江西萤火虫微电子科技有限公司 | Task scheduling method, system, electronic device and readable storage medium |
| CN115509721A (en) * | 2022-10-27 | 2022-12-23 | 中国兵器工业计算机应用技术研究所 | Data processing task cooperative control scheduling method and system |
| WO2023213118A1 (en) * | 2022-05-06 | 2023-11-09 | 北京快乐茄信息技术有限公司 | Task scheduling method and apparatus, and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110704173A (en) * | 2019-09-29 | 2020-01-17 | 深圳前海环融联易信息科技服务有限公司 | Task scheduling method, scheduling system, electronic device and computer storage medium |
| CN110858158A (en) * | 2018-08-23 | 2020-03-03 | 北京京东金融科技控股有限公司 | Distributed task scheduling method and device, electronic equipment and storage medium |
| CN111950988A (en) * | 2020-08-18 | 2020-11-17 | 北京字节跳动网络技术有限公司 | Distributed workflow scheduling method and device, storage medium and electronic equipment |
| CN111949386A (en) * | 2020-07-09 | 2020-11-17 | 北京齐尔布莱特科技有限公司 | Task scheduling method, system, computing device and readable storage medium |
-
2020
- 2020-12-04 CN CN202011414208.XA patent/CN112559143A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110858158A (en) * | 2018-08-23 | 2020-03-03 | 北京京东金融科技控股有限公司 | Distributed task scheduling method and device, electronic equipment and storage medium |
| CN110704173A (en) * | 2019-09-29 | 2020-01-17 | 深圳前海环融联易信息科技服务有限公司 | Task scheduling method, scheduling system, electronic device and computer storage medium |
| CN111949386A (en) * | 2020-07-09 | 2020-11-17 | 北京齐尔布莱特科技有限公司 | Task scheduling method, system, computing device and readable storage medium |
| CN111950988A (en) * | 2020-08-18 | 2020-11-17 | 北京字节跳动网络技术有限公司 | Distributed workflow scheduling method and device, storage medium and electronic equipment |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023213118A1 (en) * | 2022-05-06 | 2023-11-09 | 北京快乐茄信息技术有限公司 | Task scheduling method and apparatus, and device |
| CN115292021A (en) * | 2022-09-28 | 2022-11-04 | 江西萤火虫微电子科技有限公司 | Task scheduling method, system, electronic device and readable storage medium |
| CN115509721A (en) * | 2022-10-27 | 2022-12-23 | 中国兵器工业计算机应用技术研究所 | Data processing task cooperative control scheduling method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10430735B2 (en) | Designer tool for managing cloud computing services | |
| US10635437B1 (en) | Techniques to deploy an application as a cloud computing service | |
| US11016673B2 (en) | Optimizing serverless computing using a distributed computing framework | |
| CN112559143A (en) | Task scheduling method and system and computing device | |
| US10146599B2 (en) | System and method for a generic actor system container application | |
| JP2022008781A (en) | Decentralized training method, system, device, storage medium and program | |
| US20130268913A1 (en) | Managing application template artifacts in a networked computing environment | |
| CN109117252B (en) | Method and system for task processing based on container and container cluster management system | |
| WO2020215752A1 (en) | Graph computing method and device | |
| US20220171652A1 (en) | Distributed container image construction scheduling system and method | |
| EP4024761A1 (en) | Communication method and apparatus for multiple management domains | |
| US11625242B1 (en) | Automated generation of dependency hierarchy based on input and output requirements of information | |
| CN114168252A (en) | Information processing system and method, network scheme recommendation component and method | |
| CN109508193B (en) | Application deployment and operation method, device, terminal equipment and medium | |
| CN112367205B (en) | Processing method and scheduling system for HTTP scheduling request | |
| CN115617480A (en) | Task scheduling method, device and system and storage medium | |
| CN115658471A (en) | Test task scheduling method, test task execution method and test system | |
| US11366648B2 (en) | Compiling monoglot function compositions into a single entity | |
| CN114579250A (en) | Method, device and storage medium for constructing virtual cluster | |
| CN113126961B (en) | Pipeline processing method, device and storage medium | |
| CN115454450B (en) | Method and device for resource management of data job, electronic equipment and storage medium | |
| US11768704B2 (en) | Increase assignment effectiveness of kubernetes pods by reducing repetitive pod mis-scheduling | |
| CN114168280A (en) | Task scheduling method, computing device and storage medium | |
| CN118051487A (en) | Unified file processing method and system under micro-service architecture | |
| CN116800748A (en) | Message processing system, method, computing device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |