CN111338770A - Task scheduling method, server and computer readable storage medium - Google Patents
Task scheduling method, server and computer readable storage medium Download PDFInfo
- Publication number
- CN111338770A CN111338770A CN202010088428.1A CN202010088428A CN111338770A CN 111338770 A CN111338770 A CN 111338770A CN 202010088428 A CN202010088428 A CN 202010088428A CN 111338770 A CN111338770 A CN 111338770A
- Authority
- CN
- China
- Prior art keywords
- task
- scheduling
- processed
- cluster
- designated
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5083—Techniques for rebalancing the load in a distributed system
Abstract
The embodiment of the invention relates to the technical field of internet information, and discloses a task scheduling method, a server and a computer readable storage medium. The task scheduling method is applied to a scheduling server and comprises the following steps: if the trigger condition of the task to be processed is met, searching a task queue corresponding to the designated cluster according to the designated cluster configured by the task to be processed; distributing the tasks to be processed to a task queue, and enabling a scheduling agent of the designated cluster to capture the tasks to be processed in the task queue and lift the tasks to be processed to the designated cluster according to the load information of the designated cluster, and feeding back a processing result of the designated cluster for processing the tasks to be processed; and acquiring a processing result returned by the scheduling agent of the specified cluster. The task scheduling method of the embodiment can reasonably coordinate and schedule the designated cluster to process the to-be-processed task, and improve the task scheduling efficiency.
Description
Technical Field
The embodiment of the invention relates to the technical field of internet information, in particular to a task scheduling method, a server and a computer readable storage medium.
Background
The development languages supported by the technical framework of the big data platform are various, and many different types of programs (tasks) are generated to run on the big data platform, and the task types can be various programming languages such as: MapReduce, Hive, Pig, Spark, Java, Shell, Python, etc. Since tasks or task flows on large data platforms need to be triggered by the timing schedule of the scheduling system. The big data platform is generally composed of a plurality of sets of big data computing clusters, each computing cluster comprises a plurality of servers for processing the same service, each computing cluster corresponds to a scheduling system, and the scheduling system is used for managing tasks uploaded by the computing cluster.
The inventors found that at least the following problems exist in the related art: each current computing cluster corresponds to one scheduling system, and when one task needs to be processed by a plurality of clusters together, due to the fact that the scheduling systems of the plurality of computing clusters appear on the big data platform, the tasks cannot be uniformly coordinated and scheduled, and the task scheduling efficiency is affected.
Disclosure of Invention
The embodiment of the invention aims to provide a task scheduling method, a server and a computer readable storage medium, so that a designated cluster can be reasonably coordinated and scheduled to process a task to be processed, and the task scheduling efficiency is improved.
In order to solve the above technical problem, an embodiment of the present invention provides a method for task scheduling, which is applied to a scheduling server and includes: if the trigger condition of the task to be processed is met, searching a task queue corresponding to the designated cluster according to the designated cluster configured by the task to be processed; distributing the tasks to be processed to a task queue, and enabling a scheduling agent of the designated cluster to capture the tasks to be processed in the task queue and lift the tasks to be processed to the designated cluster according to the load information of the designated cluster, and feeding back a processing result of the designated cluster for processing the tasks to be processed; and acquiring a processing result returned by the scheduling agent of the specified cluster.
The embodiment of the invention also provides a task scheduling method, which is applied to the scheduling agents, wherein the scheduling agents are respectively in communication connection with the scheduling server and the corresponding designated clusters, and the task scheduling method comprises the following steps: capturing a task to be processed in an appointed task queue corresponding to an appointed cluster according to the load information of the appointed cluster; submitting the tasks to be processed to the designated cluster for the designated cluster to process the tasks to be processed, and returning a processing result to the scheduling agent; and feeding back the received processing result to the scheduling server.
The embodiment of the invention also provides a task scheduling method, which is applied to a configuration center, the configuration center is respectively in communication connection with the scheduling server and the scheduling agent, the scheduling server is used for executing a corresponding task scheduling method, the scheduling agent is used for executing a corresponding task scheduling method, and the task scheduling method comprises the following steps: searching a configuration file corresponding to the configuration information according to the configuration information of the task flow sent by the scheduling server; feeding back a configuration file to a scheduling server; according to the acquisition request sent by the scheduling agent, the acquisition request comprises: scheduling information of the designated cluster corresponding to the agent; and searching the running environment information of the designated cluster according to the designated cluster information and returning the running environment information to the scheduling agent.
An embodiment of the present invention further provides a server, including: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executable by the at least one processor to enable the at least one processor to perform a method of task scheduling corresponding to a scheduling service, or to perform a method of task scheduling corresponding to a scheduling agent, or to perform a method of task scheduling corresponding to a configuration center.
Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, where the computer program, when executed by a processor, implements a method for task scheduling corresponding to a scheduling service, or implements a method for task scheduling corresponding to a scheduling agent, or implements a method for task scheduling corresponding to a configuration center.
Compared with the prior art, the method and the device have the advantages that after the triggering condition of the task to be processed is detected to be met, the task queue corresponding to the specified cluster is searched according to the specified cluster configured with the task to be processed, the task to be processed is distributed to the searched task queue, the corresponding task queue is arranged for the specified cluster, so that the task to be processed is not required to be directly sent to the specified cluster by the scheduling server and is directly distributed to the task queue, the task to be processed in the task queue cannot directly reach the specified cluster due to the buffer function of the task queue, and the problem that the task processing fails due to the fact that the specified cluster cannot process high-concurrency task to be processed is solved; the task queues correspond to the designated clusters, so that the scheduling server can distribute the tasks to be processed with different service types to the corresponding task queues according to the designated clusters with the configured tasks to be processed, the scheduling agents of the designated clusters capture the tasks from the corresponding task queues, coordinated scheduling of multiple clusters is realized, meanwhile, the scheduling agents of the designated clusters capture the tasks to be processed in the task queues based on the load information of the designated clusters, and the designated clusters do not have the function of actively capturing the tasks, so that the tasks can be reasonably captured and processed according to the loads of the designated clusters by setting the corresponding scheduling agents, and the task scheduling efficiency is improved.
In addition, before detecting whether the triggering condition of the task to be processed is met, the method for task scheduling further comprises the following steps: receiving a creation request for creating a task flow, and creating the task flow, wherein the creation request comprises: configuration information of the task flow and information of the required cluster; and dividing the task flow into N to-be-processed tasks according to the information of the required clusters, wherein each to-be-processed task is configured with a corresponding appointed cluster, and the value of N is the same as the number of the required clusters. The task flow comprises N to-be-processed tasks, the task flow is divided into N to-be-processed tasks according to the information of the clusters required by the task flow, and each to-be-processed task is configured with a corresponding appointed cluster, so that the scheduling server can schedule and process the task flow configured with different clusters, and the efficiency of scheduling multiple clusters of the task flow is further improved.
In addition, after dividing the task flow into N tasks to be processed, the method for task scheduling further includes: acquiring a dependency relationship between each task to be processed; and setting a corresponding trigger condition for each task to be processed according to the dependency relationship. The trigger condition of the task to be processed in the task flow is set based on the dependency relationship, so that the task to be processed can be dispatched in a related manner, the designated cluster can be dispatched reasonably, and the cluster dispatching efficiency is improved.
In addition, the task scheduling method further comprises the following steps: the task scheduling method further comprises the following steps: sending configuration information of the task flow to a configuration center for the configuration center to search configuration files of the task flow and sending the configuration files to a scheduling server, wherein the configuration files comprise: executing script information of the task to be processed; and receiving a configuration file fed back by the configuration center, and configuring the task to be processed according to the configuration file. The scheduling server only stores the configuration information, and the corresponding configuration file is stored in the configuration center, so that the code modification of the scheduling server is not needed when the configuration file is modified subsequently, and the use flexibility of the scheduling server is improved.
Drawings
One or more embodiments are illustrated by way of example in the accompanying drawings, which correspond to the figures in which like reference numerals refer to similar elements and which are not to scale unless otherwise specified.
Fig. 1 is a detailed flowchart of a task scheduling method according to a first embodiment of the present invention;
FIG. 2 is a diagram of a specific implementation of configuring a task to be processed according to a second embodiment of the present invention;
FIG. 3 is a frame diagram of task scheduling according to a second embodiment of the present invention;
FIG. 4 is a detailed flowchart of a task scheduling method according to a third embodiment of the present invention;
FIG. 5 is a detailed flowchart of a task scheduling method according to a fourth embodiment of the present invention;
fig. 6 is a specific flowchart of a task scheduling method according to a fifth embodiment of the present invention;
fig. 7 is a schematic structural diagram of a server according to a sixth embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be appreciated by those of ordinary skill in the art that numerous technical details are set forth in order to provide a better understanding of the present application in various embodiments of the present invention. However, the technical solution claimed in the present application can be implemented without these technical details and various changes and modifications based on the following embodiments.
The following embodiments are divided for convenience of description, and should not constitute any limitation to the specific implementation manner of the present invention, and the embodiments may be mutually incorporated and referred to without contradiction.
The inventor finds that in the current related task scheduling method, the same service is processed by one cluster, each cluster corresponds to one scheduling system, and the respective scheduling system of each cluster independently manages task scheduling on the cluster, but for the scheduling platform, because there are multiple clusters and each cluster has a respective corresponding scheduling system, the scheduling platform has multiple cluster scheduling systems, and the platform cannot reasonably schedule the clusters.
The first embodiment of the invention relates to a task scheduling method. The task scheduling method is applied to a scheduling server. The specific flow of the task scheduling method is shown in fig. 1.
Step 101: and if the trigger condition of the task to be processed is met, searching a task queue corresponding to the specified cluster according to the specified cluster configured by the task to be processed.
The scheduling server may include a task creating module, configured to create a new to-be-processed task, and may provide an interface for creating a task for a user, so as to facilitate the user to create the new to-be-processed task, for example, the scheduling server may provide a visual operation interface, receive, through the visual operation interface, a creation request for creating a task input by the user, and create the to-be-processed task according to the creation request, where the creation request includes: the configuration information of the task to be processed and the information of the required cluster may include a configuration file, which may be an execution script of the task to be processed, and a task type, where the task type may be a type of each programming language, for example: MapReduce, Hive, Pig, Spark, Java, Shell, Python, etc. The configuration file can be stored in a database of the scheduling server, so that the configuration file corresponding to the configuration information can be acquired from the database according to the configuration information. The information of the cluster required by the task to be processed comprises the following steps: the pending task is configured.
The trigger condition of the task to be processed may be set by a user, and the trigger condition may be: reaching a preset scheduling time, or receiving an execution result based on other programs. The scheduling server stores a corresponding relation between the designated cluster and the task queue, and after detecting that the trigger condition of the task to be processed is triggered, the scheduling server can search the task queue corresponding to the designated cluster according to the designated cluster configured by the task to be processed based on the corresponding relation.
It is worth mentioning that each designated cluster has a unique corresponding task queue, and different task queues of the scheduling server are used for processing respective corresponding services because different clusters process different services, and scheduling of different clusters can be realized through the task queues.
In another example, before detecting whether a trigger condition of a task to be processed is satisfied, the dispatch server receives a create request to create a task stream, the create request including: configuration information of the task flow and information of the required cluster; and dividing the task flow into N to-be-processed tasks according to the information of the required clusters, wherein each to-be-processed task is configured with a corresponding appointed cluster, and the value of N is the same as the number of the required clusters.
Specifically, similar to creating a single task to be processed, the scheduling server may create a task flow, receive configuration information of the task flow and information of a cluster required by the task flow, and determine a configuration file of the task flow in a database according to the configuration information of the task flow, where the configuration file may include: cluster operating environment information, or interface information of external query service, etc. The information of the required clusters may be identification information of each required cluster of the task flow, for example, the cluster information of the task flow a includes: identification information of cluster 1, identification information of cluster 2. Because different cluster processing is required in the task flow, the task flow can be divided into N to-be-processed tasks according to the information of the required clusters, the value of N is the same as the number of the required clusters, and each to-be-processed task is configured with a corresponding designated cluster. For example, the number of clusters required is 2, and the task a is split into two tasks to be processed, which are: pending task a 1: cluster 1; pending task a 2: cluster 2.
In one example, after dividing the task stream into N to-be-processed tasks, the method for task scheduling further includes: acquiring a dependency relationship between each task to be processed; and setting a corresponding trigger condition for each task to be processed according to the dependency relationship.
Specifically, the dependency relationship between each task to be processed may be input by a user, or may be determined according to a trigger time point of a task flow set by the user. The trigger condition of the task flow without pre-dependency relationship may be a preset trigger condition of the task flow, for example, the trigger condition of the task flow a is triggered at time T1, and the task flow is split into the task a1 to be processed and the task a2 to be processed, where the task a1 to be processed has a dependency relationship with the task a2 to be processed and the task a2 to be processed needs to be calculated based on the processing result of the task a1 to be processed, then the trigger condition of the task a2 may be set as: when the instruction that the execution of the to-be-processed task a1 is completed is received and the to-be-processed task a1 has no pre-dependency relationship, the trigger condition of the to-be-processed task a1 may be set to trigger at time T1.
Step 102: and distributing the tasks to be processed to the task queues, so that the scheduling agent of the designated cluster captures the tasks to be processed in the task queues according to the load information of the designated cluster, and mentions the tasks to be processed to the designated cluster, and feeds back the processing result of the designated cluster for processing the tasks to be processed.
Specifically, each designated cluster may include a plurality of scheduling agents, each scheduling agent may be configured to submit a task for the designated cluster, and the scheduling agent may be a server and is connected to the scheduling server and the cluster, respectively; the scheduling agent need not be located within the cluster in order to improve the applicability of the scheduling agent.
After the triggering condition of the task to be processed is met, based on the configured appointed cluster of the task to be processed, the task queue corresponding to the appointed cluster is found, and the task to be processed is distributed to the task queue. And the scheduling agent of the designated cluster captures the tasks to be processed in the task queue according to the load information of the designated cluster, and then the tasks to be processed are referred to the designated cluster, and the processing result of the designated cluster for processing the tasks to be processed is fed back. The load information can be provided by the cluster, and the load information can also be determined by summarizing the time consumed by the cluster for processing the tasks to be processed at regular time.
In one example, the scheduling server may determine load information of each designated cluster according to the number of to-be-processed tasks captured in a task queue corresponding to the designated cluster; therefore, whether the designated cluster is healthy or not can be judged according to the load information of the designated cluster.
Step 103: and acquiring a processing result returned by the scheduling agent of the specified cluster.
Specifically, for scheduling of task flows: receiving a processing result of a previous task to be processed returned by the scheduling agent, if the triggering condition of a next task to be processed is that the processing result of the previous task to be processed is received, then triggering to search a next task queue corresponding to the specified cluster of the next task to be processed according to the specified cluster of the next task to be processed, distributing the next task to be processed to the next task queue, continuing to perform task scheduling until all tasks to be processed are completed, obtaining a processing result of a task flow, and outputting the processing result of the task flow.
If the processing result received by the scheduling server side has no dependency relationship with other tasks to be processed, the processing result can be directly output.
Compared with the prior art, the method and the device have the advantages that after the triggering condition of the task to be processed is detected to be met, the task queue corresponding to the specified cluster is searched according to the specified cluster configured with the task to be processed, the task to be processed is distributed to the searched task queue, the corresponding task queue is arranged for the specified cluster, so that the task to be processed is not required to be directly sent to the specified cluster by the scheduling server and is directly distributed to the task queue, the task to be processed in the task queue cannot directly reach the specified cluster due to the buffer function of the task queue, and the problem that the task processing fails due to the fact that the specified cluster cannot process high-concurrency task to be processed is solved; the task queues correspond to the designated clusters, so that the scheduling server can distribute the tasks to be processed with different service types to the corresponding task queues according to the designated clusters with the configured tasks to be processed, the scheduling agents of the designated clusters capture the tasks from the corresponding task queues, coordinated scheduling of multiple clusters is realized, meanwhile, the scheduling agents of the designated clusters capture the tasks to be processed in the task queues based on the load information of the designated clusters, and the designated clusters do not have the function of actively capturing the tasks, so that the tasks can be reasonably captured and processed according to the loads of the designated clusters by setting the corresponding scheduling agents, and the task scheduling efficiency is improved.
A second embodiment of the invention relates to a method of task scheduling. The second embodiment is a further improvement of the first embodiment, and the main improvements are as follows: in the second embodiment of the present invention, the scheduling server sends the configuration information of the task flow to the configuration center, and obtains the configuration file returned by the configuration center, and a specific process of configuring the task to be processed in the task scheduling method is shown in fig. 2.
Step 201: sending the configuration information of the task flow to a configuration center for the configuration center to search the configuration file of the task flow and feed back the configuration file to a scheduling server, wherein the configuration file comprises: and executing script information of the task to be processed.
Specifically, after the task flow is created, the configuration information of the task flow is sent to a configuration center, after the configuration center acquires the configuration information, the configuration center searches for a corresponding configuration file according to the index information of the configuration file, and returns the configuration file to the scheduling server, and the configuration file can be sent to a scheduling agent after receiving an acquisition request of the scheduling center. The configuration file includes: and executing script information of the task to be processed.
It is worth mentioning that the configuration file is placed in the configuration center, and the server to be scheduled is not stored, so that the configuration file can be modified subsequently, and the scheduling server is not influenced; the storage of the configuration file is isolated from the scheduling server, and the subsequent tuning processing of the configuration file is facilitated.
Step 202: and receiving the configuration file fed back by the configuration center.
Step 203: and configuring the tasks to be processed according to the configuration file.
Specifically, the configuration file includes execution script information of the to-be-processed task, and the to-be-processed task is configured according to the to-be-processed script information.
It should be noted that the steps 201 to 203 may be executed after the task stream is created.
The configuration center may be a single server, and the whole task scheduling process is described below with reference to fig. 3. Fig. 3 is a schematic diagram of the whole task scheduling framework, in fig. 3, the configuration center is denoted by reference numeral 40, the scheduling server is denoted by reference numeral 10, and the scheduling server 10 includes a plurality of modules, such as: a task management module (not shown in FIG. 3), a scheduling module, and a task queue; the task management module is used for creating a new task to be processed, or the task management module creates a task stream and divides the task stream into n tasks to be processed, wherein n is an integer larger than 0. The scheduling module is used for searching a task queue of the designated cluster corresponding to the task to be processed and putting the task to be processed into the task queue. The scheduling agent is shown at 20 and the designated cluster is shown at 30, with the DataNode representing the compute node in the designated cluster.
The scheduling server 10 sends the configuration information of the task 1 to be processed to a configuration center, the configuration center searches for a corresponding configuration file according to the configuration information, and returns the configuration file to the scheduling server, and when a scheduling module in the scheduling server detects that a trigger condition of the task 1 to be processed is met, the scheduling module searches for a task queue 1 corresponding to an assigned cluster according to the assigned cluster 31 corresponding to the task 1 to be processed, and distributes the task to be processed to the task queue 1.
The appointed cluster 1 is correspondingly provided with K scheduling agents, each scheduling agent is respectively in communication connection with the scheduling server side and the configuration center, after the scheduling agents are started, the scheduling agents acquire the operating environment information of the appointed cluster from the configuration center, and the scheduling agents adjust the current operating environment according to the acquired operating environment information. The scheduling agent 1 captures the to-be-processed task 1 from the task queue 1 according to the load information of the designated cluster 31, submits the to-be-processed task 1 to the designated cluster 31 for processing, obtains a processing result of the designated cluster 31, and returns the processing result to the scheduling server 10.
According to the task scheduling method provided by the embodiment, the scheduling server only stores the configuration information, and the corresponding configuration file is stored in the configuration center, so that the code modification of the scheduling server is not required when the configuration file is modified subsequently, and the use flexibility of the scheduling server is improved.
The steps of the above methods are divided for clarity, and the implementation may be combined into one step or split some steps, and the steps are divided into multiple steps, so long as the same logical relationship is included, which are all within the protection scope of the present patent; it is within the scope of the patent to add insignificant modifications to the algorithms or processes or to introduce insignificant design changes to the core design without changing the algorithms or processes.
The third embodiment of the invention relates to a task scheduling method, which is applied to a scheduling agent, wherein the scheduling agent is respectively in communication connection with a scheduling server and a corresponding designated cluster. The scheduling agent may be a server; fig. 4 shows a specific flow of the task scheduling method according to the present embodiment.
Step 301: and capturing the tasks to be processed in the assigned task queue corresponding to the assigned cluster according to the load information of the assigned cluster.
Specifically, each designated cluster may include a plurality of scheduling agents, and each scheduling agent may acquire load information of the designated cluster after being started, and grab a to-be-processed task in a designated task queue corresponding to the designated cluster.
The scheduling agent may submit a task to be processed to the corresponding designated cluster each time, and thus the scheduling agent may grab the task from the task queue and submit the task to the designated cluster while in an idle state. The scheduling agent may also grab the pending task from the task queue and submit to the designated cluster at preset time intervals.
Step 302: and submitting the tasks to be processed to the designated cluster for the designated cluster to process the tasks to be processed, and returning a processing result to the scheduling agent.
And adjusting the current operating environment according to the operating environment information of the designated cluster, and submitting the tasks to be processed to the designated cluster according to the interface information of the designated cluster.
In one example, if the processing result is a failure, resubmitting the to-be-processed task to the designated cluster according to a preset rule, where the preset rule includes: the time length from the last submission time reaches the preset time length, or the load in the designated cluster reaches the preset load.
Specifically, if the received processing result is a failure, the processing may fail due to a large load of the designated cluster, so that the time from the last submission time reaches a preset time, or the task to be processed is resubmitted after the load in the designated cluster reaches the preset load.
It is worth mentioning that, because the task flow includes a plurality of to-be-processed tasks, if a certain to-be-processed task fails to be processed, the processing of the whole task flow fails, and by resubmitting the to-be-processed task, only the to-be-processed task needs to be processed, so that the processing resources are saved, and the processing efficiency of the task flow is improved.
Step 303: and feeding back the received processing result to the scheduling server.
In the task scheduling method provided in this embodiment, the scheduling agent obtains the to-be-processed tasks from the task queue, so that the designated cluster can process each to-be-processed task in a load-balanced manner, and the efficiency of processing the tasks by the cluster is improved.
The fourth embodiment of the invention relates to a task scheduling method. The fourth embodiment is a further improvement of the third embodiment, and the main improvement lies in that: in a fourth embodiment of the present invention, the scheduling agent is communicatively coupled to a configuration center to obtain the operating environment information from the configuration center. The task scheduling method further includes a flow shown in fig. 5.
Step 401: and sending an acquisition request for acquiring the operating environment of the specified cluster to the configuration center, wherein the acquisition request comprises the information of the specified cluster.
Specifically, the scheduling agent is in communication connection with a configuration center, and the configuration center stores configuration files of various tasks to be processed and also stores operating environment information of various designated clusters. The configuration center independently stores the configuration files and the operating environment information of each designated cluster, so that the configuration files and the operating environment information can be conveniently modified without influencing the operation of the scheduling server.
After the scheduling agent is started, the scheduling agent sends an acquisition request to the configuration center, where the acquisition request includes information of a designated cluster corresponding to the scheduling agent, and may be, for example, identification information of the designated cluster. The scheduling agent may obtain the information of the operating environment only once after being started, or may obtain the operating environment information of the designated cluster again after receiving an instruction for updating the operating environment information sent by the configuration center. The scheduling agent may be manually started by a user, or may be automatically started after a trigger condition for starting is reached. The triggering condition for starting can receive a power-on specification; reaching the preset time.
Step 402: and receiving the running environment information of the designated cluster sent by the configuration center.
Step 403: and adjusting the current operating environment according to the operating environment information.
Specifically, the current operating environment is adjusted according to the received operating environment information. It will be appreciated that the operating environment information of each scheduling agent of the same designated cluster is the same.
Step 404: and capturing the tasks to be processed in the assigned task queue corresponding to the assigned cluster according to the load information of the assigned cluster.
Step 405: and submitting the tasks to be processed to the designated cluster for the designated cluster to process the tasks to be processed, and returning a processing result to the scheduling agent.
Step 406: and feeding back the received processing result to the scheduling server.
Since the second embodiment corresponds to the present embodiment, the present embodiment can be implemented in cooperation with the second embodiment. The related technical details mentioned in the second embodiment are still valid in this embodiment, and the technical effects that can be achieved in the second embodiment can also be achieved in this embodiment, and are not described herein again in order to reduce the repetition. Accordingly, the related-art details mentioned in the present embodiment can also be applied to the second embodiment.
A fifth embodiment of the present invention relates to a method for task scheduling, where the method for task scheduling is applied to a configuration center, the configuration center is in communication connection with a scheduling server and a scheduling agent, the scheduling server is configured to execute the method for task scheduling in the first embodiment or the second embodiment, and the scheduling agent is configured to execute the method for task scheduling in the third embodiment or the fourth embodiment. The specific flow of the task scheduling method is shown in fig. 6.
Step 501: searching a configuration file corresponding to the configuration information according to the configuration information of the task flow sent by the scheduling server; and feeding back the configuration file to the scheduling server.
Specifically, the configuration center stores configuration files, receives configuration information of the task flow, finds a corresponding configuration file through the configuration information, and feeds back the configuration file to the scheduling server, where the configuration file includes: and executing script information of the task to be processed.
Step 502: according to the acquisition request sent by the scheduling agent, the acquisition request comprises: scheduling information of the designated cluster corresponding to the agent; and searching the running environment information of the designated cluster according to the designated cluster information and returning the running environment information to the scheduling agent.
Specifically, the configuration center further stores the operating environment information of each designated cluster and the identification information of the designated cluster, the operating environment information of the designated cluster can be found through the acquisition request, the operating environment information is returned to the scheduling agent, and the scheduling agent adjusts the current operating environment according to the received operating environment information.
A sixth embodiment of the present invention relates to a server having a configuration as shown in fig. 7, including: at least one processor 601; and a memory 602 communicatively coupled to the at least one processor 601; the memory 602 stores instructions executable by the at least one processor 601, and the instructions are executed by the at least one processor 601 to enable the at least one processor 601 to perform the method for task scheduling in the first embodiment or the second embodiment, or to perform the method for task scheduling in the third embodiment or the fourth embodiment, or to perform the method for task scheduling in the fifth embodiment.
Where the memory 602 and the processor 601 are connected by a bus, the bus may comprise any number of interconnected buses and bridges, the buses linking one or more of the various circuits of the processor 601 and the memory 602 together. The bus may also link various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor 601 is transmitted over a wireless medium via an antenna, which further receives the data and transmits the data to the processor 601.
The processor 601 is responsible for managing the bus and general processing and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And the memory may be used to store data used by the processor 601 in performing operations.
A seventh embodiment of the present invention relates to a computer-readable storage medium storing a computer program which, when executed by a processor, implements the method for task scheduling of the first embodiment or the second embodiment, or implements the method for task scheduling as in the third embodiment or the fourth embodiment, or implements the method for task scheduling of the fifth embodiment.
Those skilled in the art can understand that all or part of the steps in the method of the foregoing embodiments may be implemented by a program to instruct related hardware, where the program is stored in a storage medium and includes several instructions to enable a device (which may be a single chip, a chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
It will be understood by those of ordinary skill in the art that the foregoing embodiments are specific examples for carrying out the invention, and that various changes in form and details may be made therein without departing from the spirit and scope of the invention in practice.
Claims (10)
1. A task scheduling method is applied to a scheduling server and comprises the following steps:
if the trigger condition of the task to be processed is met, searching a task queue corresponding to the specified cluster according to the specified cluster configured by the task to be processed;
distributing the tasks to be processed to the task queues, enabling the scheduling agents of the designated clusters to capture the tasks to be processed in the task queues according to the load information of the designated clusters, lifting the tasks to be processed to the designated clusters, and feeding back the processing results of the designated clusters for processing the tasks to be processed;
and acquiring a processing result returned by the scheduling agent of the specified cluster.
2. The method of task scheduling according to claim 1, wherein before detecting whether the trigger condition of the task to be processed is satisfied, the method of task scheduling further comprises:
receiving a creation request for creating a task flow, and creating the task flow, wherein the creation request comprises: the configuration information of the task flow and the information of the required cluster;
and dividing the task flow into N to-be-processed tasks according to the information of the required clusters, wherein each to-be-processed task is configured with a corresponding appointed cluster, and the value of N is the same as the number of the required clusters.
3. The method according to claim 2, wherein after the dividing the task stream into N number of the tasks to be processed, the method further comprises:
acquiring a dependency relationship between each task to be processed;
and setting a corresponding trigger condition for each task to be processed according to the dependency relationship.
4. The method of task scheduling according to claim 2, further comprising:
sending the configuration information of the task flow to a configuration center, so that the configuration center searches a configuration file of the task flow and feeds the configuration file back to the scheduling server, wherein the configuration file comprises: executing script information of the task to be processed;
receiving the configuration file fed back by the configuration center;
and configuring the tasks to be processed according to the configuration file.
5. A task scheduling method is applied to scheduling agents which are respectively in communication connection with a scheduling server and a corresponding designated cluster, and comprises the following steps:
capturing a task to be processed in a specified task queue corresponding to the specified cluster according to the load information of the specified cluster;
submitting the task to be processed to the designated cluster, allowing the designated cluster to process the task to be processed, and returning a processing result to the scheduling agent;
and feeding back the received processing result to the scheduling server.
6. The method of task scheduling according to claim 5, further comprising:
if the processing result is failure, resubmitting the task to be processed to the designated cluster according to a preset rule, wherein the preset rule comprises: and the time length from the last submission time reaches the preset time length, or the load in the appointed cluster reaches the preset load.
7. A method for task scheduling according to claim 5 or 6 wherein the scheduling agent is communicatively coupled to a configuration center;
before the capturing the to-be-processed task in the designated task queue corresponding to the designated cluster, the method for task scheduling further includes:
sending an acquisition request for acquiring the operating environment of the designated cluster to the configuration center, wherein the acquisition request comprises the information of the designated cluster;
receiving the running environment information of the designated cluster sent by the configuration center;
and adjusting the current operating environment according to the operating environment information.
8. A method for task scheduling, which is applied to a configuration center, the configuration center is respectively connected to a scheduling server and a scheduling agent in communication, the scheduling server is configured to perform the method for task scheduling according to claim 4, the scheduling agent is configured to perform the method for task scheduling according to claim 7, and the method for task scheduling includes:
searching a configuration file corresponding to the configuration information according to the configuration information of the task flow sent by the scheduling server; feeding back the configuration file to the scheduling server;
according to the acquisition request sent by the scheduling agent, the acquisition request comprises: information of an assigned cluster corresponding to the scheduling agent; and searching the running environment information of the designated cluster according to the designated cluster information and returning the running environment information to the scheduling agent.
9. A server, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method of task scheduling according to any one of claims 1 to 4, or to perform a method of task scheduling according to any one of claims 5 to 7, or to perform a method of task scheduling according to claim 8.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method for task scheduling according to one of claims 1 to 4, or carries out the method for task scheduling according to one of claims 5 to 7, or carries out the method for task scheduling according to claim 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010088428.1A CN111338770A (en) | 2020-02-12 | 2020-02-12 | Task scheduling method, server and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010088428.1A CN111338770A (en) | 2020-02-12 | 2020-02-12 | Task scheduling method, server and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111338770A true CN111338770A (en) | 2020-06-26 |
Family
ID=71185166
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010088428.1A Pending CN111338770A (en) | 2020-02-12 | 2020-02-12 | Task scheduling method, server and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111338770A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111913791A (en) * | 2020-07-02 | 2020-11-10 | 北京和瑞精准医学检验实验室有限公司 | Task scheduling method, device, equipment and computer readable storage medium |
| CN112307046A (en) * | 2020-11-26 | 2021-02-02 | 北京金堤征信服务有限公司 | Data acquisition method and device, computer readable storage medium and electronic equipment |
| CN113268318A (en) * | 2021-04-07 | 2021-08-17 | 北京思特奇信息技术股份有限公司 | Task scheduling method and distributed system |
| CN113282395A (en) * | 2021-06-09 | 2021-08-20 | 中国农业银行股份有限公司 | Redis-based job request scheduling method, device, equipment and medium |
| CN113485815A (en) * | 2021-07-27 | 2021-10-08 | 中国银行股份有限公司 | Job batch processing system, method, device, storage medium and electronic equipment |
| CN113515358A (en) * | 2021-04-30 | 2021-10-19 | 北京奇艺世纪科技有限公司 | Task scheduling method and device, electronic equipment and storage medium |
| CN113986707A (en) * | 2021-11-03 | 2022-01-28 | 北京科杰科技有限公司 | Method for monitoring and controlling slow SQL based on big data kudu partition |
| CN115297122A (en) * | 2022-09-29 | 2022-11-04 | 数字江西科技有限公司 | Government affair operation and maintenance method and system based on load automatic monitoring |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102904746A (en) * | 2012-09-13 | 2013-01-30 | 曙光信息产业(北京)有限公司 | Cluster task self-dispatching system and method based on information mechanism |
| CN109408205A (en) * | 2017-08-16 | 2019-03-01 | 北京京东尚科信息技术有限公司 | Method for scheduling task and device based on hadoop cluster |
| CN109614241A (en) * | 2018-12-17 | 2019-04-12 | 杭州玳数科技有限公司 | The method and system of more cluster multi-tenant resource isolations are realized based on Yarn queue |
| CN110362390A (en) * | 2019-06-06 | 2019-10-22 | 银江股份有限公司 | A kind of distributed data integrated operations dispatching method and device |
| CN110704186A (en) * | 2019-09-25 | 2020-01-17 | 国家计算机网络与信息安全管理中心 | Computing resource allocation method and device based on hybrid distribution architecture and storage medium |
| CN110781007A (en) * | 2019-10-31 | 2020-02-11 | 广州市网星信息技术有限公司 | Task processing method, device, server, client, system and storage medium |
-
2020
- 2020-02-12 CN CN202010088428.1A patent/CN111338770A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102904746A (en) * | 2012-09-13 | 2013-01-30 | 曙光信息产业(北京)有限公司 | Cluster task self-dispatching system and method based on information mechanism |
| CN109408205A (en) * | 2017-08-16 | 2019-03-01 | 北京京东尚科信息技术有限公司 | Method for scheduling task and device based on hadoop cluster |
| CN109614241A (en) * | 2018-12-17 | 2019-04-12 | 杭州玳数科技有限公司 | The method and system of more cluster multi-tenant resource isolations are realized based on Yarn queue |
| CN110362390A (en) * | 2019-06-06 | 2019-10-22 | 银江股份有限公司 | A kind of distributed data integrated operations dispatching method and device |
| CN110704186A (en) * | 2019-09-25 | 2020-01-17 | 国家计算机网络与信息安全管理中心 | Computing resource allocation method and device based on hybrid distribution architecture and storage medium |
| CN110781007A (en) * | 2019-10-31 | 2020-02-11 | 广州市网星信息技术有限公司 | Task processing method, device, server, client, system and storage medium |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111913791A (en) * | 2020-07-02 | 2020-11-10 | 北京和瑞精准医学检验实验室有限公司 | Task scheduling method, device, equipment and computer readable storage medium |
| CN111913791B (en) * | 2020-07-02 | 2023-10-13 | 北京和瑞精湛医学检验实验室有限公司 | Task scheduling method, device, equipment and computer readable storage medium |
| CN112307046A (en) * | 2020-11-26 | 2021-02-02 | 北京金堤征信服务有限公司 | Data acquisition method and device, computer readable storage medium and electronic equipment |
| CN113268318A (en) * | 2021-04-07 | 2021-08-17 | 北京思特奇信息技术股份有限公司 | Task scheduling method and distributed system |
| CN113515358A (en) * | 2021-04-30 | 2021-10-19 | 北京奇艺世纪科技有限公司 | Task scheduling method and device, electronic equipment and storage medium |
| CN113515358B (en) * | 2021-04-30 | 2024-04-12 | 北京奇艺世纪科技有限公司 | Task scheduling method and device, electronic equipment and storage medium |
| CN113282395A (en) * | 2021-06-09 | 2021-08-20 | 中国农业银行股份有限公司 | Redis-based job request scheduling method, device, equipment and medium |
| CN113485815A (en) * | 2021-07-27 | 2021-10-08 | 中国银行股份有限公司 | Job batch processing system, method, device, storage medium and electronic equipment |
| CN113986707A (en) * | 2021-11-03 | 2022-01-28 | 北京科杰科技有限公司 | Method for monitoring and controlling slow SQL based on big data kudu partition |
| CN113986707B (en) * | 2021-11-03 | 2022-06-14 | 北京科杰科技有限公司 | Method for monitoring and controlling slow SQL based on big data kudu partition |
| CN115297122A (en) * | 2022-09-29 | 2022-11-04 | 数字江西科技有限公司 | Government affair operation and maintenance method and system based on load automatic monitoring |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111338770A (en) | Task scheduling method, server and computer readable storage medium | |
| CN107729139B (en) | Method and device for concurrently acquiring resources | |
| CN110896404B (en) | Data processing method and device and computing node | |
| CN104657212A (en) | Task scheduling method and system | |
| US11429448B2 (en) | Background job processing framework | |
| CN111897638B (en) | Distributed task scheduling method and system | |
| US20110078297A1 (en) | Job processing system, method and program | |
| US11625233B2 (en) | Software update program for in-vehicle devices and associated server | |
| US9218210B2 (en) | Distributed processing system | |
| WO2021238796A1 (en) | Asynchronous processing method, apparatus, and system for download tasks, and storage medium | |
| WO2012137347A1 (en) | Computer system and parallel distributed processing method | |
| CN112114950A (en) | Task scheduling method and device and cluster management system | |
| CN111831191A (en) | Workflow configuration method and device, computer equipment and storage medium | |
| CN110620722B (en) | Order processing method and device | |
| CN111240824B (en) | CPU resource scheduling method and electronic equipment | |
| CN111459676A (en) | Node resource management method, device and storage medium | |
| CN101667139A (en) | Service program invocation method based on aggregate type object, device and system therefor | |
| CN112380001A (en) | Log output method, load balancing device and computer readable storage medium | |
| CN114020368A (en) | Information processing method and device based on state machine and storage medium | |
| CN110018906B (en) | Scheduling method, server and scheduling system | |
| CN103914339A (en) | Server management system and server management method | |
| CN115129438A (en) | Method and device for task distributed scheduling | |
| CN113901076A (en) | Data synchronization method, device, server and storage medium | |
| CN114546631A (en) | Task scheduling method, control method, core, electronic device and readable medium | |
| CN112084014A (en) | Data processing method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |