CN112527489A - Task scheduling method, device and equipment and computer readable storage medium - Google Patents
Task scheduling method, device and equipment and computer readable storage medium Download PDFInfo
- Publication number
- CN112527489A CN112527489A CN202011533657.6A CN202011533657A CN112527489A CN 112527489 A CN112527489 A CN 112527489A CN 202011533657 A CN202011533657 A CN 202011533657A CN 112527489 A CN112527489 A CN 112527489A
- Authority
- CN
- China
- Prior art keywords
- task
- target
- target task
- execution node
- execution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000004590 computer program Methods 0.000 claims description 10
- 238000012544 monitoring process Methods 0.000 claims description 10
- 230000000007 visual effect Effects 0.000 claims description 10
- 230000000694 effects Effects 0.000 abstract description 3
- 238000012545 processing Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 4
- 230000001419 dependent effect Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000013515 script Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N silicon dioxide Inorganic materials O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 description 2
- 238000013475 authorization Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 239000010453 quartz Substances 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5061—Partitioning or combining of resources
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a task scheduling method, which comprises the following steps: when the generation of a target task is detected, sending a first distributed lock acquisition request to a Zookeeper distributed application program coordination system; judging whether a first distributed lock corresponding to the target task is acquired; if yes, acquiring a target task type to which the target task belongs; and sending the target task to a task execution node group corresponding to the target task type. By applying the task scheduling method provided by the invention, the task scheduling efficiency is greatly improved. The invention also discloses a task scheduling device, equipment and a storage medium, and has corresponding technical effects.
Description
Technical Field
The present invention relates to the field of computer application technologies, and in particular, to a task scheduling method, apparatus, device, and computer-readable storage medium.
Background
Today's big data platforms need to execute various data integration scripts, such as data synchronization tasks, index calculation tasks, and even machine learning training tasks. For these tasks with complex relationships, the tasks need to be scheduled, executed, progress monitored, exception processed and resource isolated. Task scheduling is the most basic and most core function in a big data platform.
The existing task scheduling mode is to perform corresponding node configuration according to the configuration requirement of the current task execution node, and submit and execute the node in a jar packet form, so that the configuration efficiency and the task scheduling efficiency are low.
In summary, how to effectively solve the problem of low scheduling efficiency of the existing task scheduling method is a problem that needs to be solved urgently by those skilled in the art at present.
Disclosure of Invention
The invention aims to provide a task scheduling method, which greatly improves the task scheduling efficiency; another object of the present invention is to provide a task scheduling device, apparatus and computer readable storage medium.
In order to solve the technical problems, the invention provides the following technical scheme:
a task scheduling method comprises the following steps:
when the generation of a target task is detected, sending a first distributed lock acquisition request to a Zookeeper distributed application program coordination system;
judging whether a first distributed lock corresponding to the target task is acquired;
if yes, acquiring a target task type to which the target task belongs;
and sending the target task to a task execution node group corresponding to the target task type.
In a specific embodiment of the present invention, when the target task includes a plurality of subtasks, before obtaining a target task type to which the target task belongs, the method further includes:
acquiring directed acyclic graph scheduling metadata corresponding to the target task;
segmenting the target task according to the directed acyclic graph scheduling metadata to obtain sub-tasks;
obtaining a target task type to which the target task belongs, including:
respectively acquiring the target task type of each subtask;
sending the target task to a task execution node group corresponding to the target task type, including:
and distributing each subtask to a corresponding task execution node group according to each target task type.
In a specific embodiment of the present invention, the respectively obtaining the target task type to which each of the subtasks belongs includes:
respectively acquiring target execution environment identifications corresponding to the subtasks;
distributing each subtask to a corresponding task execution node group according to each target task type, including:
and distributing each subtask to a corresponding task execution node group according to each target execution environment identifier.
In a specific embodiment of the present invention, after distributing each of the subtasks to the corresponding task execution node group according to each of the target execution environment identifiers, the method further includes:
receiving a task execution state viewing request;
and performing visual display operation on the execution state of each subtask.
In a specific embodiment of the present invention, after distributing each of the subtasks to the corresponding task execution node group according to each of the target task types, the method further includes:
determining target task execution nodes for executing corresponding subtasks in each task execution node group;
monitoring the running state of each target task execution node;
when monitoring a task execution node disconnection event, selecting a task execution replacing node from a task execution node group where the disconnection task execution node is located;
and transferring the subtasks of the offline task execution node to the task execution successor node.
In one embodiment of the present invention, the method further comprises:
when a task control node disconnection event is monitored, sending a second distributed lock acquisition request to the Zookeeper distributed application program coordination system;
judging whether a second distributed lock corresponding to the task control node disconnection event is acquired;
and if so, performing task taking-over operation on the offline task control node corresponding to the task control node offline event.
In a specific embodiment of the present invention, after sending the target task to the task execution node group corresponding to the target task type, the method further includes:
and releasing the first distributed lock.
A task scheduling apparatus comprising:
the first request sending module is used for sending a first distributed lock acquisition request to the Zookeeper distributed application program coordination system when the generation of the target task is detected;
the first judging module is used for judging whether a first distributed lock corresponding to the target task is acquired;
the task type obtaining module is used for obtaining a target task type to which the target task belongs when the first distributed lock corresponding to the target task is determined to be obtained;
and the task sending module is used for sending the target task to the task execution node group corresponding to the target task type.
A task scheduling apparatus comprising:
a memory for storing a computer program;
a processor for implementing the steps of the task scheduling method as described above when executing the computer program.
A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the task scheduling method as set forth above.
According to the task scheduling method, when the generation of a target task is detected, a first distributed lock acquisition request is sent to a Zookeeper distributed application program coordination system; judging whether a first distributed lock corresponding to the target task is acquired; if yes, acquiring a target task type to which the target task belongs; and sending the target task to a task execution node group corresponding to the target task type.
According to the technical scheme, the task types are divided, corresponding task execution node groups are configured for the task types in advance, after the first distributed lock corresponding to the target task is obtained, the task execution node group corresponding to the target task type to which the target task belongs is directly searched, and the target task is sent to the corresponding task execution node group. Compared with the conventional mode of temporarily configuring the task execution node, the method and the device have the advantage that the task scheduling efficiency is greatly improved.
Correspondingly, the invention also provides a task scheduling device, equipment and a computer readable storage medium corresponding to the task scheduling method, which have the technical effects and are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a flowchart of an implementation of a task scheduling method according to an embodiment of the present invention;
FIG. 2 is a flowchart of another implementation of a task scheduling method according to an embodiment of the present invention;
FIG. 3 is a main flow chart of task scheduling according to an embodiment of the present invention;
FIG. 4 is a block diagram of a task scheduler according to an embodiment of the present invention;
fig. 5 is a block diagram of a task scheduling device according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the disclosure, the invention will be described in further detail with reference to the accompanying drawings and specific embodiments. It is to be understood that the described embodiments are merely exemplary of the invention, and not restrictive of the full scope of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The first embodiment is as follows:
referring to fig. 1, fig. 1 is a flowchart of an implementation of a task scheduling method in an embodiment of the present invention, where the method may include the following steps:
s101: and when the generation of the target task is detected, sending a first distributed lock acquisition request to the Zookeeper distributed application program coordination system.
The Zookeeper distributed application program coordination system is preset to manage complex tasks in the distributed platform through a simple framework and an application program interface API. And each task control node master in the distributed platform is in communication connection with the Zookeeper distributed application program coordination system, and the Zookeeper distributed application program coordination system stores the first distributed lock of the task. And each task control node performs task detection, and when the generation of a target task is detected, a first distributed lock acquisition request is sent to the Zookeeper distributed application program coordination system.
The ZooKeeper distributed application coordination system allows developers to focus on the core application logic without having to worry about the distributed nature of the application.
S102: and judging whether a first distributed lock corresponding to the target task is acquired, if so, executing the step S103, and if not, not processing.
After a first distributed lock acquisition request is sent to the Zookeeper distributed application program coordination system, whether a first distributed lock corresponding to a target task is acquired or not is judged, if yes, it is indicated that a current task control node has permission to perform task scheduling, step S103 is executed, and if not, it is indicated that the current task control node has no permission to perform task scheduling, and no processing is performed.
S103: and acquiring the target task type to which the target task belongs.
After the first distributed lock corresponding to the target task is determined to be obtained, it is indicated that the current task control node has permission to carry out task scheduling, and the target task type to which the target task belongs is obtained. Such as a target task type, for distinguishing the dependency of the target task on the environment configuration.
S104: and sending the target task to a task execution node group corresponding to the target task type.
The method comprises the steps of configuring each task execution node group according to task types in advance, determining the task execution node group corresponding to the target task type after the target task type to which the target task belongs is obtained, and sending the target task to the task execution node group (worker) corresponding to the target task type.
According to the technical scheme, the task types are divided, corresponding task execution node groups are configured for the task types in advance, after the first distributed lock corresponding to the target task is obtained, the task execution node group corresponding to the target task type to which the target task belongs is directly searched, and the target task is sent to the corresponding task execution node group. Compared with the conventional mode of temporarily configuring the task execution node, the method and the device have the advantage that the task scheduling efficiency is greatly improved.
It should be noted that, based on the first embodiment, the embodiment of the present invention further provides a corresponding improvement scheme. In the following embodiments, steps that are the same as or correspond to those in the first embodiment may be referred to each other, and corresponding advantageous effects may also be referred to each other, which are not described in detail in the following modified embodiments.
Example two:
referring to fig. 2, fig. 2 is a flowchart of another implementation of a task scheduling method in an embodiment of the present invention, where the method may include the following steps:
s201: and when the generation of the target task is detected, sending a first distributed lock acquisition request to the Zookeeper distributed application program coordination system.
S202: and judging whether a first distributed lock corresponding to the target task is acquired, if so, executing the step S203, and if not, not processing.
S203: and when the target task comprises a plurality of subtasks, acquiring directed acyclic graph scheduling metadata corresponding to the target task.
When the target task comprises a plurality of subtasks, directed acyclic graph DAG (direct acyclic graph) scheduling metadata corresponding to each target task are stored in advance, and the directed acyclic graph scheduling metadata records the execution priority and the in-degree information of each subtask contained in the target task.
S204: and segmenting the target task according to the metadata of the directed acyclic graph scheduling to obtain each subtask.
After the directed acyclic graph scheduling metadata corresponding to the target task is obtained, the target task is segmented according to the directed acyclic graph scheduling metadata to obtain sub-tasks. After the task is split, the sub-tasks with the cut in-degree of 0 can be distributed to a memory priority queue.
S205: and respectively acquiring target execution environment identifications corresponding to the subtasks.
And each subtask carries a corresponding target execution environment identifier, and after the target tasks are segmented according to the directed acyclic graph scheduling metadata to obtain each subtask, the target execution environment identifiers corresponding to the subtasks are obtained respectively.
S206: and distributing each subtask to a corresponding task execution node group according to each target execution environment identifier.
And after the target execution environment identifications corresponding to the subtasks are respectively obtained, distributing the subtasks to the corresponding task execution node groups according to the target execution environment identifications.
Each task execution node group integrates common components in big data ecology, and comprises a data security layer integrated with kerberos computer network authorization protocol authentication, and different tenants adopt respective keytab file authentication isolation authorities; the data exchange layer supports an open source tool of sqoop, and realizes data exchange between hdfs and the relational database by configuring a structured query language (sql); the warehouse layer supports a hive data warehouse tool, and a MapReduce programming model task can be executed by configuring sql; the computing layer supports a spark computing engine, computing tasks are submitted to a yarn cluster through a spark-submit through configuring a pyspark script or a java program, and the task execution state is inquired through a yarn rest api; and in addition, the method comprises common python and shell script components, and other tasks can be flexibly executed by configuring the shell or python script.
The big data components are executed on a specific task execution node worker, and the task execution nodes can be grouped according to actual use purposes, such as a data exchange task execution node group and a data calculation submission task execution node group. The method includes the steps that required components are installed for different task execution nodes, different container (docker) images can be manufactured, configuration files and ports are exposed and used for mounting corresponding configuration files when the task execution nodes are started. Taking a spark calculation engine as an example, uploading a jar packet and a pyspark virtual environment which are dependent on spark and an execution code which is dependent on a current task to a remote hdfs directory through a resource management module of the system, then selecting a spark component in a workflow, selecting a code file for executing the task, and configuring a cpu, a memory resource and a task execution node group where a spark-submit program is executed, wherein the cpu, the memory resource and the spark-submit program are required by calculation execution.
S207: and releasing the first distributed lock.
And after distributing each subtask to the corresponding task execution node group, releasing the first distributed lock. For scheduling of other tasks to follow.
S208: and receiving a task execution state viewing request.
In the process that each task execution node group executes each subtask, the user side can send a task execution state viewing request to the visual display center, and the visual display center receives the task execution state viewing request.
S209: and performing visual display operation on the execution state of each subtask.
And after receiving the task execution state viewing request, the visual display center performs visual display operation on the execution state of each subtask, so that the task execution state is visually monitored. In addition, the visual editing of the directed acyclic graph DAG can adopt a jsPlumb plug-in to realize what you see is what you get operations such as the drawing, node selection, node deletion, node connection, node position movement and the like of the directed acyclic graph.
S210: and determining target task execution nodes for executing corresponding subtasks in each task execution node group.
And after distributing each subtask to the corresponding task execution node group according to each target execution environment identifier, determining a target task execution node for executing the corresponding subtask in each task execution node group.
S211: and monitoring the running state of each target task execution node.
And after determining target task execution nodes for executing corresponding subtasks in each task execution node group, monitoring the running state of each target task execution node.
S212: and when the offline event of the task execution node is monitored, selecting a task execution replacing node from the task execution node group where the offline task execution node is located.
And when the offline event of the task execution node is monitored, selecting a task execution replacing node from the task execution node group where the offline task execution node is located. The task execution successor node may preset a designated task execution node, or may randomly select from the idle task execution nodes.
S213: and transferring the subtasks of the offline task execution node to the task execution successor node.
And after selecting a task execution successor node from the task execution node group where the offline task execution node is located, transferring the subtasks of the offline task execution node to the task execution successor node. Therefore, fault-tolerant setting of the task execution nodes is realized, and normal operation of each subtask is guaranteed.
S214: and when monitoring the disconnection event of the task control node, sending a second distributed lock acquisition request to the Zookeeper distributed application program coordination system.
Each task control node monitors the task control node disconnection event in the distributed platform, and when the task control node disconnection event is monitored, a second distributed lock acquisition request is sent to the Zookeeper distributed application program coordination system.
S215: and judging whether a second distributed lock corresponding to the task control node disconnection event is acquired, if so, executing the step S216, and if not, not processing.
After sending a second distributed lock acquisition request to the Zookeeper distributed application program coordination system, judging whether a second distributed lock corresponding to the task control node disconnection event is acquired, if so, indicating that the current task control node has permission to perform task succession, executing step S216, and if not, indicating that the current task control node has no permission to perform task succession, and not performing processing.
S216: and carrying out task taking-over operation on the offline task control node corresponding to the offline event of the task control node.
And after the second distributed lock corresponding to the task control node disconnection event is determined to be acquired, performing task taking-over operation on the disconnection task control node corresponding to the task control node disconnection event. Therefore, fault-tolerant setting of the task control nodes is realized, and normal operation of task scheduling is guaranteed.
The present embodiment is different from the first embodiment corresponding to the technical solution claimed in independent claim 1, and the technical solutions claimed in the dependent claims 2 to 7 are added, and of course, according to different practical situations and requirements, the technical solutions claimed in the dependent claims can be flexibly combined on the basis of not affecting the completeness of the solutions, so as to better meet the requirements of different use scenarios.
In an embodiment application of the present invention, referring to fig. 3, fig. 3 is a main flow chart of task scheduling according to an embodiment of the present invention. Task triggering can include two ways, one is to submit a task at the front-end server frontserver, or the quartz service automatically triggers a task on the task control node. After a task is triggered, a command record is inserted into a database, metadata of a DAG (directed acyclic graph) flow scheduled currently is stored, each task control node competes for a distributed lock to a zookeeper distributed application program coordination system to acquire the execution right of the command, and after the task control node competes for the distributed lock, the earliest command is acquired from the database and processing is started in sequence. Adding a flow example, performing directed acyclic graph segmentation, distributing the task with the entry degree of 0 after segmentation to a memory priority queue, and then releasing the distributed lock. The task control node can obtain the task from the priority queue by a thread, select the task execution node for executing the task by adopting the configured strategy and distribute the task. After receiving the task distributed by the task control node, the task execution node returns a confirmation character ack packet to confirm the receipt of the task. And after receiving the ack packet, the task control node updates the state of the task and the current task execution node, the task execution node then starts to execute the acquired task, after the execution is finished, the execution result is returned to the task control node by a response result response packet, and after receiving the response result, the task control node updates the task state in the database and finally finishes the execution process of the task.
Corresponding to the above method embodiments, the present invention further provides a task scheduling device, and the task scheduling device described below and the task scheduling method described above may be referred to correspondingly.
Referring to fig. 4, fig. 4 is a block diagram of a task scheduling apparatus according to an embodiment of the present invention, where the task scheduling apparatus may include:
a first request sending module 41, configured to send a first distributed lock acquisition request to the Zookeeper distributed application coordination system when it is detected that the target task is generated;
the first judging module 42 is configured to judge whether a first distributed lock corresponding to the target task is acquired;
the task type obtaining module 43 is configured to obtain a target task type to which the target task belongs when it is determined that the first distributed lock corresponding to the target task is obtained;
and the task sending module 44 is configured to send the target task to the task execution node group corresponding to the target task type.
According to the technical scheme, the task types are divided, corresponding task execution node groups are configured for the task types in advance, after the first distributed lock corresponding to the target task is obtained, the task execution node group corresponding to the target task type to which the target task belongs is directly searched, and the target task is sent to the corresponding task execution node group. Compared with the conventional mode of temporarily configuring the task execution node, the method and the device have the advantage that the task scheduling efficiency is greatly improved.
In one embodiment of the present invention, the apparatus may further include:
the metadata acquisition module is used for acquiring directed acyclic graph scheduling metadata corresponding to the target task before acquiring the target task type to which the target task belongs when the target task comprises a plurality of subtasks;
the task segmentation module is used for segmenting the target task according to the directed acyclic graph scheduling metadata to obtain each subtask;
the task type obtaining module 43 is specifically a module that obtains the target task types to which the respective subtasks belong;
the task sending module 44 is specifically a module that distributes each subtask to a corresponding task execution node group according to each target task type.
In a specific embodiment of the present invention, the task type obtaining module 43 is specifically a module for respectively obtaining target execution environment identifiers corresponding to the subtasks;
the task sending module 44 is specifically a module that distributes each subtask to a corresponding task execution node group according to each target execution environment identifier.
In one embodiment of the present invention, the apparatus may further include:
the checking request receiving module is used for receiving the task execution state checking request after distributing each subtask to the corresponding task execution node group according to each target execution environment identifier;
and the visual display module is used for carrying out visual display operation on the execution state of each subtask.
In one embodiment of the present invention, the apparatus may further include:
the execution node determining module is used for determining target task execution nodes for executing corresponding subtasks in each task execution node group after distributing each subtask to the corresponding task execution node group according to each target task type;
the running state monitoring module is used for monitoring the running state of each target task execution node;
the relay node selection module is used for selecting a task execution relay node from a task execution node group where the offline task execution node is located when monitoring an offline event of the task execution node;
and the task handover module is used for handing over the subtasks of the offline task execution node to the task execution successor node.
In one embodiment of the present invention, the apparatus may further include:
the second request sending module is used for sending a second distributed lock acquisition request to the Zookeeper distributed application program coordination system when a task control node disconnection event is monitored;
the second judgment module is used for judging whether a second distributed lock corresponding to the task control node disconnection event is acquired;
and the task taking-over module is used for carrying out task taking-over operation on the offline task control node corresponding to the task control node offline event when the second distributed lock corresponding to the task control node offline event is determined to be acquired.
In one embodiment of the present invention, the apparatus may further include:
and the distributed lock releasing module is used for releasing the first distributed lock after the target task is sent to the task execution node group corresponding to the target task type.
Corresponding to the above method embodiment, referring to fig. 4, fig. 4 is a schematic diagram of a task scheduling apparatus provided in the present invention, where the apparatus may include:
a memory 41 for storing a computer program;
the processor 42, when executing the computer program stored in the memory 41, may implement the following steps:
when the generation of a target task is detected, sending a first distributed lock acquisition request to a Zookeeper distributed application program coordination system; judging whether a first distributed lock corresponding to the target task is acquired; if yes, acquiring a target task type to which the target task belongs; and sending the target task to a task execution node group corresponding to the target task type.
For the introduction of the device provided by the present invention, please refer to the above method embodiment, which is not described herein again.
Corresponding to the above method embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the steps of:
when the generation of a target task is detected, sending a first distributed lock acquisition request to a Zookeeper distributed application program coordination system; judging whether a first distributed lock corresponding to the target task is acquired; if yes, acquiring a target task type to which the target task belongs; and sending the target task to a task execution node group corresponding to the target task type.
The computer-readable storage medium may include: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
For the introduction of the computer-readable storage medium provided by the present invention, please refer to the above method embodiments, which are not described herein again.
The embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts among the embodiments are referred to each other. The device, the apparatus and the computer-readable storage medium disclosed in the embodiments correspond to the method disclosed in the embodiments, so that the description is simple, and the relevant points can be referred to the description of the method.
The principle and the implementation of the present invention are explained in the present application by using specific examples, and the above description of the embodiments is only used to help understanding the technical solution and the core idea of the present invention. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (10)
1. A method for task scheduling, comprising:
when the generation of a target task is detected, sending a first distributed lock acquisition request to a Zookeeper distributed application program coordination system;
judging whether a first distributed lock corresponding to the target task is acquired;
if yes, acquiring a target task type to which the target task belongs;
and sending the target task to a task execution node group corresponding to the target task type.
2. The task scheduling method according to claim 1, wherein when the target task includes a plurality of subtasks, before obtaining a target task type to which the target task belongs, the method further includes:
acquiring directed acyclic graph scheduling metadata corresponding to the target task;
segmenting the target task according to the directed acyclic graph scheduling metadata to obtain sub-tasks;
obtaining a target task type to which the target task belongs, including:
respectively acquiring the target task type of each subtask;
sending the target task to a task execution node group corresponding to the target task type, including:
and distributing each subtask to a corresponding task execution node group according to each target task type.
3. The task scheduling method according to claim 2, wherein the step of respectively obtaining the target task type to which each of the subtasks belongs comprises:
respectively acquiring target execution environment identifications corresponding to the subtasks;
distributing each subtask to a corresponding task execution node group according to each target task type, including:
and distributing each subtask to a corresponding task execution node group according to each target execution environment identifier.
4. The task scheduling method according to claim 3, wherein after distributing each of the subtasks to the corresponding task execution node group according to each of the target execution environment identifiers, further comprising:
receiving a task execution state viewing request;
and performing visual display operation on the execution state of each subtask.
5. The task scheduling method according to any one of claims 2 to 4, wherein after distributing each of the subtasks to the corresponding task execution node group according to each of the target task types, further comprising:
determining target task execution nodes for executing corresponding subtasks in each task execution node group;
monitoring the running state of each target task execution node;
when monitoring a task execution node disconnection event, selecting a task execution replacing node from a task execution node group where the disconnection task execution node is located;
and transferring the subtasks of the offline task execution node to the task execution successor node.
6. The task scheduling method according to claim 1, further comprising:
when a task control node disconnection event is monitored, sending a second distributed lock acquisition request to the Zookeeper distributed application program coordination system;
judging whether a second distributed lock corresponding to the task control node disconnection event is acquired;
and if so, performing task taking-over operation on the offline task control node corresponding to the task control node offline event.
7. The task scheduling method according to claim 1, wherein after sending the target task to the task execution node group corresponding to the target task type, the method further comprises:
and releasing the first distributed lock.
8. A task scheduling apparatus, comprising:
the first request sending module is used for sending a first distributed lock acquisition request to the Zookeeper distributed application program coordination system when the generation of the target task is detected;
the first judging module is used for judging whether a first distributed lock corresponding to the target task is acquired;
the task type obtaining module is used for obtaining a target task type to which the target task belongs when the first distributed lock corresponding to the target task is determined to be obtained;
and the task sending module is used for sending the target task to the task execution node group corresponding to the target task type.
9. A task scheduling apparatus, comprising:
a memory for storing a computer program;
a processor for implementing the steps of the task scheduling method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the task scheduling method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011533657.6A CN112527489B (en) | 2020-12-22 | 2020-12-22 | Task scheduling method, device, equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011533657.6A CN112527489B (en) | 2020-12-22 | 2020-12-22 | Task scheduling method, device, equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112527489A true CN112527489A (en) | 2021-03-19 |
| CN112527489B CN112527489B (en) | 2024-07-19 |
Family
ID=74975790
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011533657.6A Active CN112527489B (en) | 2020-12-22 | 2020-12-22 | Task scheduling method, device, equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112527489B (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113065779A (en) * | 2021-04-07 | 2021-07-02 | 网易（杭州）网络有限公司 | Data processing method and device and electronic equipment |
| CN113127096A (en) * | 2021-04-27 | 2021-07-16 | 上海商汤科技开发有限公司 | Task processing method and device, electronic equipment and storage medium |
| CN113553155A (en) * | 2021-07-23 | 2021-10-26 | 中信银行股份有限公司 | Task scheduling method and device and electronic equipment |
| CN113641493A (en) * | 2021-08-11 | 2021-11-12 | 杭州安恒信息技术股份有限公司 | Task pushing method, device and equipment and computer readable storage medium |
| CN113687932A (en) * | 2021-08-30 | 2021-11-23 | 上海商汤科技开发有限公司 | Task scheduling method, device and system, electronic equipment and storage medium |
| CN113792292A (en) * | 2021-09-14 | 2021-12-14 | 山石网科通信技术股份有限公司 | Response method and device of security script, storage medium and processor |
| CN113810479A (en) * | 2021-08-31 | 2021-12-17 | 山石网科通信技术股份有限公司 | Service coordination system and service coordination method |
| CN116069480A (en) * | 2023-04-06 | 2023-05-05 | 杭州登临瀚海科技有限公司 | Processor and computing device |
| CN116991591A (en) * | 2023-09-25 | 2023-11-03 | 腾讯科技（深圳）有限公司 | Data scheduling method, device and storage medium |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105159767A (en) * | 2015-09-06 | 2015-12-16 | 北京京东尚科信息技术有限公司 | Method and device for realizing distributed scheduling on the basis of zookeeper |
| CN107766129A (en) * | 2016-08-17 | 2018-03-06 | 北京金山云网络技术有限公司 | A kind of task processing method, apparatus and system |
| CN108304255A (en) * | 2017-12-29 | 2018-07-20 | 北京城市网邻信息技术有限公司 | Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing |
| CN109814998A (en) * | 2019-01-22 | 2019-05-28 | 中国联合网络通信集团有限公司 | A kind of method and device of multi-process task schedule |
| CN110677453A (en) * | 2019-08-15 | 2020-01-10 | 平安普惠企业管理有限公司 | ZooKeeper-based distributed lock service implementation method, device, equipment and storage medium |
| CN110673933A (en) * | 2019-08-15 | 2020-01-10 | 平安普惠企业管理有限公司 | ZooKeeper-based distributed asynchronous queue implementation method, device, equipment and medium |
| WO2020024405A1 (en) * | 2018-08-03 | 2020-02-06 | 平安科技（深圳）有限公司 | Test method, device, server and storage medium based on distributed coordination |
| CN110955508A (en) * | 2019-11-28 | 2020-04-03 | 广州鼎甲计算机科技有限公司 | Asynchronous task scheduling method and device based on distributed architecture and computer equipment |
| CN111147571A (en) * | 2019-12-24 | 2020-05-12 | 京东数字科技控股有限公司 | Distributed task scheduling method, device, system, storage medium and electronic equipment |
| CN111324435A (en) * | 2020-02-06 | 2020-06-23 | 北京奇艺世纪科技有限公司 | Distributed task scheduling and registering method, device and distributed task scheduling system |
| CN111597056A (en) * | 2020-05-18 | 2020-08-28 | 北京思特奇信息技术股份有限公司 | Distributed scheduling method, system, storage medium and device |
| CN111970148A (en) * | 2020-08-14 | 2020-11-20 | 北京金山云网络技术有限公司 | Distributed task scheduling method and system |
-
2020
- 2020-12-22 CN CN202011533657.6A patent/CN112527489B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105159767A (en) * | 2015-09-06 | 2015-12-16 | 北京京东尚科信息技术有限公司 | Method and device for realizing distributed scheduling on the basis of zookeeper |
| CN107766129A (en) * | 2016-08-17 | 2018-03-06 | 北京金山云网络技术有限公司 | A kind of task processing method, apparatus and system |
| CN108304255A (en) * | 2017-12-29 | 2018-07-20 | 北京城市网邻信息技术有限公司 | Distributed task dispatching method and device, electronic equipment and readable storage medium storing program for executing |
| WO2020024405A1 (en) * | 2018-08-03 | 2020-02-06 | 平安科技（深圳）有限公司 | Test method, device, server and storage medium based on distributed coordination |
| CN109814998A (en) * | 2019-01-22 | 2019-05-28 | 中国联合网络通信集团有限公司 | A kind of method and device of multi-process task schedule |
| CN110677453A (en) * | 2019-08-15 | 2020-01-10 | 平安普惠企业管理有限公司 | ZooKeeper-based distributed lock service implementation method, device, equipment and storage medium |
| CN110673933A (en) * | 2019-08-15 | 2020-01-10 | 平安普惠企业管理有限公司 | ZooKeeper-based distributed asynchronous queue implementation method, device, equipment and medium |
| CN110955508A (en) * | 2019-11-28 | 2020-04-03 | 广州鼎甲计算机科技有限公司 | Asynchronous task scheduling method and device based on distributed architecture and computer equipment |
| CN111147571A (en) * | 2019-12-24 | 2020-05-12 | 京东数字科技控股有限公司 | Distributed task scheduling method, device, system, storage medium and electronic equipment |
| CN111324435A (en) * | 2020-02-06 | 2020-06-23 | 北京奇艺世纪科技有限公司 | Distributed task scheduling and registering method, device and distributed task scheduling system |
| CN111597056A (en) * | 2020-05-18 | 2020-08-28 | 北京思特奇信息技术股份有限公司 | Distributed scheduling method, system, storage medium and device |
| CN111970148A (en) * | 2020-08-14 | 2020-11-20 | 北京金山云网络技术有限公司 | Distributed task scheduling method and system |
Non-Patent Citations (6)
| Title |
|---|
| 刘芬;王芳;田昊;: "基于Zookeeper的分布式锁服务及性能优化", 计算机研究与发展, no. 1, 15 December 2014 (2014-12-15) * |
| 孙良君;袁庆祝;陆佃龙;: "分布式实时抽取计算框架设计与应用", 信息技术, no. 08, 25 August 2016 (2016-08-25) * |
| 承林;王海宁;高春成;: "分布式任务调度在电力市场交易系统中的应用设计", 计算机应用与软件, no. 11, 12 November 2018 (2018-11-12) * |
| 李荣辉;杨志龙;: "通过tomcat manager和Zookeeper实现服务的高可用", 通信电源技术, no. 09, 25 September 2018 (2018-09-25) * |
| 王文峰;袁庆祝;陆佃龙;: "基于Zookeeper综合任务调度平台的设计与应用", 信息技术, no. 06, 25 June 2016 (2016-06-25) * |
| 陈天伟;: "基于ZooKeeper的一种分布式通信服务中间件的研究", 移动通信, no. 24, 30 December 2017 (2017-12-30) * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113065779B (en) * | 2021-04-07 | 2023-08-11 | 网易（杭州）网络有限公司 | Data processing method and device and electronic equipment |
| CN113065779A (en) * | 2021-04-07 | 2021-07-02 | 网易（杭州）网络有限公司 | Data processing method and device and electronic equipment |
| CN113127096A (en) * | 2021-04-27 | 2021-07-16 | 上海商汤科技开发有限公司 | Task processing method and device, electronic equipment and storage medium |
| CN113553155A (en) * | 2021-07-23 | 2021-10-26 | 中信银行股份有限公司 | Task scheduling method and device and electronic equipment |
| CN113641493A (en) * | 2021-08-11 | 2021-11-12 | 杭州安恒信息技术股份有限公司 | Task pushing method, device and equipment and computer readable storage medium |
| CN113687932A (en) * | 2021-08-30 | 2021-11-23 | 上海商汤科技开发有限公司 | Task scheduling method, device and system, electronic equipment and storage medium |
| CN113810479A (en) * | 2021-08-31 | 2021-12-17 | 山石网科通信技术股份有限公司 | Service coordination system and service coordination method |
| CN113810479B (en) * | 2021-08-31 | 2023-11-07 | 山石网科通信技术股份有限公司 | Service coordination system and service coordination method |
| CN113792292A (en) * | 2021-09-14 | 2021-12-14 | 山石网科通信技术股份有限公司 | Response method and device of security script, storage medium and processor |
| CN116069480A (en) * | 2023-04-06 | 2023-05-05 | 杭州登临瀚海科技有限公司 | Processor and computing device |
| CN116069480B (en) * | 2023-04-06 | 2023-06-13 | 杭州登临瀚海科技有限公司 | Processor and computing device |
| CN116991591A (en) * | 2023-09-25 | 2023-11-03 | 腾讯科技（深圳）有限公司 | Data scheduling method, device and storage medium |
| CN116991591B (en) * | 2023-09-25 | 2024-01-09 | 腾讯科技（深圳）有限公司 | Data scheduling method, device and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112527489B (en) | 2024-07-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112527489B (en) | Task scheduling method, device, equipment and computer readable storage medium | |
| US6411982B2 (en) | Thread based governor for time scheduled process execution | |
| CN112513813A (en) | Performing auxiliary functions in an on-demand network code execution system | |
| CN110941481A (en) | Resource scheduling method, device and system | |
| CN105700939A (en) | Method and system for multi-thread synchronization in distributed system | |
| CN112052068A (en) | Method and device for binding CPU (central processing unit) of Kubernetes container platform | |
| CN110971700B (en) | Method and device for realizing distributed lock | |
| CN109697112B (en) | Distributed intensive one-stop operating system and implementation method | |
| US7219345B2 (en) | System and method for terminating processes in a distributed computing system | |
| GB2518894A (en) | A method and a system for operating programs on a computer cluster | |
| CN111709723B (en) | RPA business process intelligent processing method, device, computer equipment and storage medium | |
| CN114356521A (en) | Task scheduling method and device, electronic equipment and storage medium | |
| CN109117244B (en) | Method for implementing virtual machine resource application queuing mechanism | |
| CN109902028A (en) | Automated testing method, device, equipment and the storage medium of ACL characteristic | |
| JP2016091555A (en) | Data staging management system | |
| CN112597511A (en) | Remote government affair service cooperation method and device | |
| CN112882765A (en) | Digital twin model scheduling method and device | |
| CN111241540A (en) | Service processing method and device | |
| CN115827331A (en) | Processing method | |
| CN113448775B (en) | Multi-source heterogeneous data backup method and device | |
| CN115858499A (en) | Database partition processing method and device, computer equipment and storage medium | |
| CN114780217A (en) | Task scheduling method and device, computer equipment and medium | |
| CN114218329A (en) | Data synchronization method, device, storage medium and computer terminal | |
| CN114201284A (en) | Timed task management method and system | |
| CN113722091A (en) | Simulation task deployment method, system and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |