CN110764910A - Batch job scheduling processing method and device - Google Patents
Batch job scheduling processing method and device Download PDFInfo
- Publication number
- CN110764910A CN110764910A CN201911010066.8A CN201911010066A CN110764910A CN 110764910 A CN110764910 A CN 110764910A CN 201911010066 A CN201911010066 A CN 201911010066A CN 110764910 A CN110764910 A CN 110764910A
- Authority
- CN
- China
- Prior art keywords
- scheduling
- job
- node
- instruction
- persistent
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
- G06F9/5038—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals considering the execution order of a plurality of tasks, e.g. taking priority or time dependency constraints into consideration
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a batch job scheduling processing method and a device, wherein the method is applied to a plurality of scheduling nodes and comprises the following steps: a scheduling node receives a scheduling instruction of a job; the scheduling node creates a persistent node corresponding to the job in the zookeeper; and if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node triggers the scheduling of the job according to the scheduling instruction. The invention avoids the problems that all job scheduling can not be carried out and the whole batch job execution is interrupted due to the crash of one core server with a scheduling function, avoids the problem that multiple scheduling nodes repeatedly schedule the same job, and ensures the normal operation of the system.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a batch job scheduling processing method and device.
Background
At present, an Elastic-Job-Lite framework is adopted for distributed batch Job scheduling of a core bank X86 system, and sequential execution of batch jobs is realized by deploying the framework on a core server. However, once the core server deploying the scheduling function crashes, scheduling of all jobs cannot be performed, execution of the whole batch job is interrupted, and normal operation of the system is seriously affected.
Disclosure of Invention
The embodiment of the invention provides a batch job scheduling processing method, which is applied to a plurality of scheduling nodes and is used for realizing scheduling of batch jobs, avoiding the problems that all job scheduling cannot be carried out and the whole batch jobs are interrupted due to the crash of a core server with a scheduling function, and ensuring the normal operation of a system, and comprises the following steps:
a scheduling node receives a scheduling instruction of a job;
the scheduling node creates a persistent node corresponding to the job in the zookeeper;
and if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node triggers the scheduling of the job according to the scheduling instruction.
The embodiment of the invention provides a batch job scheduling processing device, which is applied to a plurality of scheduling nodes and is used for realizing scheduling of batch jobs, avoiding the problems that all job scheduling cannot be carried out and the whole batch jobs are interrupted due to the crash of a core server with a scheduling function, and ensuring the normal operation of a system, wherein each scheduling node in the device comprises:
the instruction receiving module is used for receiving a scheduling instruction of the job;
the node creating module is used for creating a persistent node corresponding to the job in the zookeeper;
and the scheduling module is used for triggering the scheduling of the job according to the scheduling instruction if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly.
The embodiment of the invention also provides computer equipment which comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the batch job scheduling processing method is realized when the processor executes the computer program.
The embodiment of the invention also provides computer equipment which comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the batch job scheduling processing method is realized when the processor executes the computer program.
Compared with the scheme that the frame is deployed on the core server to realize batch job scheduling in the prior art, the embodiment of the invention is applied to a plurality of scheduling nodes, and the scheduling nodes receive scheduling instructions of jobs; the scheduling node creates a persistent node corresponding to the job in the zookeeper; and if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node triggers the scheduling of the job according to the scheduling instruction. In the embodiment of the invention, a plurality of scheduling nodes receive scheduling instructions of jobs, each scheduling node creates a persistent node corresponding to the jobs in a zookeeper, if a crash situation occurs in one scheduling node, other scheduling nodes can continue to process the scheduling instructions, so that the problems that all job scheduling cannot be performed and the whole batch jobs are executed and interrupted due to the crash of a core server deploying a scheduling function are avoided, meanwhile, the characteristic that only one persistent node can be successfully created for the same job by the zookeeper is utilized, and only the scheduling node which creates the persistent node corresponding to the job at the earliest in order can trigger the scheduling of the job according to the scheduling instructions, so that the problem that the same job is repeatedly scheduled by a plurality of scheduling nodes is avoided, and the normal operation of a system is ensured.
The embodiment of the invention provides a batch job scheduling processing method, which is applied to a plurality of scheduling nodes and is used for realizing scheduling of batch jobs, avoiding the problems that all job scheduling cannot be carried out and the whole batch jobs are interrupted due to the crash of a core server with a scheduling function, and ensuring the normal operation of a system, and comprises the following steps:
the scheduling node generates a scheduling instruction of the job as follows:
obtaining an execution state tag of a previous job corresponding to the job, wherein the execution state tag is as follows: an executed state tag or an unexecuted state tag;
if the execution state labels of all the previous jobs corresponding to the jobs are the executed state labels, generating a scheduling instruction of the jobs;
the scheduling node creates a persistent node corresponding to the job in the zookeeper according to the scheduling instruction of the job;
and if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node triggers the scheduling of the job according to the scheduling instruction.
The embodiment of the invention provides a batch job scheduling processing device, which is applied to a plurality of scheduling nodes and is used for realizing scheduling of batch jobs, avoiding the problems that all job scheduling cannot be carried out and the whole batch jobs are interrupted due to the crash of a core server with a scheduling function, and ensuring the normal operation of a system, wherein each scheduling node in the device comprises:
the instruction generation module is used for generating a scheduling instruction of the job as follows:
obtaining an execution state tag of a previous job corresponding to the job, wherein the execution state tag is as follows: an executed state tag or an unexecuted state tag;
if the execution state labels of all the previous jobs corresponding to the jobs are the executed state labels, generating a scheduling instruction of the jobs;
the node creating module is used for creating a persistent node corresponding to the job in the zookeeper according to the scheduling instruction of the job;
and the scheduling module is used for triggering the scheduling of the job according to the scheduling instruction if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly.
The embodiment of the invention also provides computer equipment which comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the batch job scheduling processing method is realized when the processor executes the computer program.
The embodiment of the invention also provides computer equipment which comprises a memory, a processor and a computer program which is stored on the memory and can run on the processor, wherein the batch job scheduling processing method is realized when the processor executes the computer program.
Compared with the scheme that the frame is deployed on the core server to realize batch job scheduling in the prior art, the embodiment of the invention generates the scheduling instruction of the job by the scheduling node according to the following mode: obtaining an execution state tag of a previous job corresponding to the job, wherein the execution state tag is as follows: an executed state tag or an unexecuted state tag; if the execution state labels of all the previous jobs corresponding to the jobs are the executed state labels, generating a scheduling instruction of the jobs; the scheduling node creates a persistent node corresponding to the job in the zookeeper according to the scheduling instruction of the job; and if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node triggers the scheduling of the job according to the scheduling instruction. The method and the device are applied to a plurality of scheduling nodes, firstly, whether the job scheduling is triggered or not is determined according to the execution state tags of the previous jobs corresponding to the jobs, and if the execution state tags of all the previous jobs corresponding to the jobs are the executed state tags, the scheduling instruction of the jobs is generated, so that the problem that the previous jobs start to run without completing the subsequent jobs is solved, the accurate scheduling of the batch jobs is realized corresponding to the actual job running condition, and the normal running of the system is guaranteed. The persistent nodes corresponding to the jobs are created in the zookeeper by each scheduling node according to the scheduling instructions, if a crash situation occurs in one scheduling node, other scheduling nodes can continue to process the scheduling instructions, the problems that all job scheduling cannot be performed and the whole batch jobs are executed and interrupted due to the crash of one core server deploying the scheduling function are solved, meanwhile, the characteristic that the zookeeper can only successfully create one persistent node for the same job is utilized, and only the scheduling node which successfully creates the persistent node corresponding to the job at the first time can trigger the scheduling of the job according to the scheduling instructions, so that the problem that multiple scheduling nodes repeatedly schedule the same job is solved, and the normal operation of the system is guaranteed.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
FIG. 1 is a schematic diagram illustrating a batch job scheduling method according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a batch job scheduling apparatus according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a batch job scheduling apparatus according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of another batch job scheduling method according to an embodiment of the present invention;
FIG. 5 is a block diagram of another batch job scheduling apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. The exemplary embodiments and descriptions of the present invention are provided to explain the present invention, but not to limit the present invention.
As mentioned above, the distributed batch Job scheduling of the current core bank X86 system adopts an Elastic-Job-Lite framework, and the framework is deployed on a core server to realize the sequential execution of batch jobs. However, once the core server deploying the scheduling function crashes, scheduling of all jobs cannot be performed, execution of the whole batch job is interrupted, normal operation of the system is seriously affected, but if a plurality of scheduling nodes are simply deployed, the problem that the same job is repeatedly scheduled by the plurality of scheduling nodes exists.
In order to implement scheduling of batch jobs, avoid the problems that all job scheduling cannot be performed and the whole batch job is interrupted in execution due to the crash of a core server deploying a scheduling function, and ensure the normal operation of a system, an embodiment of the present invention provides a batch job scheduling processing method, which is applied to a plurality of scheduling nodes, and as shown in fig. 1, the method may include:
102, the scheduling node creates a persistent node corresponding to the job in the zookeeper;
As shown in fig. 1, the embodiment of the present invention is applied to a plurality of scheduling nodes, and receives a scheduling instruction of a job through the scheduling nodes; the scheduling node creates a persistent node corresponding to the job in the zookeeper; and if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node triggers the scheduling of the job according to the scheduling instruction. In the embodiment of the invention, a plurality of scheduling nodes receive scheduling instructions of jobs, each scheduling node creates a persistent node corresponding to the jobs in a zookeeper, if a crash situation occurs in one scheduling node, other scheduling nodes can continue to process the scheduling instructions, so that the problems that all job scheduling cannot be performed and the whole batch jobs are executed and interrupted due to the crash of a core server deploying a scheduling function are avoided, meanwhile, the characteristic that only one persistent node can be successfully created for the same job by the zookeeper is utilized, and only the scheduling node which creates the persistent node corresponding to the job at the earliest in order can trigger the scheduling of the job according to the scheduling instructions, so that the problem that the same job is repeatedly scheduled by a plurality of scheduling nodes is avoided, and the normal operation of a system is ensured.
In specific implementation, the scheduling node receives a scheduling instruction of a job.
In an embodiment, before receiving a scheduling instruction of a job, the scheduling node initializes a data table and a zookeeper where the job is located. The method can carry out initialization operation at regular time, carry out initialization at fixed time every day, modify the execution state label of the job in the data table into an unexecuted state label, and clean all scheduling nodes under the root directory. For example, a fixed time point 12:00:00 of a natural day is set, an initialization operation is performed, an execution status tag of a job in the data table after initialization is an unexecuted status tag, and a batch date is a date on which the job is to be executed.
During specific implementation, the scheduling node creates a persistent node corresponding to the job in the zookeeper.
In specific implementation, if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes first, the scheduling node triggers scheduling of the job according to the scheduling instruction.
The inventor finds that Zookeeper is software for providing a consistency service for distributed applications, and the provided functions comprise: configuration maintenance, domain name service, distributed synchronization, group service, etc. The Zookeeper comprises four types of nodes, namely a persistent node, a persistent sequence numbering catalog node, a temporary catalog node and a temporary sequence numbering catalog node, wherein the persistent node is a node which can exist all the time after being created and can not be deleted due to the fact that a client session is effective. Therefore, the embodiment of the invention applies the characteristic of the persistent node in the Zookeeper to the scheduling of the batch jobs, each scheduling node is locked by creating the persistent node through the Zookeeper, only the scheduling node which is successfully created firstly can take the resource lock to perform job scheduling, and other nodes do not schedule the job repeatedly.
In an embodiment, if the scheduling node successfully creates the persistent node corresponding to the job first in the plurality of scheduling nodes, the scheduling node triggers scheduling of the job according to the scheduling instruction. And if the scheduling node does not successfully create the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node stops creating the persistent node corresponding to the job.
Based on the same inventive concept, an embodiment of the present invention further provides a batch job scheduling processing apparatus, as described in the following embodiments. Because the principles of solving the problems are similar to the batch job scheduling processing method, the implementation of the device can refer to the implementation of the method, and repeated details are not repeated.
Fig. 2 is a structural diagram of a batch job scheduling processing apparatus according to an embodiment of the present invention, and as shown in fig. 2, each scheduling node in the apparatus includes:
an instruction receiving module 201, configured to receive a scheduling instruction of a job;
a node creating module 202, configured to create a persistent node corresponding to the job in the zookeeper;
and the scheduling module 203 is configured to trigger scheduling of the job according to the scheduling instruction if the scheduling node successfully creates the persistent node corresponding to the job in the plurality of scheduling nodes first.
In one embodiment, the node creation module 202 is further configured to:
and if the scheduling node does not successfully create the persistent node corresponding to the job in the scheduling nodes firstly, stopping creating the persistent node corresponding to the job.
In one embodiment, as shown in fig. 3, the batch job scheduling apparatus in fig. 2 further includes:
an initialization module 204, configured to initialize the data table where the job is located and the zookeeper before receiving a scheduling instruction of the job.
In order to implement scheduling of batch jobs, avoid the problems that all job scheduling cannot be performed and the whole batch job is interrupted in execution due to the crash of a core server deploying a scheduling function, and ensure the normal operation of a system, an embodiment of the present invention provides a batch job scheduling processing method, which is applied to a plurality of scheduling nodes, and as shown in fig. 4, the method may include:
obtaining an execution state tag of a previous job corresponding to the job, wherein the execution state tag is as follows: an executed state tag or an unexecuted state tag;
if the execution state labels of all the previous jobs corresponding to the jobs are the executed state labels, generating a scheduling instruction of the jobs;
As shown in fig. 4, in the embodiment of the present invention, the scheduling node generates the scheduling instruction of the job as follows: obtaining an execution state tag of a previous job corresponding to the job, wherein the execution state tag is as follows: an executed state tag or an unexecuted state tag; if the execution state labels of all the previous jobs corresponding to the jobs are the executed state labels, generating a scheduling instruction of the jobs; the scheduling node creates a persistent node corresponding to the job in the zookeeper according to the scheduling instruction of the job; and if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node triggers the scheduling of the job according to the scheduling instruction. The method and the device are applied to a plurality of scheduling nodes, firstly, whether the job scheduling is triggered or not is determined according to the execution state tags of the previous jobs corresponding to the jobs, and if the execution state tags of all the previous jobs corresponding to the jobs are the executed state tags, the scheduling instruction of the jobs is generated, so that the problem that the previous jobs start to run without completing the subsequent jobs is solved, the accurate scheduling of the batch jobs is realized corresponding to the actual job running condition, and the normal running of the system is guaranteed. The persistent nodes corresponding to the jobs are created in the zookeeper by each scheduling node according to the scheduling instructions, if a crash situation occurs in one scheduling node, other scheduling nodes can continue to process the scheduling instructions, the problems that all job scheduling cannot be performed and the whole batch jobs are executed and interrupted due to the crash of one core server deploying the scheduling function are solved, meanwhile, the characteristic that the zookeeper can only successfully create one persistent node for the same job is utilized, and only the scheduling node which successfully creates the persistent node corresponding to the job at the first time can trigger the scheduling of the job according to the scheduling instructions, so that the problem that multiple scheduling nodes repeatedly schedule the same job is solved, and the normal operation of the system is guaranteed.
In specific implementation, the scheduling node generates a scheduling instruction of the job as follows:
obtaining an execution state tag of a previous job corresponding to the job, wherein the execution state tag is as follows: an executed state tag or an unexecuted state tag;
and if the execution state tags of all the previous jobs corresponding to the jobs are the executed state tags, generating a scheduling instruction of the jobs.
The inventor finds that the distributed batch Job scheduling of the core bank X86 system is realized by adopting an Elastic-Job-Lite framework, which is a lightweight decentralized solution and provides a coordination service of distributed tasks in the form of jar packages. Under the framework, the time for starting execution of each job is set, and then the jobs are scheduled to be executed in sequence according to the timing sequence of each job. However, the method cannot accurately correspond to the actual operation condition, the problems that the subsequent operation starts after the previous operation is not completed easily occur, the scheduling accuracy of the batch operation is low, and the normal operation of the system is seriously influenced. Therefore, the embodiment of the invention is applied to a plurality of scheduling nodes, firstly, whether the job scheduling is triggered is determined according to the execution state tags of the previous jobs corresponding to the jobs, and if the execution state tags of all the previous jobs corresponding to the jobs are the executed state tags, the scheduling instruction of the jobs is generated, so that the problem that the previous jobs start to run without completing the subsequent jobs is solved, the accurate scheduling of the batch jobs is realized corresponding to the actual job running condition, and the normal running of the system is ensured.
In an embodiment, the executed state tag comprises: an executed token and a timestamp, where the timestamp is an execution date, such as "COMPLETED + date".
During specific implementation, the scheduling node creates a persistent node corresponding to the job in the zookeeper according to the scheduling instruction of the job.
In specific implementation, if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes first, the scheduling node triggers scheduling of the job according to the scheduling instruction.
The inventor finds that Zookeeper is software for providing a consistency service for distributed applications, and the provided functions comprise: configuration maintenance, domain name service, distributed synchronization, group service, etc. The Zookeeper comprises four types of nodes, namely a persistent node, a persistent sequence numbering catalog node, a temporary catalog node and a temporary sequence numbering catalog node, wherein the persistent node is a node which can exist all the time after being created and can not be deleted due to the fact that a client session is effective. Therefore, the embodiment of the invention applies the characteristic of the persistent node in the Zookeeper to the scheduling of the batch jobs, each scheduling node is locked by creating the persistent node through the Zookeeper, only the scheduling node which is successfully created firstly can take the resource lock to perform job scheduling, and other nodes do not schedule the job repeatedly.
In an embodiment, if the scheduling node successfully creates the persistent node corresponding to the job first in the plurality of scheduling nodes, the scheduling node triggers scheduling of the job according to the scheduling instruction. And if the scheduling node does not successfully create the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node stops creating the persistent node corresponding to the job.
Based on the same inventive concept, an embodiment of the present invention further provides a batch job scheduling processing apparatus, as described in the following embodiments. Because the principles of solving the problems are similar to the batch job scheduling processing method, the implementation of the device can refer to the implementation of the method, and repeated details are not repeated.
Fig. 5 is a structural diagram of a batch job scheduling processing apparatus according to an embodiment of the present invention, and as shown in fig. 5, each scheduling node in the apparatus includes:
an instruction generating module 501, configured to generate a scheduling instruction of a job as follows:
obtaining an execution state tag of a previous job corresponding to the job, wherein the execution state tag is as follows: an executed state tag or an unexecuted state tag;
if the execution state labels of all the previous jobs corresponding to the jobs are the executed state labels, generating a scheduling instruction of the jobs;
a node creating module 502, configured to create a persistent node corresponding to the job in the zookeeper according to the scheduling instruction of the job;
a scheduling module 503, configured to trigger scheduling of the job according to the scheduling instruction if the scheduling node successfully creates the persistent node corresponding to the job in the plurality of scheduling nodes first.
In one embodiment, the node creation module 502 is further configured to:
and if the scheduling node does not successfully create the persistent node corresponding to the job in the scheduling nodes firstly, stopping creating the persistent node corresponding to the job.
In summary, the embodiments of the present invention are applied to a plurality of scheduling nodes, and receive scheduling instructions of jobs through the scheduling nodes; the scheduling node creates a persistent node corresponding to the job in the zookeeper; and if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node triggers the scheduling of the job according to the scheduling instruction. In the embodiment of the invention, a plurality of scheduling nodes receive scheduling instructions of jobs, each scheduling node creates a persistent node corresponding to the jobs in a zookeeper, if a crash situation occurs in one scheduling node, other scheduling nodes can continue to process the scheduling instructions, so that the problems that all job scheduling cannot be performed and the whole batch jobs are executed and interrupted due to the crash of a core server deploying a scheduling function are avoided, meanwhile, the characteristic that only one persistent node can be successfully created for the same job by the zookeeper is utilized, and only the scheduling node which creates the persistent node corresponding to the job at the earliest in order can trigger the scheduling of the job according to the scheduling instructions, so that the problem that the same job is repeatedly scheduled by a plurality of scheduling nodes is avoided, and the normal operation of a system is ensured.
The embodiment of the invention generates the scheduling instruction of the job by the scheduling node according to the following modes: obtaining an execution state tag of a previous job corresponding to the job, wherein the execution state tag is as follows: an executed state tag or an unexecuted state tag; if the execution state labels of all the previous jobs corresponding to the jobs are the executed state labels, generating a scheduling instruction of the jobs; the scheduling node creates a persistent node corresponding to the job in the zookeeper according to the scheduling instruction of the job; and if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node triggers the scheduling of the job according to the scheduling instruction. The method and the device are applied to a plurality of scheduling nodes, firstly, whether the job scheduling is triggered or not is determined according to the execution state tags of the previous jobs corresponding to the jobs, and if the execution state tags of all the previous jobs corresponding to the jobs are the executed state tags, the scheduling instruction of the jobs is generated, so that the problem that the previous jobs start to run without completing the subsequent jobs is solved, the accurate scheduling of the batch jobs is realized corresponding to the actual job running condition, and the normal running of the system is guaranteed. The persistent nodes corresponding to the jobs are created in the zookeeper by each scheduling node according to the scheduling instructions, if a crash situation occurs in one scheduling node, other scheduling nodes can continue to process the scheduling instructions, the problems that all job scheduling cannot be performed and the whole batch jobs are executed and interrupted due to the crash of one core server deploying the scheduling function are solved, meanwhile, the characteristic that the zookeeper can only successfully create one persistent node for the same job is utilized, and only the scheduling node which successfully creates the persistent node corresponding to the job at the first time can trigger the scheduling of the job according to the scheduling instructions, so that the problem that multiple scheduling nodes repeatedly schedule the same job is solved, and the normal operation of the system is guaranteed.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are intended to illustrate the objects, technical solutions and advantages of the present invention in further detail, and it should be understood that the above-mentioned embodiments are only exemplary embodiments of the present invention, and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (12)
1. A batch job scheduling processing method is applied to a plurality of scheduling nodes, and is characterized by comprising the following steps:
a scheduling node receives a scheduling instruction of a job;
the scheduling node creates a persistent node corresponding to the job in the zookeeper;
and if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node triggers the scheduling of the job according to the scheduling instruction.
2. The method of claim 1, further comprising:
and if the scheduling node does not successfully create the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node stops creating the persistent node corresponding to the job.
3. The method of claim 1, further comprising:
before receiving a scheduling instruction of a job, the scheduling node initializes a data table in which the job is located and a zookeeper.
4. A batch job scheduling processing method is applied to a plurality of scheduling nodes, and is characterized by comprising the following steps:
the scheduling node generates a scheduling instruction of the job as follows:
obtaining an execution state tag of a previous job corresponding to the job, wherein the execution state tag is as follows: an executed state tag or an unexecuted state tag;
if the execution state labels of all the previous jobs corresponding to the jobs are the executed state labels, generating a scheduling instruction of the jobs;
the scheduling node creates a persistent node corresponding to the job in the zookeeper according to the scheduling instruction of the job;
and if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node triggers the scheduling of the job according to the scheduling instruction.
5. The method of claim 4, further comprising:
and if the scheduling node does not successfully create the persistent node corresponding to the job in the scheduling nodes firstly, the scheduling node stops creating the persistent node corresponding to the job.
6. A batch job scheduling device is applied to a plurality of scheduling nodes, wherein each scheduling node comprises:
the instruction receiving module is used for receiving a scheduling instruction of the job;
the node creating module is used for creating a persistent node corresponding to the job in the zookeeper;
and the scheduling module is used for triggering the scheduling of the job according to the scheduling instruction if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly.
7. The batch job scheduling apparatus of claim 6, wherein the node creation module is further to:
and if the scheduling node does not successfully create the persistent node corresponding to the job in the scheduling nodes firstly, stopping creating the persistent node corresponding to the job.
8. The batch job scheduling apparatus according to claim 6, further comprising:
the initialization module is used for initializing the data table where the job is located and the zookeeper before receiving a scheduling instruction of the job.
9. A batch job scheduling device is applied to a plurality of scheduling nodes, wherein each scheduling node comprises:
the instruction generation module is used for generating a scheduling instruction of the job as follows:
obtaining an execution state tag of a previous job corresponding to the job, wherein the execution state tag is as follows: an executed state tag or an unexecuted state tag;
if the execution state labels of all the previous jobs corresponding to the jobs are the executed state labels, generating a scheduling instruction of the jobs;
the node creating module is used for creating a persistent node corresponding to the job in the zookeeper according to the scheduling instruction of the job;
and the scheduling module is used for triggering the scheduling of the job according to the scheduling instruction if the scheduling node successfully creates the persistent node corresponding to the job in the scheduling nodes firstly.
10. The batch job scheduling apparatus of claim 9, wherein the node creation module is further to:
and if the scheduling node does not successfully create the persistent node corresponding to the job in the scheduling nodes firstly, stopping creating the persistent node corresponding to the job.
11. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 5 when executing the computer program.
12. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for executing the method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911010066.8A CN110764910A (en) | 2019-10-23 | 2019-10-23 | Batch job scheduling processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911010066.8A CN110764910A (en) | 2019-10-23 | 2019-10-23 | Batch job scheduling processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110764910A true CN110764910A (en) | 2020-02-07 |
Family
ID=69333170
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911010066.8A Pending CN110764910A (en) | 2019-10-23 | 2019-10-23 | Batch job scheduling processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110764910A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103197960A (en) * | 2013-04-12 | 2013-07-10 | 中国银行股份有限公司 | Scheduling method and scheduling system for batch job system |
| CN104536811A (en) * | 2014-12-26 | 2015-04-22 | 广州华多网络科技有限公司 | HIVE task based task scheduling method and device |
| CN105791354A (en) * | 2014-12-23 | 2016-07-20 | 中兴通讯股份有限公司 | Job scheduling method and cloud scheduling server |
| WO2017018978A1 (en) * | 2015-07-24 | 2017-02-02 | Hewlett Packard Enterprise Development Lp | Scheduling jobs in a computing cluster |
| US20180081722A1 (en) * | 2016-09-20 | 2018-03-22 | International Business Machines Corporation | Multi-platform scheduler for permanent and transient applications |
| CN110287022A (en) * | 2019-05-28 | 2019-09-27 | 北京大米科技有限公司 | A kind of scheduling node selection method, device, storage medium and server |
-
2019
- 2019-10-23 CN CN201911010066.8A patent/CN110764910A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103197960A (en) * | 2013-04-12 | 2013-07-10 | 中国银行股份有限公司 | Scheduling method and scheduling system for batch job system |
| CN105791354A (en) * | 2014-12-23 | 2016-07-20 | 中兴通讯股份有限公司 | Job scheduling method and cloud scheduling server |
| CN104536811A (en) * | 2014-12-26 | 2015-04-22 | 广州华多网络科技有限公司 | HIVE task based task scheduling method and device |
| WO2017018978A1 (en) * | 2015-07-24 | 2017-02-02 | Hewlett Packard Enterprise Development Lp | Scheduling jobs in a computing cluster |
| US20180081722A1 (en) * | 2016-09-20 | 2018-03-22 | International Business Machines Corporation | Multi-platform scheduler for permanent and transient applications |
| CN110287022A (en) * | 2019-05-28 | 2019-09-27 | 北京大米科技有限公司 | A kind of scheduling node selection method, device, storage medium and server |
Non-Patent Citations (1)
| Title |
|---|
| 周龙骧著: "《数据库管理系统实现技术》", 31 December 1990 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Fagg et al. | FT-MPI: Fault tolerant MPI, supporting dynamic applications in a dynamic world | |
| US8862933B2 (en) | Apparatus, systems and methods for deployment and management of distributed computing systems and applications | |
| CN110888721A (en) | Task scheduling method and related device | |
| EP2630567B1 (en) | Coordinated upgrades in distributed systems | |
| CN106708740B (en) | Script testing method and device | |
| CN113569987A (en) | Model training method and device | |
| CN113220431B (en) | Cross-cloud distributed data task scheduling method, device and storage medium | |
| CN108710532A (en) | Across dependence implementation method, device, equipment and the storage medium of dispatching platform | |
| US9983988B1 (en) | Resuming testing after a destructive event | |
| CN110569113A (en) | Method and system for scheduling distributed tasks and computer readable storage medium | |
| CN110611707A (en) | Task scheduling method and device | |
| CN111190732A (en) | Timed task processing system and method, storage medium and electronic device | |
| CN105955820A (en) | Method, device, and system for job stream control | |
| CN110019144A (en) | A kind of method and system of big data platform data O&M | |
| KR102602151B1 (en) | How to implement sequencing plans that ensure low-latency communication between real-time tasks | |
| CN113467908B (en) | Task execution method, device, computer readable storage medium and terminal equipment | |
| CN109684051A (en) | A kind of method and system of the hybrid asynchronous submission of big data task | |
| CN112131188B (en) | Batch file distribution processing method and device | |
| CN110633914A (en) | Batch job scheduling processing method and device | |
| CN110764910A (en) | Batch job scheduling processing method and device | |
| CN109426564A (en) | A kind of IT resource data acquisition unifies manufacture timeout control method and system | |
| US9766940B2 (en) | Enabling dynamic job configuration in mapreduce | |
| CN112817706A (en) | Distributed task scheduling system and method | |
| CN113326098B (en) | Cloud management platform supporting KVM virtualization and container virtualization | |
| CN110782040A (en) | Method, device, equipment and medium for training tasks of pitorch |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200207 |