CN115048195A - Job flow implementation method and scheduling system based on MQ and relational database - Google Patents
Job flow implementation method and scheduling system based on MQ and relational database Download PDFInfo
- Publication number
- CN115048195A CN115048195A CN202210425439.3A CN202210425439A CN115048195A CN 115048195 A CN115048195 A CN 115048195A CN 202210425439 A CN202210425439 A CN 202210425439A CN 115048195 A CN115048195 A CN 115048195A
- Authority
- CN
- China
- Prior art keywords
- task
- execution
- queue
- message
- job
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 51
- 230000008569 process Effects 0.000 claims abstract description 17
- 230000001360 synchronised effect Effects 0.000 claims abstract description 16
- 230000002159 abnormal effect Effects 0.000 claims abstract description 4
- 230000000977 initiatory effect Effects 0.000 claims description 5
- 238000013515 script Methods 0.000 claims description 3
- 230000009191 jumping Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a method for realizing job flow based on MQ and a relational database and a scheduling system, wherein the method comprises the following steps: step S1, the operation triggering module initiates a new task, and writes the MQ message corresponding to the first task in the configured operation schedule into the task queue; step S2, the father process in the operation dispatching module monitors the task queue in real time, reads MQ information from the task queue, writes the execution MQ information to the execution queue after finding the position of the MQ information task in the operation schedule, updates and registers a new task according to the execution state, and initiates the next task MQ; step S3, the task execution module reads the MQ message from the execution queue, and after the execution is finished, the execution result is synchronously or asynchronously replied. Compared with the prior art, the method and the system can realize the automatic operation of the operation flow by simply configuring each task executed in sequence, and have the advantages of simultaneously supporting synchronous and asynchronous operation, failure jump and abnormal breakpoint continuous operation.
Description
Technical Field
The invention relates to the technical field of job scheduling control, in particular to a job flow implementation method and a scheduling system based on MQ and a relational database.
Background
In banking systems, it is often necessary to complete a set of transactions (tasks) in sequence for an operation object (e.g., a text).
Through retrieval, chinese patent CN112613840B discloses a job scheduling control method, which comprises: responding to a task execution instruction, and acquiring a preset workflow task state table; determining the operation state of the task to be executed according to the task execution instruction and the task state table; determining a job scheduling sequence according to the job state of the task to be executed; and controlling the tasks to be executed to execute the tasks to be executed according to the job scheduling sequence. The job scheduling method and the job scheduling system determine the job scheduling sequence through the job state information of the tasks to be executed, the job scheduling sequence can be determined by the task execution instruction and the preset workflow task state table, and then the job scheduling of the tasks to be executed is obtained.
However, it is difficult for the above technologies to support synchronous and asynchronous tasks, job breakpoint continuous operation and failed task jumping at the same time, and it is necessary to design a job invoking system and a job flow implementing method for implementing the above functions.
Disclosure of Invention
The invention aims to overcome the defects of the prior art and provides a job flow implementation method and a scheduling system based on MQ and a relational database.
The purpose of the invention can be realized by the following technical scheme:
according to a first aspect of the present invention, there is provided a data stream implementation method based on MQ and relational database, the method comprising the steps of:
step S1, the operation triggering module initiates a new task, and writes the MQ message corresponding to the first task in the configured operation schedule into the task queue;
step S2, the father process in the operation dispatching module monitors the task queue in real time, reads MQ information from the task queue, writes the execution MQ information to the execution queue after finding the position of the MQ information task in the operation schedule, updates and registers a new task according to the execution state, and initiates the next task MQ;
step S3, the task execution module reads the MQ message from the execution queue, and after the execution is finished, the execution result is synchronously or asynchronously replied.
Preferably, the step S2 includes the following sub-steps:
step S21, the father process in the job scheduling module monitors the task queue in real time, reads MQ information from the task queue, and processes the fork child process;
step S22, the subprocess analyzes the read MQ message according to the job scheduling interface and carries out null judgment on the task ID in the task execution record table; if the task ID is empty, after inquiring various items of information of a first task of the corresponding operation in the operation schedule according to the operation number, turning to the step S26, otherwise, turning to the step S23;
step S23, judging whether the corresponding task exists in the task execution record table, if not, determining the corresponding task as an invalid MQ message, and turning to step S27; if the corresponding task exists in the task execution record table, turning to step S24;
step S24, judging whether the content of the reply item is a reply message, if yes, turning to step S25; if not, writing MQ messages into the execution queue according to the task execution interface, reading reply messages from the reply queue if the task inquired from the job schedule is a synchronous execution task, analyzing the MQ messages according to the job scheduling interface, and turning to the step S25, otherwise, turning to the step S27;
step S25, changing the state of the task, inquiring the next task information or the failure skip task information from the operation schedule according to the current state, and turning to step S26;
step S26, generating a task ID for the new task, inserting the new task into a task execution record table, and writing MQ messages into the task queue according to the job scheduling interface;
and step S27, exiting the sub-process.
Preferably, the fields of the job schedule table include job number, task name, synchronous and asynchronous flag, and failed jump task number; wherein, the "job number + task number" is the primary key.
Preferably, the fields of the task execution record table include a task ID, an operation object name, a job number, a task name, a start time, an end time, and a task execution state; wherein, the task ID is a primary key.
Preferably, the fields corresponding to the job scheduling interface include an operation object name, a task ID, a job number, and reply content;
the task ID is generated by a job scheduling module sequence, the task ID of each task is unique, and if the task ID is a null value, the task is a first task in a job flow; the reply content is success, failure and new task.
Preferably, the fields corresponding to the job scheduling interface include an operation object name, a task ID, a transaction code and a trigger time.
Preferably, the MQ messages read by the task queue in step S2 include a first MQ message written by the job triggering module, a second MQ message written by the job scheduling module, and a third MQ message written by the task executing module;
the first MQ message is a first task for triggering the job flow; the second MQ message is used for triggering a new task which is recorded and is not executed yet; the third MQ message is an asynchronous task reply message.
Preferably, the specific transactions performed by the task execution module include extensible C, Java and Shell scripts.
Preferably, if the abnormal interruption of the job flow occurs, the MQ message is rewritten in the task queue through an external program, and the corresponding task is executed again.
According to a second aspect of the present invention, there is provided a MQ and relational database based job scheduling system, comprising:
the job triggering module is used for initiating a new task and writing a first task MQ message in the job schedule into the task queue;
the operation scheduling module is used for reading the MQ message from the task queue, writing the execution MQ message to the execution queue after finding the position of the MQ message task in the operation schedule, updating and registering a new task according to the execution state, and initiating the next task MQ;
and the task execution module is used for reading the MQ message from the execution queue and synchronously or asynchronously replying an execution result after the execution is finished.
Compared with the prior art, the invention has the following advantages:
1) the job flow implementation method realizes the automatic operation of the job flow by simply configuring various tasks executed in sequence, and simultaneously supports synchronous tasks, asynchronous tasks, failed task skipping and job abnormal breakpoint continuous operation;
2) the method is based on MQ and a relational database, the input and the output of all modules are carried out through queues, the coupling degree is small, and the transportability is strong;
3) the operation schedule and the task execution record table are configurable, the configuration among the modules is simple, and the development amount is reduced.
Drawings
FIG. 1 is a first flow chart of a workflow implementation of the present invention;
FIG. 2 is a flow chart of a workflow implementation of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be obtained by a person skilled in the art without any inventive step based on the embodiments of the present invention, shall fall within the scope of protection of the present invention.
Examples
Interpretation of terms:
operation: one Job (Job) includes a plurality of tasks (tasks). After all tasks are executed, the operation is finished;
and (3) synchronizing tasks: after the job scheduling initiates a task (writing MQ messages in the queue), the execution result of the task needs to be waited, and the task is called a synchronous task. If the execution time of the task is short, the task can be designed as a synchronous task;
asynchronous task: after the job scheduling initiates the task (writing MQ message in the queue), other operations can be continuously executed without waiting for the execution result of the task, and the task is called an asynchronous task. Since the job scheduling does not need to wait for the execution result of the asynchronous task, the resources can be released immediately, so that the asynchronous task can be designed to be asynchronous for some tasks with longer execution time.
The system embodiment of the invention is given firstly, a job flow scheduling system based on MQ and relational database comprises a job triggering module, a job scheduling module and a task execution module, wherein the modules communicate with each other through MQ messages, and the system embodiment specifically comprises the following steps:
the operation trigger is used for initiating a new task and writing a first task MQ message in the operation schedule into the task queue;
the operation scheduling module is used for reading the MQ message from the task queue, writing the execution MQ message to the execution queue after finding the position of the MQ message task in the operation schedule, updating and registering a new task according to the execution state, and launching the next task MQ;
and the task execution module is used for reading the MQ message from the execution queue, and synchronously or asynchronously replying the execution result based on the MQ and the relational database after the execution is finished.
Wherein, tasks needing to be executed in sequence are configured in the operation schedule.
The following provides an embodiment of the method of the present invention, and a data stream implementation method based on MQ and relational database, the method includes the following steps:
step S1, the operation triggering module initiates a new task, and writes the MQ message corresponding to the first task in the configured operation schedule into the task queue;
step S2, the parent process in the job scheduling module monitors the task queue in real time, reads the MQ message from the task queue, finds the position of the MQ message task in the job schedule, writes the execution MQ message to the execution queue, updates and registers a new task according to the execution state, and initiates the next task MQ, specifically:
step S21, the father process in the job scheduling module monitors the task queue in real time, reads MQ information from the task queue, and processes the fork child process;
step S22, the subprocess analyzes the read MQ message according to the job scheduling interface and carries out null judgment on the task ID in the task execution record table; if the task ID is null, after inquiring various items of information of a first task of the corresponding operation in the operation schedule according to the operation number, turning to the step S26, otherwise, turning to the step S23;
step S23, judging whether the corresponding task exists in the task execution record table, if not, determining the corresponding task as an invalid MQ message, and turning to step S27; if the corresponding task exists in the task execution record table, turning to step S24;
step S24, judging whether the content of the reply item is a reply message, if yes, turning to step S25; if not, writing MQ messages into the execution queue according to the task execution interface, reading reply messages from the reply queue if the task inquired from the job schedule is a synchronous execution task, analyzing the MQ messages according to the job scheduling interface, and turning to the step S25, otherwise, turning to the step S27;
step S25, changing the state of the task, inquiring the next task information or the failure skip task information from the operation schedule according to the current state, and turning to step S26;
step S26, generating a task ID for the new task, inserting the new task into a task execution record table, and writing MQ messages into the task queue according to the job scheduling interface;
step S27, quitting the subprocess;
and step S3, the task execution module reads the MQ message from the execution queue, and synchronously or asynchronously replies the execution result after the execution is finished.
The method of the present invention is described in detail below with reference to figures 1 and 2.
[ MQ queue X: QUEUE _ JOB ]: and monitoring the queue by the process of the job scheduling module, and processing by the fork subprocess if the queue has a message. The messages in the queue have 3 sources, MQ message 1 (job trigger module writes, triggers the first task in the job flow), MQ message 2 (job scheduling process writes, triggers a new task that has not yet been executed yet), MQ message 3 (task execution module writes reply message to asynchronous task)
[ MQ queue Y: QUEUE _ EXE _ REQ): and the task execution module monitors the queue, and if the queue has a message, executes the corresponding transaction in the message.
[ MQ queue Z: QUEUE _ EXE _ RSP): for the synchronous task, after the task execution is finished, the execution module writes the reply message into the queue.
Data structure a-job scheduling interface: object name, TASKID (generated by job scheduling module sequence, each task has unique TASKID. if null, the first task in the job flow), JOBNO, reply content (SUCC-success, FAIL, null-new task)
Data structure B-task execution interface: name of operation object, TASKID, transaction code, trigger time (YYYYMMDDHH24MISS)
TABLE 1 task execution RECORDING TABLE
Description of the drawings: recording each specific task ID, execution date, time, state, operation object, corresponding job number and task number
Field: task ID (TASKID, home key), operation object name, job number, task name, start time, end time, task execution status (S-success, F-failure, I-processing)
Sample example: xml of text abc is an operation target in this embodiment
TABLE 1
[ TABLE 2-job Schedule ]
Description of the drawings: all tasks needing to be executed are configured for different jobs, and the tasks are divided into synchronous tasks and asynchronous tasks. For synchronous tasks, the job scheduling module needs to wait for execution results; for asynchronous tasks, the task scheduling subprocess is triggered to exit after execution, and system resources are not occupied.
A field: job number, task name, synchronous and asynchronous mark and failure skip task number; wherein the job number + the task number is a main key; and if the execution of the task is successful, sequentially executing the next task, and if the execution fails, the new task is a failed jump numbering task.
Sample example: wherein the time required by the 0001-text splitting and 0005-posting tasks is longer, so the tasks are set to be asynchronous tasks; and if the 0001-text splitting execution fails, jumping to the 0006-text reply task.
TABLE 2
| Job number | Task numbering | Task name | Synchronous and asynchronous sign | Failed jump task numbering |
| JOB0001 | 0001 | Text splitting | Asynchronous | 0006 |
| JOB0001 | 0002 | Answering | Synchronization | |
| JOB0001 | 0003 | Missing information entry | Synchronization | |
| JOB0001 | 0004 | Type identification | Synchronization | |
| JOB0001 | 0005 | Posting account | Asynchronous | |
| JOB0001 | 0006 | Text reply | Synchronization |
[ job trigger ]: write MQ messages to [ MQ queue X: QUEUE _ JOB ], comprising: name of operation object (text name), TASKID (null, trigger job first task, no TASKID generated yet), reply content (null)
[ job trigger ]: the resident process listens [ MQ queue X: QUEUE _ JOB ], if there is a message, the fork sub-process proceeds.
Job trigger (subprocess): and analyzing the MQ message, and splitting the object name, the TASKID, the JOBNO and the reply content.
(branch 1) if TASKID is empty, generating a new TASKID according to the first task of the JOBNO in [ table 2-job schedule ], inserting [ table 1-task execution record table ], writing MQ message to [ MQ queue X: QUEUE _ JOB ].
(branch 2.2) if the TASKID is not empty and there is no corresponding task in [ table 1-task execution log table ], then the sub-process is exited as an invalid MQ message.
(branch 2.1.1) if the TASKID is not empty and the corresponding task can be queried in the [ table 1-task execution record table ], and the MQ message is a reply message, updating the [ table 1-task execution record table ] according to the reply state, sequentially looking up the next execution task in the [ table 2-job schedule table (the reply state is successful), or looking up the failed jump task of the task from the [ table 2-job schedule table (the reply state is failed), generating the TASKID for the new task, inserting the related information of the new task into the [ table 1-task execution record table ], and finally writing the message to the [ MQ queue X: QUEUE _ JOB ].
(branch 2.1.2.1) if TASKID is not empty, the corresponding task can be queried in [ table 1-task execution records table ], MQ message is not reply message, and from [ table 2-job schedule table ] query that the task (JOBNO + TASKNO) is a synchronous execution task, then write MQ message to [ queue Y: QUEUE _ EXE _ REQ ], comprising: operand name, TASKID, transaction code, trigger time (YYYYMMDDHH24MISS), and listen [ queue Z: QUEUE _ EXE _ RSP ], reading the reply of the task execution module. According to the reply state, updating [ table 1-task execution record table ], sequentially searching the next execution task (the reply state is successful) in [ table 2-operation schedule table ], or searching the failed jump task (the reply state is failed) of the task from [ table 2-operation schedule table ], generating a TASKID for the new task, inserting the related information of the new task into [ table 1-task execution record table ], and finally writing the MQ message to [ MQ queue X: QUEUE _ JOB ].
(branch 2.1.2.2) if TASKID is not empty, the corresponding task can be queried in [ table 1-task execution records table ], MQ message is not reply message, and from [ table 2-job schedule table ] query that the task (JOBNO + TASKNO) is an asynchronous execution task, then write MQ message to [ queue Y: QUEUE _ EXE _ REQ ] comprising: the operation object name, TASKID, transaction code, trigger time (YYYYMMDDHH24MISS), and exit the sub-process.
[ task execution ]: resident process snoop [ MQ queue Y: QUEUE _ EXE _ REQ), the name of the operation object, TASKID, transaction code, etc. are parsed from the message. And executing corresponding tasks according to the transaction codes. And generates a reply MQ message, which the synchronous transaction then writes to [ MQ queue Z: quese _ EXE _ RSP, otherwise write to MQ QUEUE X: QUEUE _ JOB ]
Description of the drawings: the specific transaction executed by the module can be C, java, shell script and the like, and is extensible.
The application mainly describes a job flow implementation method, and specific executed transactions are not the core content of the text and are not described in detail herein;
if a job flow abort occurs, the transaction may be terminated by the external program at [ MQ queue X: QUEUE _ JOB ] to re-write the message and re-execute the corresponding task. The MQ message only needs to conform to a data structure a-job scheduling interface, and the specific implementation manner of the external program can refer to the prior art.
While the invention has been described with reference to specific embodiments, the invention is not limited thereto, and various equivalent modifications and substitutions can be easily made by those skilled in the art within the technical scope of the invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A method for implementing data stream based on MQ and relational database, the method comprising the steps of:
step S1, the operation triggering module initiates a new task, and writes the MQ message corresponding to the first task in the configured operation schedule into the task queue;
step S2, the father process in the operation dispatching module monitors the task queue in real time, reads MQ information from the task queue, writes the execution MQ information to the execution queue after finding the position of the MQ information task in the operation schedule, updates and registers a new task according to the execution state, and initiates the next task MQ;
step S3, the task execution module reads the MQ message from the execution queue, and after the execution is finished, the execution result is synchronously or asynchronously replied.
2. The MQ and relational database-based data stream implementation method according to claim 1, wherein the step S2 comprises the sub-steps of:
step S21, the father process in the job scheduling module monitors the task queue in real time, reads MQ information from the task queue, and processes the fork child process;
step S22, the subprocess analyzes the read MQ message according to the job scheduling interface and carries out null judgment on the task ID in the task execution record list; if the task ID is empty, after inquiring various items of information of a first task of the corresponding operation in the operation schedule according to the operation number, turning to the step S26, otherwise, turning to the step S23;
step S23, judging whether the corresponding task exists in the task execution record table, if not, determining the corresponding task as an invalid MQ message, and turning to step S27; if the corresponding task exists in the task execution record table, turning to step S24;
step S24, judging whether the content of the reply item is a reply message, if yes, turning to step S25; if not, writing MQ messages into the execution queue according to the task execution interface, reading reply messages from the reply queue if the task inquired from the job schedule is a synchronous execution task, analyzing the MQ messages according to the job scheduling interface, and turning to the step S25, otherwise, turning to the step S27;
step S25, changing the state of the task, inquiring the next task information or the failure skip task information from the operation schedule according to the current state, and turning to step S26;
step S26, generating a task ID for the new task, inserting the new task into a task execution record table, and writing MQ messages into the task queue according to the job scheduling interface;
and step S27, exiting the sub-process.
3. The MQ and relational database based data stream implementation method of claim 2, wherein the fields of the job schedule comprise job number, task name, homoasynchronous flag, failed jump task number; wherein, the "job number + task number" is the primary key.
4. The MQ and relational database based data stream implementation method of claim 2, wherein the fields of the task execution record table comprise task ID, operand name, job number, task name, start time, end time and task execution status; wherein the task ID is a primary key.
5. The MQ and relational database based data stream implementation method according to claim 2, wherein the fields corresponding to the job scheduling interface comprise an operation object name, a task ID, a job number and reply content;
the task ID is generated by a job scheduling module sequence, the task ID of each task is unique, and if the task ID is a null value, the task is a first task in a job flow; the reply content is success, failure and new task.
6. The MQ and relational database based data stream implementation method of claim 2, wherein the fields corresponding to the job scheduling interface comprise an operation object name, a task ID, a transaction code and a trigger time.
7. The MQ and relational database-based data stream implementation method according to claim 1, wherein the MQ messages read by the task queue in the step S2 comprise a first MQ message written by the job trigger module, a second MQ message written by the job scheduling module, and a third MQ message written by the task execution module;
the first MQ message is a first task for triggering the job flow; the second MQ message is used for triggering a new task which is recorded and is not executed yet; the third MQ message is an asynchronous task reply message.
8. The MQ and relational database based data stream implementation method of claim 1, wherein the specific transactions performed by the task execution module comprise extensible C, Java and Shell scripts.
9. The MQ and relational database based data stream implementation method as claimed in claim 1, wherein if abnormal interruption of the job stream occurs, the MQ message is rewritten in the task queue by an external program to re-execute the corresponding task.
10. A MQ and relational database based job flow scheduling system, comprising:
the job triggering module is used for initiating a new task and writing a first task MQ message in the job schedule into the task queue;
the operation scheduling module is used for reading the MQ message from the task queue, writing the execution MQ message to the execution queue after finding the position of the MQ message task in the operation schedule, updating and registering a new task according to the execution state, and initiating the next task MQ;
and the task execution module is used for reading the MQ message from the execution queue and synchronously or asynchronously replying the execution result after the execution is finished.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210425439.3A CN115048195A (en) | 2022-04-21 | 2022-04-21 | Job flow implementation method and scheduling system based on MQ and relational database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210425439.3A CN115048195A (en) | 2022-04-21 | 2022-04-21 | Job flow implementation method and scheduling system based on MQ and relational database |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115048195A true CN115048195A (en) | 2022-09-13 |
Family
ID=83157299
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210425439.3A Pending CN115048195A (en) | 2022-04-21 | 2022-04-21 | Job flow implementation method and scheduling system based on MQ and relational database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115048195A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116485302A (en) * | 2023-04-06 | 2023-07-25 | 宝驷智慧物流(珠海)有限公司 | Control system, method, equipment and medium for cross-border transportation |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120324462A1 (en) * | 2009-10-31 | 2012-12-20 | Rutgers, The State University Of New Jersey | Virtual flow pipelining processing architecture |
| CN102981904A (en) * | 2011-09-02 | 2013-03-20 | 阿里巴巴集团控股有限公司 | Task scheduling method and system |
| CN112667375A (en) * | 2020-12-22 | 2021-04-16 | 杭州东信北邮信息技术有限公司 | Task scheduling method and system based on big data service |
-
2022
- 2022-04-21 CN CN202210425439.3A patent/CN115048195A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120324462A1 (en) * | 2009-10-31 | 2012-12-20 | Rutgers, The State University Of New Jersey | Virtual flow pipelining processing architecture |
| CN102981904A (en) * | 2011-09-02 | 2013-03-20 | 阿里巴巴集团控股有限公司 | Task scheduling method and system |
| CN112667375A (en) * | 2020-12-22 | 2021-04-16 | 杭州东信北邮信息技术有限公司 | Task scheduling method and system based on big data service |
Non-Patent Citations (1)
| Title |
|---|
| 李鑫等: "数据流Eager传输：一种分布式流体系结构中的性能优化技术", 《计算机工程与科学》, vol. 37, no. 11, 15 November 2015 (2015-11-15), pages 2035 - 2044 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116485302A (en) * | 2023-04-06 | 2023-07-25 | 宝驷智慧物流(珠海)有限公司 | Control system, method, equipment and medium for cross-border transportation |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9311064B1 (en) | Systems and methods for automated centralized build/merge management | |
| JP4283328B2 (en) | A method for scheduling and associating events with partially ordered transactions | |
| CN101567013B (en) | Method and apparatus for implementing ETL scheduling | |
| CN110941502A (en) | Message processing method, device, storage medium and equipment | |
| CN107577586B (en) | Method and equipment for determining service execution link in distributed system | |
| US20090260011A1 (en) | Command line transactions | |
| US20120221536A1 (en) | System and method for data transfer | |
| CN111400011B (en) | Real-time task scheduling method, system, equipment and readable storage medium | |
| CN113656116B (en) | Business process processing method and device | |
| US11704139B2 (en) | Service processing method and apparatus, electronic device, and storage medium | |
| CN111767346B (en) | Database data synchronization method, device, equipment and storage medium | |
| CN115048195A (en) | Job flow implementation method and scheduling system based on MQ and relational database | |
| CN113515297A (en) | Version updating method and device, electronic equipment and storage medium | |
| CN116644035B (en) | File batch warehousing method, device, equipment and storage medium | |
| JP2526020B2 (en) | Exception condition processing method and apparatus | |
| CN111984447A (en) | Registration compensation system and method in overtime or abnormal situation of bank transaction | |
| CN113297081B (en) | Execution method and device of continuous integrated pipeline | |
| US20030088572A1 (en) | Method, computer program product, and system for unloading a hierarchical database utilizing segment specific selection critera | |
| CN116701011A (en) | Algorithm service dispatching management system based on rule engine | |
| CN111124370A (en) | Data processing method and related equipment | |
| CN107844340B (en) | Method and system for data acquisition and file downloading | |
| CN106055322A (en) | Flow scheduling method and device | |
| CN111078666B (en) | Automatic unloading and supplying method based on multi-database crossing center | |
| CN111831271B (en) | Git code review system supporting simulation pre-assembly pipeline and method thereof | |
| CN112579156A (en) | Processing system, processing method, processing device and processing equipment of business event |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |