CN111400011A - Real-time task scheduling method, system, equipment and readable storage medium - Google Patents
Real-time task scheduling method, system, equipment and readable storage medium Download PDFInfo
- Publication number
- CN111400011A CN111400011A CN202010195854.5A CN202010195854A CN111400011A CN 111400011 A CN111400011 A CN 111400011A CN 202010195854 A CN202010195854 A CN 202010195854A CN 111400011 A CN111400011 A CN 111400011A
- Authority
- CN
- China
- Prior art keywords
- job
- group
- jobs
- execution
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000003860 storage Methods 0.000 title claims abstract description 13
- 238000012545 processing Methods 0.000 claims description 44
- 230000002159 abnormal effect Effects 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 16
- 230000008569 process Effects 0.000 claims description 8
- 238000012937 correction Methods 0.000 claims description 6
- 238000010586 diagram Methods 0.000 description 14
- 230000006870 function Effects 0.000 description 7
- 238000012805 post-processing Methods 0.000 description 5
- 230000000737 periodic effect Effects 0.000 description 4
- 238000001914 filtration Methods 0.000 description 3
- 238000003780 insertion Methods 0.000 description 3
- 230000037431 insertion Effects 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 2
- 239000012634 fragment Substances 0.000 description 2
- 238000012423 maintenance Methods 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 238000013475 authorization Methods 0.000 description 1
- 239000007795 chemical reaction product Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013499 data model Methods 0.000 description 1
- 238000013524 data verification Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- QHLFJSMAWAWVRU-UHFFFAOYSA-N n-(5-chloro-4-methyl-1,3-thiazol-2-yl)propanamide Chemical compound CCC(=O)NC1=NC(C)=C(Cl)S1 QHLFJSMAWAWVRU-UHFFFAOYSA-N 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 238000011112 process operation Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The application discloses a real-time task scheduling method, a real-time task scheduling system, real-time task scheduling equipment and a readable storage medium, wherein the method comprises the following steps: the batch reservation external interface receives the operation instruction and reads the operation configuration information in the database; generating a corresponding job group execution instruction according to the job configuration information, and sending the job group execution instruction to target execution equipment; after receiving the job group feedback information of the job group execution instruction executed by the target execution equipment, inquiring whether the job related to the job logic exists or not according to the job group feedback information and the job logic rule information; and when determining that the jobs associated with the job logic exist, scheduling the jobs associated with the job logic until all jobs in a job group to which the jobs associated with the job logic belong are scheduled to be completed. Therefore, batch operation unified operation is realized, and homogeneous and heterogeneous operations can be configured and operated cooperatively.
Description
Technical Field
The present application relates to the field of computers and communications, and in particular, to a real-time task scheduling method, system, device, and readable storage medium.
Background
This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.
With the rapid development of bank electronic banking business, the functions of electronic channels such as online banking, mobile banking APP and the like are gradually improved, people can enjoy high-quality service of banks at any time and any place, and the electronic banks provide convenient service for people, and are stable and efficient operations of a plurality of large-scale online transaction systems supporting the electronic banks, and the transaction systems have a large amount of batch operations for processing mass data of bank background DB. At present, the data is an important asset of banks in a big data era, batch operation for processing the data is efficient and stable to operate, and the data is an important matter in the security banking industry.
However, the processing method in the prior art is difficult to implement real-time monitoring and automatic processing, and in order to better manage and maintain the safe and stable operation of the batch jobs, a uniform batch scheduling platform is required to schedule and monitor the operation of the batch jobs. The existing job scheduling technical scheme has at least the following defects: the first scheme only schedules isomorphic jobs, and does not schedule heterogeneous jobs of different types; while the second scheme does not describe the logical relationship and branching decisions between jobs.
Disclosure of Invention
The embodiment of the application provides a real-time task scheduling method, a real-time task scheduling system, a real-time task scheduling device and a readable storage medium, and the method can be used for uniformly operating and scheduling all data by inquiring, terminating, running again and the like task instances of a job group, and enabling isomorphic and heterogeneous jobs to be uniformly configured and cooperatively work.
In a first aspect, an embodiment of the present application provides a real-time task scheduling method, where the method includes:
receiving a job instruction through a batch reservation external interface, and reading job configuration information in a database, wherein the job configuration information comprises reserved job time, job number and job group number corresponding to the job, and the database is used for storing the job configuration information and job logic rule information;
generating a corresponding job group execution instruction according to the job configuration information, and sending the job group execution instruction to target execution equipment;
after receiving the job group feedback information of the job group execution instruction executed by the target execution equipment, inquiring whether the job related to the job logic exists or not according to the job group feedback information and the job logic rule information;
and when determining that the jobs associated with the job logic exist, scheduling the jobs associated with the job logic until all jobs in a job group to which the jobs associated with the job logic belong are scheduled to be completed.
Optionally, the querying whether there is a job associated with the job logic according to the job group feedback information and the job logic rule information includes:
analyzing the feedback information of the job group, and determining the execution state of the jobs in the job group;
and inquiring the logic rule information according to the execution state of the jobs in the job group, and determining whether the jobs logically associated with the jobs exist.
Optionally, the job configuration information further includes: job type, target device information, and job content information.
Optionally, after sending the job group execution instruction corresponding to the job to the target execution device, the method further includes:
and if receiving the information that the target execution equipment fails to execute the job group, performing automatic rerun processing on the jobs in the job group with failed execution.
Optionally, the performing automatic rerun processing on the jobs in the job group with failed execution includes:
searching abnormal data according to the database node corresponding to the operation in the operation group with failed execution, and performing correction processing;
after correcting the abnormal data, skipping the job which is successfully executed, and executing the job which is failed due to the abnormal data.
Optionally, the method further comprises: and correspondingly executing data related to one database node by each job, and simultaneously recording the execution condition of each job.
Optionally, the method further comprises: and when determining that the job logically associated with the job does not exist, finishing processing the job.
In a second aspect, an embodiment of the present application provides a real-time task scheduling system, where the system includes:
the system comprises a job configuration information reading module, a database and a data processing module, wherein the job configuration information reading module is used for receiving job instructions through a batch reservation external interface and reading job configuration information in the database, the job configuration information comprises reserved job time, job number and job group number corresponding to a job, and the database is used for storing the job configuration information and job logic rule information;
the job group execution instruction module is used for generating a corresponding job group execution instruction according to the job configuration information and sending the job group execution instruction to the target execution equipment;
the state query module is used for querying whether the operation related to the operation logic exists or not according to the operation group feedback information and the operation logic rule information after receiving the operation group feedback information of the operation group execution indication executed by the target execution equipment;
and the job scheduling module is used for scheduling the job related to the job logic until all jobs in the job group to which the job related to the job logic belongs are scheduled and finished when the job related to the job logic exists.
Optionally, the status query module is specifically configured to: analyzing the feedback information of the job group, and determining the execution state of the jobs in the job group;
and inquiring the logic rule information according to the execution state of the jobs in the job group, and determining whether the jobs logically associated with the jobs exist.
Optionally, the job configuration information further includes: job type, target device information, and job content information.
Optionally, the system further comprises: and the rerun module is used for carrying out automatic rerun processing on the jobs in the job group with execution failure if the information that the target execution equipment fails to execute the job group is received.
Optionally, the rerun module is specifically configured to: searching abnormal data according to the database node corresponding to the operation in the operation group with failed execution, and performing correction processing; after correcting the abnormal data, skipping the job which is successfully executed, and executing the job which is failed due to the abnormal data.
Optionally, the system further comprises: and the state recording module is used for correspondingly executing data related to one database node by each job and simultaneously recording the execution condition of each job.
Optionally, the job scheduling module is further configured to: and when determining that the job logically associated with the job does not exist, finishing processing the job.
In a third aspect, an embodiment of the present application further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements any one of the methods of the first aspect when executing the computer program.
In a fourth aspect, this application further provides a computer-readable storage medium storing a computer program for executing any one of the methods in the first aspect.
To sum up, the real-time task scheduling method, system, device and readable storage medium provided by the embodiments of the present application receive a job instruction through a batch reservation external interface, and read job configuration information in a database, where the job configuration information includes a reserved job time, a job number and a job group number corresponding to a job, and the database is used to store the job configuration information and job logic rule information; generating a corresponding job group execution instruction according to the job configuration information, and sending the job group execution instruction to target execution equipment; after receiving the job group feedback information of the job group execution instruction executed by the target execution equipment, inquiring whether the job related to the job logic exists or not according to the job group feedback information and the job logic rule information; and when determining that the jobs associated with the job logic exist, scheduling the jobs associated with the job logic until all jobs in a job group to which the jobs associated with the job logic belong are scheduled to be completed. The method realizes the unified operation of batch operation, and enables the isomorphic operation and the heterogeneous operation to be configured in a unified way and to work in a cooperative way.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts. In the drawings:
fig. 1 is a schematic flowchart of a real-time task scheduling method provided in an embodiment of the present application;
FIG. 2 is a schematic diagram of a workgroup architecture provided in an embodiment of the present application;
FIG. 3 is a diagram of an embodiment of a real-time task scheduling method provided in an embodiment of the present application;
fig. 4 is a schematic diagram of a real-time task scheduling method provided in an embodiment of the present application;
fig. 5 is a schematic diagram of a real-time task scheduling system provided in an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
The principles and spirit of the present application are explained in detail below with reference to several representative embodiments of the present application.
Although the present application provides method operational steps or apparatus configurations as illustrated in the following examples or figures, more or fewer operational steps or modular units may be included in the methods or apparatus based on conventional or non-inventive efforts. In the case of steps or structures which do not logically have the necessary cause and effect relationship, the execution sequence of the steps or the module structure of the apparatus is not limited to the execution sequence or the module structure shown in the embodiment or the drawings of the present application. The described methods or modular structures, when applied in an actual device or end product, may be executed sequentially or in parallel according to embodiments or the methods or modular structures shown in the figures.
Technical terms referred to in the embodiments of the present application will be first described below.
Asynchronous and batch tasks: also called automatic tasks, called tasks for short. The English and Chinese of the data dictionary is abbreviated as Aschn _ Tsk, and the frame code is generally called as Task. Is a collection of minimum schedulable unit jobs. According to the scheduling time, the scheduling method can be divided into periodic tasks (which are repeatedly executed in a certain period), real-time tasks (which need to be executed immediately after being generated), and timing tasks (which are to be executed at a certain designated time).
Operation: the English and Chinese of the data dictionary is abbreviated as Job, and the code is generally called Job. The task is a minimum unit which can be scheduled in an automatic task, a plurality of jobs form a task in a sequence, branch and combined job flow mode, and the task is a functional module capable of completing a specific operation. The typical operation comprises three steps of pretreatment, single treatment and post-treatment.
The operation group: a batch of same jobs are processed by the same service, have a common job group ID, and can carry out operations such as inquiry, termination, rerun and the like uniformly through the job group ID.
And (3) treatment: the processing is a step in a job, and a typical job includes three steps of pre-processing, single processing, and post-processing.
The preprocessor is responsible for preparing data, the corresponding field in the job table is the name of the preprocessor service component (PSH _ BFR _ PCSG _ SVC _ NM), and for the case of large data volume, the preprocessor also provides the function of acquiring data in pages instead of acquiring all data at once.
The single processing is responsible for processing single data, and a corresponding field in the job table is a single processing service component name (SVC _ CMPT _ NM) which is specific logic for processing each piece of single data distributed by the processing.
The post-processing is responsible for summarizing the results, the corresponding field in the job table is the name of the post-processing service component (PSH _ AF _ PCSG _ SVC _ NM), the post-processing can calculate the current processing progress according to the total number of data distributed by the pre-processing and the processed data amount of a single processing, and the state in the table is updated when all the processing is completed.
A single piece of processing is mandatory to define, pre-and post-processing may not be defined, and the framework will provide default processing logic.
Defining: the English and Chinese in the data dictionary is abbreviated as Def, and the code is generally called Def or Definition. The method comprises effective time period information and period information, is special for the periodic task, and has no definition on real-time and timing tasks. The periodic task and the JOB included therein have definitions, which are referred to as a task definition and a JOB definition, respectively, and are stored in a task definition table (S08T1_ ASCHN _ TSK _ DEF) and a JOB definition table (S08T1_ JOB _ DEF). The framework will continually generate instances based on the defined period information and valid period information.
Example (c): english in the data dictionary is abbreviated as Exmp, and the code is generally called Exmp or Instance. Only once information of a specific execution time. Real-time and timing tasks are only examples and are not defined; each specific execution of a periodic task generated according to a definition forms an instance. The instances also include task instances and JOB instances, which are saved in a task instance table (S08T1_ ASCHN _ TSK _ EXMP) and a JOB instance table (S08T1_ JOB _ EXMP), respectively.
Scanning: commonly referred to as Scan or Scanner in the code. Is the process of filtering the definitions and instances in the auto task table. For definition, the scanning process is an instantiated process (section 2.2 will be described with emphasis), in this process, the task definition is a main scanning dimension, and the period information and the valid time period information contained in the task definition are main filtering conditions; for an example, the process of scanning is an execution process, in which a job instance is a main scanning dimension, and job execution state information contained in the job instance is a main filtering condition.
The processing mode of batch jobs is generally divided into real-time batches, quasi-real-time asynchronous batches and daily final batches, wherein the online small batches respond the requirements of clients in real time by online transaction driving, but more batches related to big data are the quasi-real-time asynchronous batches and the daily final batches, the quasi-real-time asynchronous batches uniformly run and execute batch jobs according to a fixed period after receiving transaction requests of the clients, the daily final batch jobs are used for processing mass data generated by a transaction system in the evening, the jobs support safe and stable execution of banking services, the bank online transaction system is used as a 7 × 24-hour system, the safe and stable execution of the system is a first element, the management requirements for a large number of batch jobs are higher and higher, a batch job console performs uniform automatic scheduling management on the batch jobs, convenient and flexible parameterized job setting is provided, scheduling supporting different periods of the batch jobs is greatly improved, the jobs can be executed in different batch servers under uniform scheduling of batch control, the consumption of batch jobs on system resources is reduced, a monitoring mechanism of batch jobs is perfected, and the alarm of batch jobs operation monitoring and the normal operation is greatly avoided from being influenced by abnormal and unstable operation.
When a large amount of data is operated, in order to ensure the execution efficiency, the distributed or multi-thread execution operation needs to be performed after splitting. But to ensure that some unified queries and unified operations can be performed on the data after splitting. To this end, an embodiment of the present application provides a real-time task scheduling method, as shown in fig. 1, the method includes the following steps:
step 101: the method comprises the steps of receiving a job instruction through a batch reservation external interface, reading job configuration information in a database, wherein the job configuration information comprises reserved job time, job number and job group number corresponding to the job, and the database is used for storing the job configuration information and job logic rule information.
Step 102: and generating a corresponding job group execution instruction according to the job configuration information, and sending the instruction to the target execution equipment.
Step 103: and after receiving the job group feedback information of the job group execution instruction executed by the target execution equipment, inquiring whether the job related to the job logic exists or not according to the job group feedback information and the job logic rule information.
Step 104: and when determining that the jobs associated with the job logic exist, scheduling the jobs associated with the job logic until all jobs in a job group to which the jobs associated with the job logic belong are scheduled to be completed.
In some embodiments, there may be a plurality of target execution devices, and the target execution devices may be devices having intelligent job execution functions, such as middleware servers, application servers, databases, short message servers, and the like, for executing various types of homogeneous and heterogeneous jobs in the system.
In step 103, the querying whether there is a job associated with the job logic according to the job group feedback information and the job logic rule information includes: analyzing the feedback information of the job group, and determining the execution state of the jobs in the job group; and inquiring the logic rule information according to the execution state of the jobs in the job group, and determining whether the jobs logically associated with the jobs exist.
In step 103, after receiving the information that the target execution device fails to execute the job group, performing automatic rerun processing on the jobs in the job group that failed to execute. The automatic rerun processing of the jobs in the job group with failed execution specifically includes: searching abnormal data according to the database node corresponding to the operation in the operation group with failed execution, and performing correction processing; after correcting the abnormal data, skipping the job which is successfully executed, and executing the job which is failed due to the abnormal data.
The real-time task scheduling method of the embodiment of the application adopts a failed operation automatic rerun mode, but not all failed batches can be directly sent to a rerun mechanism, for example, batches related to financial funds can be executed again after operation and maintenance personnel position and investigate the reason. The operation and maintenance personnel can identify the operation which can fail and automatically rerun according to the characteristics of batch operation, a failed rerun mechanism is arranged on the console, and when the management server of the console captures the execution failure of the operation, a warning notice short message is sent out, and meanwhile rerun is automatically sent out.
In some embodiments, each job correspondingly executes data related to one of the database nodes, and the execution condition of each job is recorded.
In some embodiments, processing the job is terminated upon determining that there is no job logically associated with the job.
In some embodiments, a database stores job configuration information for a plurality of homogeneous, heterogeneous jobs, the job configuration information including: information such as a job group number, a job type, a target terminal, job contents, authorization information, specific instructions and the like, wherein the job type comprises application operation, database operation, middleware operation, network operation and the like; the operation configuration information is set through a uniform and standardized abstract architecture, so that the difference of different types of operations can be shielded, the cooperation between isomorphic and heterogeneous operations is realized, and various isomorphic and heterogeneous operations such as process operation, file system, database data operation, server operation and the like can be uniformly configured and cooperatively operated.
In some embodiments, the logic rule information describes the logic relationship and branch judgment between jobs, uses a standardized rule language to configure the relationship of circulation, jump and the like between jobs, and implements the isomorphic and heterogeneous job cooperation job under a distributed or centralized system architecture by configuring the logic rules between jobs.
The scheme provides an operation interface for the whole job group, can inquire the execution state of the whole job group instance, terminate the unexecuted job instance, rerun the job instance with failed execution, and can reserve the jobs in batch.
A large batch of data operations needs to be split into multiple task instance executions. To ensure that all data execution is completed and all data can be operated uniformly, the task instances of the job group can be queried, terminated, rerun and the like. FIG. 2 shows a schematic diagram of an embodiment of a job group, and as shown in FIG. 2, each job group has a plurality of task instances of an automatic task, and each task instance corresponds to one job instance. The technical scheme is designed for overcoming the defect that the quasi-real-time tasks cannot be reserved and processed in batches. The business side can perform batch reservation and batch insertion on tasks of the same job group, so that reservation efficiency is improved. For the tasks of the job group, batch operations may be performed. The interrupt and rerun logic can ensure more flexible control of the execution of the task of the job group.
Fig. 3 shows a schematic diagram of an embodiment of the batch reservation external interface, which is to organize data from a database to return to the batch reservation external interface through data verification, data preparation and data insertion stages. The data checking module is used for checking the capacity of the non-empty fields and the fragment ID set, and checking the lengths of other fields by the database; the data preparation module is used for preparing an AschTskExmp instance inserted into the database and a jobExmp set to which the AschTskExmp instance belongs; the database insertion module is used for inserting the prepared data into a database; and the organization return data module is used for returning the type Map < String, String > wherein key represents fragment ID and Value represents job instance primary key ID.
FIG. 4 shows a second embodiment, which is to return a specified data list from the database through stages of data checking, updating corresponding job status, and querying corresponding job list, wherein the data checking module is used for checking non-null fields, the field length check is handed to the database for checking, the updating corresponding job status module is used for updating the queried job status according to specified conditions, the querying corresponding job list module is used for obtaining the queried job status according to the specified conditions, the returning specified data list module is returned, the micro-service return type is specified Vo type, and the interface return type is L ist < String >.
In short, the task information of each job group is recorded and fed back to the database. And capturing the states of the tasks according to the recorded task group tasks. And simultaneously updating the task state. And the grasped state is inconsistent with the expected state, and the running is restarted. This job group is completed until all task states meet the job group requirements. The data of the same batch can be uniformly managed during processing, and the operation of inquiring, terminating, running again and the like on the operation instance of the operation group is ensured.
The embodiment of the application mainly relates to 2 data models. Only the essential key fields are listed in the table below, and other detailed fields are not directly related to the present invention and are not listed.
The task example table is shown in the following table (1):
watch (1)
The operation example table is shown in the following table (2):
watch (2)
To sum up, the real-time task scheduling method provided by the embodiment of the present application receives a job instruction through a batch reservation external interface, reads job configuration information in a database, where the job configuration information includes reserved job time, job number and job group number corresponding to a job, and the database is used to store the job configuration information and job logic rule information; generating a corresponding job group execution instruction according to the job configuration information, and sending the job group execution instruction to target execution equipment; after receiving the job group feedback information of the job group execution instruction executed by the target execution equipment, inquiring whether the job related to the job logic exists or not according to the job group feedback information and the job logic rule information; and when determining that the jobs associated with the job logic exist, scheduling the jobs associated with the job logic until all jobs in a job group to which the jobs associated with the job logic belong are scheduled to be completed. The method realizes the unified operation of batch operation, and enables the isomorphic operation and the heterogeneous operation to be configured in a unified way and to work in a cooperative way.
Based on the same technical concept, an embodiment of the present application provides a real-time task scheduling system, as shown in fig. 5, the system includes:
the job configuration information reading module 501 is configured to receive a job instruction through the batch reservation external interface, and read job configuration information in a database, where the job configuration information includes reserved job time, job number, and job group number corresponding to a job, and the database is configured to store the job configuration information and job logic rule information.
The job group execution instruction module 502 is configured to generate a corresponding job group execution instruction according to the job configuration information, and send the job group execution instruction to the target execution device.
The status query module 503 is configured to, after receiving the job group feedback information indicating that the target execution device executes the job group, query whether there is a job associated with the job logic according to the job group feedback information and the job logic rule information.
And the job scheduling module 504 is configured to schedule the job associated with the job logic until all jobs in the job group to which the job associated with the job logic belongs are scheduled.
In some embodiments, the status query module 503 is specifically configured to: analyzing the feedback information of the job group, and determining the execution state of the jobs in the job group; and inquiring the logic rule information according to the execution state of the jobs in the job group, and determining whether the jobs logically associated with the jobs exist.
In some embodiments, the job configuration information further includes: job type, target device information, and job content information.
In certain embodiments, the system further comprises: and the rerun module is used for carrying out automatic rerun processing on the jobs in the job group with execution failure if the information that the target execution equipment fails to execute the job group is received.
In some embodiments, the rerun module is specifically configured to: searching abnormal data according to the database node corresponding to the operation in the operation group with failed execution, and performing correction processing; after correcting the abnormal data, skipping the job which is successfully executed, and executing the job which is failed due to the abnormal data.
In certain embodiments, the system further comprises: and the state recording module is used for correspondingly executing data related to one database node by each job and simultaneously recording the execution condition of each job.
In some embodiments, the job scheduling module 504 is further configured to: and when determining that the job logically associated with the job does not exist, finishing processing the job.
Based on the same technical concept, the embodiment of the present application further provides a computer device, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor implements the above method when executing the computer program.
Based on the same technical concept, the embodiment of the present application further provides a computer-readable storage medium storing a computer program of the above method.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
The above-mentioned embodiments are further described in detail for the purpose of illustrating the invention, and it should be understood that the above-mentioned embodiments are only illustrative of the present invention and are not intended to limit the scope of the present invention, and any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (16)
1. A real-time task scheduling method, comprising:
receiving a job instruction through a batch reservation external interface, and reading job configuration information in a database, wherein the job configuration information comprises reserved job time, job number and job group number corresponding to the job, and the database is used for storing the job configuration information and job logic rule information;
generating a corresponding job group execution instruction according to the job configuration information, and sending the job group execution instruction to target execution equipment;
after receiving the job group feedback information of the job group execution instruction executed by the target execution equipment, inquiring whether the job related to the job logic exists or not according to the job group feedback information and the job logic rule information;
and when determining that the jobs associated with the job logic exist, scheduling the jobs associated with the job logic until all jobs in a job group to which the jobs associated with the job logic belong are scheduled to be completed.
2. The method of claim 1, wherein said querying whether a job associated with the job logic exists based on the job group feedback information and job logic rule information comprises:
analyzing the feedback information of the job group, and determining the execution state of the jobs in the job group;
and inquiring the logic rule information according to the execution state of the jobs in the job group, and determining whether the jobs logically associated with the jobs exist.
3. The method of claim 1, wherein the job configuration information further comprises: job type, target device information, and job content information.
4. The method of claim 1, wherein after sending the job group execution indication corresponding to the job to the target execution device, the method further comprises:
and if receiving the information that the target execution equipment fails to execute the job group, performing automatic rerun processing on the jobs in the job group with failed execution.
5. The method of claim 4, wherein performing an auto-rerun process on jobs in the job group that failed execution comprises:
searching abnormal data according to the database node corresponding to the operation in the operation group with failed execution, and performing correction processing;
after correcting the abnormal data, skipping the job which is successfully executed, and executing the job which is failed due to the abnormal data.
6. The method of claim 1, wherein the method further comprises:
and correspondingly executing data related to one database node by each job, and simultaneously recording the execution condition of each job.
7. The method of claim 1, wherein the method further comprises:
and when determining that the job logically associated with the job does not exist, finishing processing the job.
8. A real-time task scheduling system, the system comprising:
the system comprises a job configuration information reading module, a database and a data processing module, wherein the job configuration information reading module is used for receiving job instructions through a batch reservation external interface and reading job configuration information in the database, the job configuration information comprises reserved job time, job number and job group number corresponding to a job, and the database is used for storing the job configuration information and job logic rule information;
the job group execution instruction module is used for generating a corresponding job group execution instruction according to the job configuration information and sending the job group execution instruction to the target execution equipment;
the state query module is used for querying whether the operation related to the operation logic exists or not according to the operation group feedback information and the operation logic rule information after receiving the operation group feedback information of the operation group execution indication executed by the target execution equipment;
and the job scheduling module is used for scheduling the job related to the job logic until all jobs in the job group to which the job related to the job logic belongs are scheduled and finished when the job related to the job logic exists.
9. The system of claim 8, wherein the status query module is specifically configured to: analyzing the feedback information of the job group, and determining the execution state of the jobs in the job group;
and inquiring the logic rule information according to the execution state of the jobs in the job group, and determining whether the jobs logically associated with the jobs exist.
10. The system of claim 8, wherein the job configuration information further comprises: job type, target device information, and job content information.
11. The system of claim 8, wherein the system further comprises:
and the rerun module is used for carrying out automatic rerun processing on the jobs in the job group with execution failure if the information that the target execution equipment fails to execute the job group is received.
12. The system of claim 11, wherein the rerunning module is specifically configured to: searching abnormal data according to the database node corresponding to the operation in the operation group with failed execution, and performing correction processing;
after correcting the abnormal data, skipping the job which is successfully executed, and executing the job which is failed due to the abnormal data.
13. The system of claim 8, wherein the system further comprises:
and the state recording module is used for correspondingly executing data related to one database node by each job and simultaneously recording the execution condition of each job.
14. The system of claim 8, wherein the job scheduling module is further to:
and when determining that the job logically associated with the job does not exist, finishing processing the job.
15. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 7 when executing the computer program.
16. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for executing the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010195854.5A CN111400011B (en) | 2020-03-19 | 2020-03-19 | Real-time task scheduling method, system, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010195854.5A CN111400011B (en) | 2020-03-19 | 2020-03-19 | Real-time task scheduling method, system, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111400011A true CN111400011A (en) | 2020-07-10 |
| CN111400011B CN111400011B (en) | 2023-10-20 |
Family
ID=71434370
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010195854.5A Active CN111400011B (en) | 2020-03-19 | 2020-03-19 | Real-time task scheduling method, system, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111400011B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112035233A (en) * | 2020-09-01 | 2020-12-04 | 中国银行股份有限公司 | Big data batch job task scheduling method and device |
| CN112559158A (en) * | 2020-12-29 | 2021-03-26 | 中国人民银行清算总中心 | Micro-service timing task scheduling method and device |
| CN112579320A (en) * | 2020-12-16 | 2021-03-30 | 中国建设银行股份有限公司 | Communication method and device between open system and ZOS system |
| CN113010278A (en) * | 2021-02-19 | 2021-06-22 | 建信金融科技有限责任公司 | Batch processing method and system for financial insurance core system |
| CN113204420A (en) * | 2021-05-28 | 2021-08-03 | 中国工商银行股份有限公司 | Job scheduling method and device based on distributed cache |
| CN114070845A (en) * | 2021-11-11 | 2022-02-18 | 中国银行股份有限公司 | Method and device for cooperatively submitting transaction information |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104765641A (en) * | 2014-01-02 | 2015-07-08 | 中国移动通信集团安徽有限公司 | Job scheduling method and system |
| CN109379400A (en) * | 2018-09-04 | 2019-02-22 | 中国建设银行股份有限公司 | Batch jobs dispatch deal system, method, apparatus and storage medium |
| CN110308979A (en) * | 2019-06-27 | 2019-10-08 | 苏宁消费金融有限公司 | The asynchronous process frame and its implementation of task based access control scheduling |
| CN110727539A (en) * | 2019-12-19 | 2020-01-24 | 北京江融信科技有限公司 | Method and system for processing exception in batch processing task and electronic equipment |
| CN110825535A (en) * | 2019-10-12 | 2020-02-21 | 中国建设银行股份有限公司 | Job scheduling method and system |
| CN110879756A (en) * | 2019-10-31 | 2020-03-13 | 深圳市泰洲科技有限公司 | Method and device for processing abnormal cooperation process, computer equipment and storage medium |
-
2020
- 2020-03-19 CN CN202010195854.5A patent/CN111400011B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104765641A (en) * | 2014-01-02 | 2015-07-08 | 中国移动通信集团安徽有限公司 | Job scheduling method and system |
| CN109379400A (en) * | 2018-09-04 | 2019-02-22 | 中国建设银行股份有限公司 | Batch jobs dispatch deal system, method, apparatus and storage medium |
| CN110308979A (en) * | 2019-06-27 | 2019-10-08 | 苏宁消费金融有限公司 | The asynchronous process frame and its implementation of task based access control scheduling |
| CN110825535A (en) * | 2019-10-12 | 2020-02-21 | 中国建设银行股份有限公司 | Job scheduling method and system |
| CN110879756A (en) * | 2019-10-31 | 2020-03-13 | 深圳市泰洲科技有限公司 | Method and device for processing abnormal cooperation process, computer equipment and storage medium |
| CN110727539A (en) * | 2019-12-19 | 2020-01-24 | 北京江融信科技有限公司 | Method and system for processing exception in batch processing task and electronic equipment |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112035233A (en) * | 2020-09-01 | 2020-12-04 | 中国银行股份有限公司 | Big data batch job task scheduling method and device |
| CN112579320A (en) * | 2020-12-16 | 2021-03-30 | 中国建设银行股份有限公司 | Communication method and device between open system and ZOS system |
| CN112559158A (en) * | 2020-12-29 | 2021-03-26 | 中国人民银行清算总中心 | Micro-service timing task scheduling method and device |
| CN113010278A (en) * | 2021-02-19 | 2021-06-22 | 建信金融科技有限责任公司 | Batch processing method and system for financial insurance core system |
| CN113010278B (en) * | 2021-02-19 | 2023-03-28 | 建信金融科技有限责任公司 | Batch processing method and system for financial insurance core system |
| CN113204420A (en) * | 2021-05-28 | 2021-08-03 | 中国工商银行股份有限公司 | Job scheduling method and device based on distributed cache |
| CN114070845A (en) * | 2021-11-11 | 2022-02-18 | 中国银行股份有限公司 | Method and device for cooperatively submitting transaction information |
| CN114070845B (en) * | 2021-11-11 | 2024-04-16 | 中国银行股份有限公司 | Method and device for cooperatively reporting transaction information |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111400011B (en) | 2023-10-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111400011B (en) | Real-time task scheduling method, system, equipment and readable storage medium | |
| US8938421B2 (en) | Method and a system for synchronizing data | |
| CN108076098B (en) | Service processing method and system | |
| CN109684057A (en) | Task processing method, device and storage medium | |
| CN103425462A (en) | Method and device for workflow data persistence | |
| CN101201753A (en) | Method for configuring and managing multimode machine supervising engine | |
| CN111580939B (en) | Method and device for processing transactions in hierarchical and asynchronous mode | |
| CN109669976A (en) | Data service method and equipment based on ETL | |
| CN111210340B (en) | Automatic task processing method, device, server and storage medium | |
| CN112559300A (en) | Fault reason determining system, method and device | |
| CN112559525B (en) | Data checking system, method, device and server | |
| CN113157411A (en) | Reliable configurable task system and device based on Celery | |
| CN112035230A (en) | Method and device for generating task scheduling file and storage medium | |
| CN116719623A (en) | Job scheduling method, job result processing method and device | |
| CN113793213B (en) | Method and device for implementing decision mode of asynchronous credit wind control breakpoint continuous operation | |
| CN115756765A (en) | Distributed transaction processing method and device | |
| CN115437766A (en) | Task processing method and device | |
| CN113837870B (en) | Financial risk data approval method and device | |
| CN112527497B (en) | Serialized multithreaded data processing system | |
| KR101888131B1 (en) | Method for Performing Real-Time Changed Data Publish Service of DDS-DBMS Integration Tool | |
| CN111401837B (en) | Event processing method and system, and computer readable storage medium | |
| CN111130955B (en) | Distributed link monitoring method based on internet credit system | |
| CN113986537A (en) | Middleware management method and device | |
| CN114579280A (en) | Quasi-real-time scheduling method and system | |
| CN117667362B (en) | Method, system, equipment and readable medium for scheduling process engine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220907 Address after: 25 Financial Street, Xicheng District, Beijing 100033 Applicant after: CHINA CONSTRUCTION BANK Corp. Address before: 25 Financial Street, Xicheng District, Beijing 100033 Applicant before: CHINA CONSTRUCTION BANK Corp. Applicant before: Jianxin Financial Science and Technology Co.,Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |