CN116302390A - Data scheduling method and device - Google Patents
Data scheduling method and device Download PDFInfo
- Publication number
- CN116302390A CN116302390A CN202310034998.6A CN202310034998A CN116302390A CN 116302390 A CN116302390 A CN 116302390A CN 202310034998 A CN202310034998 A CN 202310034998A CN 116302390 A CN116302390 A CN 116302390A
- Authority
- CN
- China
- Prior art keywords
- task
- scheduling
- data
- database
- preset
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 66
- 230000008569 process Effects 0.000 claims abstract description 21
- 238000012544 monitoring process Methods 0.000 claims description 18
- 230000004048 modification Effects 0.000 abstract description 15
- 238000012986 modification Methods 0.000 abstract description 15
- 230000006870 function Effects 0.000 abstract description 11
- 238000012423 maintenance Methods 0.000 abstract description 6
- 238000000926 separation method Methods 0.000 abstract description 6
- 238000013461 design Methods 0.000 description 12
- 239000010453 quartz Substances 0.000 description 9
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N silicon dioxide Inorganic materials O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 230000003068 static effect Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000005856 abnormality Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 239000003607 modifier Substances 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000009517 secondary packaging Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application discloses a data scheduling method and device, task data of an application program are obtained, the task data are configured into a preset table of a database according to a preset field form, the task data in the database are loaded into a task driving engine through preset scheduling task parameters, and the task data in the task driving engine are executed through the preset scheduling parameters. According to the scheme, an operator does not need to log in an application server to carry out manual modification in the process of updating task configuration, the functions of decoupling between task scheduling configuration and service codes, centralization of task scheduling data, convenience in maintenance and the like are realized only by utilizing a database to provide data dependence of task scheduling, multiple scheduling modes such as front-end and back-end separation, front-end pages and the like are provided for selection, and the task state and task scheduling are controlled, managed and optimized for different scheduling modes, so that the stability and accuracy of task scheduling are ensured, and the efficiency of task scheduling is also improved.
Description
Technical Field
The present disclosure relates to the field of electronic information technologies, and in particular, to a data scheduling method and apparatus.
Background
The domestic civil aviation industry develops rapidly, the domestic freight rate business is diversified and three-dimensional more and more, and the problem of paying attention to how to ensure that freight rate data are accurately, rapidly and stably transmitted to data consumers and provide users with safe and rapid use experience is solved.
In order to accurately, quickly and stably respond to the freight rate data to the data consumers, it is necessary to schedule the execution of application tasks for the timed task configuration schedule to transmit the freight rate data to the data consumers.
The current domestic freight rate service performs application tasks through the configuration and scheduling of a Linux server timing task (corntab), and in the process of updating task configuration, operators need to log in an application server to perform manual modification, so that operation risks are increased, and timeliness and accuracy of task scheduling cannot be ensured.
Therefore, how to ensure timeliness and accuracy of task scheduling is a problem that needs to be solved in the present application.
Disclosure of Invention
In view of this, the application discloses a data scheduling method and device, which aim to control, manage and optimize task state and task scheduling according to different scheduling modes, so as to ensure stability and accuracy of task scheduling and improve efficiency of task scheduling.
In order to achieve the above purpose, the technical scheme disclosed by the method is as follows:
the first aspect of the application discloses a data scheduling method, which comprises the following steps:
acquiring task data of an application program; the task data represents data of a task belonging group and task execution parameters of an application program;
according to a preset field form, configuring the task data into a preset table of a database;
and loading the task data in the database into a task driving engine through preset scheduling task parameters, and executing the task data in the task driving engine through the preset scheduling parameters.
Preferably, the preset table includes a task log table, a task table and a task resource table, and the configuring the task data into the database according to a preset field form includes:
configuring a service log in task data into the task log table of a database through the implementation class of a scheduling interface and a preset incoming method;
configuring a class path of an implementation class of a scheduling task abstract interface in task data into the task table of a database;
and configuring task job resources in the task data into the task resource table of the database.
Preferably, the process of setting the scheduling task parameter includes:
setting triggering time of a scheduling task, triggering parameters of the scheduling task, triggering nodes of the scheduling task, configuration files of a scheduling program in a cluster and cluster names.
Preferably, the loading the task data in the database into the task driving engine through the preset scheduling task parameters, and executing the task data in the task driving engine through the preset scheduling parameters includes:
a task program is deployed and started through a program realization scheduling framework interface;
after the task program is operated, task data in the database is loaded into a task driving engine through preset scheduling task parameters;
judging the type of an execution mode for executing the task data in the task driving engine;
if the execution mode of executing the task data in the task driving engine is a single-point execution mode, executing the task data in the task driving engine through the triggering time of the dispatching task, the triggering parameter of the dispatching task and the triggering node of the dispatching task;
and if the execution mode of executing the task data in the task driving engine is a multi-point execution mode, executing the task data in the task driving engine through the triggering time of the dispatching task, the triggering parameter of the dispatching task, the triggering node of the dispatching task, the configuration file of the dispatching program in the cluster and the cluster name.
Preferably, after executing the task data in the task driving engine through the preset scheduling parameters, the method further includes:
and checking the task running state and the running log through a task log table of the database.
Preferably, the method further comprises:
the state of each node in the cluster deployment is monitored in real time through a task resource table in a database; the state includes an offline state or an online state.
Preferably, the method further comprises:
and monitoring the task log and the cluster execution node address of the task in real time through the task data.
A second aspect of the present application discloses a data scheduling apparatus, the apparatus comprising:
the acquisition unit is used for acquiring task data of the application program; the task data represents data of a task belonging group and task execution parameters of an application program;
the configuration unit is used for configuring the task data into a preset table of a database according to a preset field form;
and the loading execution unit is used for loading the task data in the database into the task driving engine through preset scheduling task parameters and executing the task data in the task driving engine through the preset scheduling parameters.
Preferably, the configuration unit includes:
the first configuration module is used for configuring the business log in the task data into the task log table of the database through the implementation class of the scheduling interface and a preset incoming method;
the second configuration module is used for configuring a class path of an implementation class of the dispatching task abstract interface in the task data into the task table of the database;
and the third configuration module is used for configuring task job resources in the task data into the task resource table of the database.
Preferably, the load execution unit for setting the scheduling task parameter is specifically configured to:
setting triggering time of a scheduling task, triggering parameters of the scheduling task, triggering nodes of the scheduling task, configuration files of a scheduling program in a cluster and cluster names.
According to the technical scheme, the application discloses a data scheduling method and device, task data of an application program are obtained, the task data represent data of a task belonging group and task execution parameters of the application program, the task data are configured into a preset table of a database according to a preset field form, the task data in the database are loaded into a task driving engine through preset scheduling task parameters, and the task data in the task driving engine are executed through the preset scheduling parameters. According to the scheme, an operator does not need to log in an application server to carry out manual modification in the process of updating task configuration, the functions of decoupling between task scheduling configuration and service codes, centralization of task scheduling data, convenience in maintenance and the like are realized only by utilizing a database to provide data dependence of task scheduling, multiple scheduling modes such as front-end and back-end separation, front-end pages and the like are provided for selection, and the task state and task scheduling are controlled, managed and optimized for different scheduling modes, so that the stability and accuracy of task scheduling are ensured, and the efficiency of task scheduling is also improved.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings that are required to be used in the embodiments or the description of the prior art will be briefly described below, and it is obvious that the drawings in the following description are only embodiments of the present application, and that other drawings may be obtained according to the provided drawings without inventive effort to a person skilled in the art.
FIG. 1 is a block diagram of a task scheduling center system disclosed in an embodiment of the present application;
FIG. 2 is a schematic diagram of a task scheduling center according to an embodiment of the present disclosure;
fig. 3 is a schematic flow chart of a data scheduling method disclosed in an embodiment of the present application;
fig. 4 is a schematic structural diagram of a data scheduling device disclosed in an embodiment of the present application;
fig. 5 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
In this application, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.
As known from the background technology, the current domestic freight rate service performs application tasks through the Linux server corntab timing task configuration scheduling, and in the process of updating task configuration, operators need to log in the application server to perform manual modification, so that the operation risk is increased, and the timeliness and accuracy of task scheduling cannot be ensured. Therefore, how to ensure timeliness and accuracy of task scheduling is a problem that needs to be solved in the present application.
In order to solve the problems, the application discloses a data scheduling method and device, in the scheme, an operator does not need to log in an application server to carry out manual modification in the process of updating task configuration, only the database is used for providing data dependence of task scheduling, the functions of decoupling between task scheduling configuration and service codes, centralization of task scheduling data, convenience in maintenance and the like are realized, multiple scheduling modes are provided, such as front-end and back-end separation, front-end pages and the like can be selected, and control management optimization scheduling is carried out on task states and task scheduling according to different scheduling modes, so that the stability and accuracy of task scheduling are ensured, and the efficiency of task scheduling is also improved. Specific implementations are illustrated by the following examples.
The scheme is based on open source project (Quartz) framework and comprises secondary development, dispatcher increment loading task data design, custom scanning task design, task realization interface reflection design and newly added service dimension. Wherein the business dimension includes scheduler resources, tasks, task logs, etc.
Secondary development on the basis of the Quartz framework:
quartz is another open source project of the OpenSymphonyl open source organization in the Job scheduling field, which can be used in conjunction with J2EE and J2SE applications or alone. Quartz can be used to create complex programs that are simple or run tens, hundreds, and even tens of thousands of Jobs.
And on the basis of the latest version of the Quartz, secondary packaging is carried out, so that the stability of task scheduling is ensured by the realized task scheduling center.
When the business application uses the method, a unique interface (ITaskJob. DoTaskJob (Map param)) needs to be realized, and specific task scheduling is realized, and development is completed in the realization method of the interface.
The scheduler incrementally loads task data design:
after the task scheduling center system is started, the task scheduling center system initializes a static variable date, after the task scheduling center system scans data, the maximum value of the last update date of the scheduler in the resource table is stored in the field, and then the next scanning is carried out according to the value when scanning is carried out next time, so that the design of incremental update data is realized, and therefore, after each time of data modification, the newly modified data can be identified by the scheduler, the last update date field of the scheduler in the resource table should be modified at the same time, and the current date is generated as much as possible.
Custom scan task design:
the application node executes timing scanning according to the configuration file and the specified configuration time interval, scans task data, assembles the task data of the scheduler after the task data is processed by the program checksum, and loads the task data into the specified scheduler, and the scheduler can trigger corresponding tasks on time after loading the data of the scheduler.
The task implementation interface reflection design:
an interface method is designed, in the process of system execution, the data of the execution class configured in the task data is obtained through a database, whether the data is the implementation class of the interface is judged through reflection, if the data is the implementation class, the implementation method of the implementation class is executed, the specific service implementation is realized in the implementation class, and the requirements of different services are met through different parameters.
Newly added service dimension scheduler resources, tasks and task logs:
through the design of the database tables with three dimensions, the front-end page can realize that the task management page can check the specific task execution state and task data, and can check the specific task log through the task data and also check the cluster execution node address of the task.
Referring to fig. 1, the task scheduling center system includes a task management component 11, a log management component 12, a data center 13, an operation report component 14, a failure alarm component 15, a system monitoring component 16, and the like.
The task management component 11 can manage schedulers, task modes, task states, and the like. The task mode comprises fixed time execution, repeated execution and the like; task states include pause, run, etc.
The task management component 11 configures corresponding task data according to a task table defined in the database, and is used for loading the task data into a task driving engine after the program is operated; the database is scanned regularly, and the start and stop of the task are controlled through the change of data. Wherein each task has its own parameters and trigger time definition.
Wherein the task management component 11 incrementally loads the task data design via the scheduler. After the task scheduling center system is started, the task scheduling center system initializes a static variable date, after the system scans data, the maximum value of the last update date of the scheduler in the resource table is stored in the field, and then the next scanning is carried out according to the value when the next scanning is carried out, so that the design of incremental update data is realized, and after each time of data modification, the newly modified data can be identified by the scheduler, the last update date field of the scheduler in the resource table is also modified, and the current date is generated as much as possible.
The task data comprises a group to which the task belongs, a task execution strategy, task execution parameters, task alarm mail configuration, task execution interface class paths and the like.
The log management component 12 includes a dispatch log function, a view system running log and operation log function, and the like. The log management component 12 records the execution process of the task in the life cycle of the task execution, and the recorded data is stored in a database, so that the problem can be conveniently checked, managed and maintained.
The running report component 14 is used for carrying out report statistics; the real-time checking of operation data such as the number of tasks, the scheduling times, the number of executors and the like is supported; a scheduling report, a scheduling situation profile, etc. may be generated.
When the task execution fails or the task execution result caused by the occurrence of an abnormality is failure, the failure alarm component 15 may send the failure alarm component to a specified relevant person for alarm reminding processing by means of mail or the like.
On the basis of the Quartz framework, operations such as function authority control and data authority control are performed through the data center 13.
The system monitoring component 16 is configured to view the task running state and the running log through a task log table of the database; the state of each node in the cluster deployment is monitored in real time through a task resource table in a database; the state includes an offline state or an online state.
The interaction process between the specific task management component 11 and the data center 13 is as follows:
the task management component 11 acquires task data of an application program; the task data characterizes data of a task-affiliated group and task execution parameters of the application program.
The data center 13 configures the task data into a preset table of the database according to a preset field form. The preset table comprises a task log table, a task table and a task resource table.
The data center 13 loads task data in the database into the task driving engine through preset scheduling task parameters, and executes the task data in the task driving engine through the preset scheduling parameters. The scheduling task parameters comprise triggering time of the scheduling task, triggering parameters of the scheduling task, triggering nodes of the scheduling task, configuration files of scheduling programs in the cluster, cluster names and the like.
Specifically, the data center 13 deploys and starts a task program through a program implementation scheduling framework interface; after the task program is operated, task data in a database is loaded into a task driving engine through preset scheduling task parameters; judging the type of an execution mode of task data in an execution task driving engine; if the execution mode of the task data in the execution task driving engine is a single-point execution mode, executing the task data in the task driving engine through the triggering time of the dispatching task, the triggering parameter of the dispatching task and the triggering node of the dispatching task; if the execution mode of the task data in the execution task driving engine is a multi-point execution mode, the task data in the task driving engine is executed through the triggering time of the dispatching task, the triggering parameter of the dispatching task, the triggering node of the dispatching task, the configuration file of the dispatching program in the cluster and the cluster name.
The scheme abstracts the scheduling behavior into a common platform of the task scheduling center, the platform does not bear task business logic, the task scheduling center is responsible for initiating scheduling requests, the task scheduling flow is fully asynchronous, and the flow peak clipping can be carried out on intensive scheduling. Decoupling the two parts of the scheduling and the task, and improving the overall stability and expansibility of the task scheduling center system; visual configuration can be easily supported in a data driving mode, and the task is added, deleted and checked through the page. The internal implementation mechanism is completed by using the core scheduling of Quartz, a main task is used for scanning the specified data of the database according to the configured time interval, the operations of adding, deleting, modifying and the like are dynamically carried out on the scheduling data in the scheduler, the changes of the scheduling data in the scheduler are dynamically synchronized, the scheduling data are stored in the database, and later data can be provided for front-end display in various modes or report statistics can be carried out by operating the report component 14.
The task scheduling center system is implemented as shown in fig. 2.
In FIG. 2, scheduler configuration management, task monitoring, system monitoring, and log management are the various functional components of the task scheduling center; the task configuration data center is the data dependence of the task scheduling center; examples of DFI applications, DFP applications, and night-dimensional applications access a dispatch center.
And in the process of accessing the DIF application into the dispatching center, firstly, carrying out DFI task scanning, then, transmitting the scanned DFI task into a task queue to be executed, executing threads from the task queue to be executed through a dispatcher interface, and carrying out log monitoring.
And in the process of accessing the DFP application into the dispatching center, firstly, carrying out DFP task scanning, then, transmitting the scanned DFP task into a task queue to be executed, executing threads from the task queue to be executed through a dispatcher interface, and carrying out log monitoring.
And in the process of accessing the night-dimensional application into the dispatching center, firstly, performing night-dimensional task scanning, then, transmitting the scanned night-dimensional task into a task queue to be executed, executing threads from the task queue to be executed through a dispatcher interface, and performing log monitoring.
The design target of the timing dispatching center is light, convenient to operate, simple to access, rapid to develop and stable in function. Simple operation: supporting the web page to carry out CRUD operation on the task; dynamic hot deployment, supporting dynamic modification of task state, starting and stopping tasks at any time; registration service: the support node application starts the resource registration, so that the possibility of configuration errors is reduced; trigger pattern classification: providing a plurality of task triggering modes, and executing once and executing for a plurality of times; mail alarm: supporting mail alarm when the task fails, and supporting configuration of a multi-mail address group to send alarm mails; failover: if one machine in the executor cluster fails, the machine is automatically switched to a normal executor for task scheduling; task state monitoring: the task execution state and the execution history can be checked through the web page; on-line log: the scheduling log and the service log can be checked on line through the web page; consistency: the task scheduling center ensures the consistency of the distributed scheduling of the clusters through the DB lock, and one-time task scheduling can only trigger one-time execution; customizing task parameters: supporting online configuration scheduling task parameter entry, and managing multidimensional parameters through JSON objects; running a report: the real-time checking of operation data such as the number of tasks, the scheduling times, the number of executors and the like is supported; scheduling report forms, scheduling situation distribution diagrams and the like can be generated; asynchronous processing: the task scheduling flow is fully asynchronous and concurrent, so that flow peak clipping is effectively carried out on dense scheduling; user management: the online management platform user is supported, various role relations can be configured, and an administrator and an operation user have their own roles; and (3) authority control: and performing function authority control according to the role dimension, and performing data authority control according to the scheduler dimension.
In the embodiment of the application, an operator does not need to log in an application server to carry out manual modification in the process of updating task configuration, the functions of decoupling between task scheduling configuration and service codes, centralization of task scheduling data, convenience in maintenance and the like are realized only by utilizing a database to provide data dependence of task scheduling, multiple scheduling modes such as front-back end separation, front-end pages and the like are provided for selection, and control, management and optimization scheduling are carried out on task states and task scheduling according to different scheduling modes, so that the stability and accuracy of task scheduling are ensured, and the efficiency of task scheduling is also improved.
Referring to fig. 3, a flow chart of a data scheduling method disclosed in an embodiment of the present application is shown, where the data scheduling method mainly includes the following steps:
s301: acquiring task data of an application program; the task data characterizes data of a task-affiliated group and task execution parameters of the application program.
The task data comprises a task belonging group, a task execution strategy, task execution parameters, task alarm mail configuration, task execution interface class paths and the like.
S302: and configuring the task data into a preset table of the database according to the preset field form.
The preset field form is configured according to the specified field configuration, and may be an implementation type of the scheduling interface, etc., and the specific preset field form is not specifically limited in this application.
The preset table comprises a task log table, a task table and a task resource table.
The task data is configured into a preset table of the database according to the preset field form, and the process is shown as A1-A3.
A1: and configuring the service log in the task data into a task log table of a database through the implementation class of the scheduling interface and a preset incoming method.
The preset incoming method is a static method logcontext.appdbusinglog (String logMessage).
And checking the task running state and the running log through a task log table of the database.
In the implementation class of the scheduling interface, the static method is used for transmitting the information of the specific record log, so that the service log can be recorded in the task log table and corresponds to the specific execution record, and the running state can be conveniently checked later. The specific codes are as follows:
Copyright(c)2021,TRAVELSKY；
All·Rights Reserved；
TRAVELSKYCONFIDENTIAL；
@authorwangsining(wangsnsining@travelsky.com)；
an @ Description external task access interface;
@Title:ITaskJob；
@Pachagecomtravelsky.fare.dff.dfm.job.consumer.interfaces；
@date:2021/3/3013:23；
public interface ITaskJob{
an interface method is realized by the @ Description service;
paramaram service request parameters
@returnvoid；
@Authorwangsining(wangsining@travelsky.com)；
@Date 14:24·2021/3/22*/；
void doTaskJob(Map param)throws Exception；
}。
The contents of a specific task log table may be as shown with reference to table 1.
TABLE 1
After the implementation class of the task abstract interface is developed, the class path of the implementation class is required to be configured into the job_business_class in the database task_job (task table), and other data in the task table is required to be configured according to actual conditions.
A2: and configuring the class path of the implementation class of the dispatching task abstract interface in the task data into a task table of a database.
The contents of the specific task table may be as shown with reference to table 2.
TABLE 2
The data in the task resource list is automatically recorded after the task scheduling center system is started without paying attention to the data in the list by a user, and the data is used for monitoring the states of all nodes when the cluster is deployed.
The task state of the registered node and the states of the two nodes, namely offline and online, can be deduced through the node resource updating time field in the task resource table.
A3: task job resources (task_job_resources) in the task data are configured into a task resource table of a database.
The state of each node in the cluster deployment is monitored in real time through a task resource table in a database; the state includes an offline state or an online state.
The specific task resource table is shown in table 3.
TABLE 3 Table 3
Before the task program is started, a task is required to be completed, and a task is configured as a task profile, most of the profile has default configuration, but a few items are required to be noted, and the task is specifically as follows:
system.scan.class.patch=com.travel, specify that the scan task scheduler interface implements a class path.
The mail part needs to be configured according to the actual situation, and the configuration codes of the specific mail part are as follows:
# mail server SMTP address;
#mailHost＝***.com.cn；
mailHost＝***.com；
port number of # mail server
mailPort＝465；
Sending notification mail-sender mailbox account number;
mailUser＝***@***.com；
send notification mail-sender mailbox password, # send notification mail-sender mailbox password;
mailPassword＝？；
the # connection time limit, the unit of bright seconds, is about the length of time for establishing connection with the mail server, and defaults to unlimited;
connectiontimeout＝3000；
the sending time of the mail is limited, the unit millisecond is set, and the time of uploading the content when the mail is sent is limited by default;
writetimeout＝3000。
the configuration of the database is also required to be configured according to actual conditions as follows:
database driver #;
driver＝oracle.jdbc.driver.0racleDriver；
a # database connection string;
url＝
a # user name;
user＝
password #;
password＝。
org.quatz.schduler.instancename=dfp-DPRICV 3-NODE999; if the scheduling task is deployed in a cluster, then there must be a unique instanceId for each instance in the cluster, but there should be a same instanceName (default "Quartz scheduler").
The configuration of the cluster is default to a Quartz scheduler, if the cluster is not used, the configuration is not needed, if the cluster is needed, for example, three servers are used as the cluster, and the instanceName of the three servers is the same.
Other configurations can be correspondingly modified according to actual conditions, or default configurations can be adopted without modification.
And starting a task program, starting the content of the log, automatically scanning data configured in a database by a dispatching center, and loading the task into a dispatcher.
Examples of the contents of a specific start log are as follows:
2022-09-08-00 [ DFP-DPRICV3-NODE3 workbench-1 ] [ com.travelsky.fare.dff.dff.job.Consumer.System ScanJob ] [ INFO ] scheduler: the total number of tasks being performed in DFP-DPRICV3-NODE3 is 0;
2022-09-08-00 [ DFP-DPRICV3-N0DE3 workbench-1 ] [ com.travelsky.fare.dff.dff.job.con.consumer.SystemScanJob ] [ INFO ] scheduler the total number of tasks present in DFP-DPRICV3-NODE3 is 5
2022-09-08-00 [ DFP-DPRICV3-N0DE3 workbench-1 ] [ com.travelsky.fare.dff.dff.job.Consumer.System ScanJob ] [ INFO ] was scanned to a satisfactory task of 0, which was not processed. S303: and loading task data in the database into the task driving engine through preset scheduling task parameters, and executing the task data in the task driving engine through the preset scheduling parameters.
The set scheduling task parameters are the trigger time of the scheduling task, the trigger parameters of the scheduling task, the trigger node of the scheduling task, the configuration file of the scheduling program in the cluster, the cluster name and the like.
Specifically, task data in a database is loaded into a task driving engine through preset scheduling task parameters, and the process of executing the task data in the task driving engine through the preset scheduling parameters is shown as B1-B5.
B1: and deploying and starting the task program through the program implementation scheduling framework interface.
B2: after the task program is run, task data in the database is loaded into the task driving engine through preset scheduling task parameters.
The task parameters comprise a task main key, a task group, a task description, a task state, whether a task is deleted, a task strategy, a task remark, a task expression, a task parameter, a task execution class, a task mail, a task creation time, a task creator, a task modification time, a task modifier, a role ID, a scheduler name and the like.
It should be noted that, the task policy, the task expression, the task execution parameters, and the task execution class are key items for the correct execution of the task.
B3: the type of execution mode of the task data in the execution task driving engine is determined.
B4: if the execution mode of the task data in the execution task driving engine is a single-point execution mode, the task data in the task driving engine is executed through the triggering time of the dispatching task, the triggering parameter of the dispatching task and the triggering node of the dispatching task.
The present invention supports cluster configuration, so that some configuration files related to clusters need to be configured as required, if the execution mode of task data in the execution task driving engine is a single-point execution mode, the configuration related to the clusters can not be set, but the triggering time, the triggering parameters and the triggering nodes are configuration items which need to be configured.
B5: if the execution mode of the task data in the execution task driving engine is a multi-point execution mode, the task data in the task driving engine is executed through the triggering time of the dispatching task, the triggering parameter of the dispatching task, the triggering node of the dispatching task, the configuration file of the dispatching program in the cluster and the cluster name.
And monitoring the task log and the cluster execution node address of the task in real time through the task data.
In the embodiment of the application, an operator does not need to log in an application server to carry out manual modification in the process of updating task configuration, the functions of decoupling between task scheduling configuration and service codes, centralization of task scheduling data, convenience in maintenance and the like are realized only by utilizing a database to provide data dependence of task scheduling, multiple scheduling modes such as front-back end separation, front-end pages and the like are provided for selection, and control, management and optimization scheduling are carried out on task states and task scheduling according to different scheduling modes, so that the stability and accuracy of task scheduling are ensured, and the efficiency of task scheduling is also improved.
Based on the data scheduling method disclosed in fig. 3 in the foregoing embodiment, the embodiment of the present application further correspondingly discloses a data scheduling device, as shown in fig. 4, where the data scheduling device includes an obtaining unit 401, a configuring unit 402, and a loading executing unit 403.
An acquiring unit 401, configured to acquire task data of an application program; the task data characterizes data of a task-affiliated group and task execution parameters of the application program.
A configuration unit 402, configured to configure task data into a preset table of the database according to a preset field form.
The loading execution unit 403 is configured to load task data in the database into the task driving engine through preset scheduling task parameters, and execute the task data in the task driving engine through the preset scheduling parameters.
Further, the configuration unit 402 includes a first configuration module, a second configuration module, and a third configuration module.
The first configuration module is used for configuring the service log in the task data into a task log table of the database through the implementation class of the scheduling interface and a preset incoming method.
And the second configuration module is used for configuring the class path of the implementation class of the dispatching task abstract interface in the task data into the task table of the database.
And the third configuration module is used for configuring task job resources in the task data into a task resource table of the database.
Further, the load execution unit 403 for setting the parameters of the scheduled task is specifically configured to set the trigger time of the scheduled task, the trigger parameters of the scheduled task, the trigger node of the scheduled task, the configuration file of the scheduler in the cluster, and the cluster name.
Further, the load execution unit 403 includes a scheduling module, a loading module, a determining module, a first execution module, and a second execution module.
And the scheduling module is used for deploying the task program through a program implementation scheduling framework interface and starting the task program.
And the loading module is used for loading the task data in the database into the task driving engine through preset scheduling task parameters after the task program is operated.
And the judging module is used for judging the type of the execution mode of the task data in the execution task driving engine.
And the first execution module is used for executing the task data in the task driving engine through the triggering time of the scheduled task, the triggering parameter of the scheduled task and the triggering node of the scheduled task if the execution mode of the task data in the task driving engine is a single-point execution mode.
And the second execution module is used for executing the task data in the task driving engine through the triggering time of the dispatching task, the triggering parameter of the dispatching task, the triggering node of the dispatching task, the configuration file of the dispatching program in the cluster and the cluster name if the execution mode of the task data in the task driving engine is a multi-point execution mode.
Further, the data scheduling device further comprises a checking unit.
And the checking unit is used for checking the task running state and the running log through a task log table of the database.
Further, the data scheduling device further comprises a first monitoring unit.
The first monitoring unit is used for monitoring the states of all nodes in the cluster deployment in real time through a task resource table in the database; the state includes an offline state or an online state.
Further, the data scheduling device further comprises a second monitoring unit.
And the second monitoring unit is used for monitoring the task log and the cluster execution node address of the task in real time through the task data.
According to the scheme, an operator does not need to log in an application server to carry out manual modification in the process of updating task configuration, the functions of decoupling between task scheduling configuration and service codes, centralization of task scheduling data, convenience in maintenance and the like are realized only by utilizing a database to provide data dependence of task scheduling, multiple scheduling modes such as front-end and back-end separation, front-end pages and the like are provided for selection, and the task state and task scheduling are controlled, managed and optimized for different scheduling modes, so that the stability and accuracy of task scheduling are ensured, and the efficiency of task scheduling is also improved.
The embodiment of the application also provides a storage medium, which comprises stored instructions, wherein the equipment where the storage medium is controlled to execute the data scheduling method when the instructions run.
The embodiment of the present application further provides an electronic device, whose structural schematic diagram is shown in fig. 5, specifically including a memory 501, and one or more instructions 502, where the one or more instructions 502 are stored in the memory 501, and configured to be executed by the one or more processors 503 to perform the above-mentioned data scheduling method by executing the one or more instructions 502.
The specific implementation and derivative manner of each embodiment are all within the protection scope of the application.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for a system or system embodiment, since it is substantially similar to a method embodiment, the description is relatively simple, with reference to the description of the method embodiment being made in part. The systems and system embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative elements and steps are described above generally in terms of functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing is merely a preferred embodiment of the present application and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present application and are intended to be comprehended within the scope of the present application.
Claims (10)
1. A method of data scheduling, the method comprising:
acquiring task data of an application program; the task data represents data of a task belonging group and task execution parameters of an application program;
according to a preset field form, configuring the task data into a preset table of a database;
and loading the task data in the database into a task driving engine through preset scheduling task parameters, and executing the task data in the task driving engine through the preset scheduling parameters.
2. The method of claim 1, wherein the preset table includes a task log table, a task table, and a task resource table, and wherein the configuring the task data into the database according to the preset field form includes:
configuring a service log in task data into the task log table of a database through the implementation class of a scheduling interface and a preset incoming method;
configuring a class path of an implementation class of a scheduling task abstract interface in task data into the task table of a database;
and configuring task job resources in the task data into the task resource table of the database.
3. The method of claim 1, wherein the process of setting the scheduling task parameter comprises:
setting triggering time of a scheduling task, triggering parameters of the scheduling task, triggering nodes of the scheduling task, configuration files of a scheduling program in a cluster and cluster names.
4. A method according to claim 3, wherein loading the task data in the database into a task driving engine by a preset scheduling task parameter, and executing the task data in the task driving engine by the preset scheduling parameter, comprises:
a task program is deployed and started through a program realization scheduling framework interface;
after the task program is operated, task data in the database is loaded into a task driving engine through preset scheduling task parameters;
judging the type of an execution mode for executing the task data in the task driving engine;
if the execution mode of executing the task data in the task driving engine is a single-point execution mode, executing the task data in the task driving engine through the triggering time of the dispatching task, the triggering parameter of the dispatching task and the triggering node of the dispatching task;
and if the execution mode of executing the task data in the task driving engine is a multi-point execution mode, executing the task data in the task driving engine through the triggering time of the dispatching task, the triggering parameter of the dispatching task, the triggering node of the dispatching task, the configuration file of the dispatching program in the cluster and the cluster name.
5. The method of claim 1, further comprising, after executing the task data in the task driving engine by a preset scheduling parameter:
and checking the task running state and the running log through a task log table of the database.
6. The method as recited in claim 1, further comprising:
the state of each node in the cluster deployment is monitored in real time through a task resource table in a database; the state includes an offline state or an online state.
7. The method as recited in claim 1, further comprising:
and monitoring the task log and the cluster execution node address of the task in real time through the task data.
8. A data scheduling apparatus, the apparatus comprising:
the acquisition unit is used for acquiring task data of the application program; the task data represents data of a task belonging group and task execution parameters of an application program;
the configuration unit is used for configuring the task data into a preset table of a database according to a preset field form;
and the loading execution unit is used for loading the task data in the database into the task driving engine through preset scheduling task parameters and executing the task data in the task driving engine through the preset scheduling parameters.
9. The apparatus of claim 8, wherein the configuration unit comprises:
the first configuration module is used for configuring the business log in the task data into the task log table of the database through the implementation class of the scheduling interface and a preset incoming method;
the second configuration module is used for configuring a class path of an implementation class of the dispatching task abstract interface in the task data into the task table of the database;
and the third configuration module is used for configuring task job resources in the task data into the task resource table of the database.
10. The apparatus according to claim 8, wherein the load execution unit is configured to set a scheduling task parameter, and is specifically configured to:
setting triggering time of a scheduling task, triggering parameters of the scheduling task, triggering nodes of the scheduling task, configuration files of a scheduling program in a cluster and cluster names.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310034998.6A CN116302390A (en) | 2023-01-10 | 2023-01-10 | Data scheduling method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310034998.6A CN116302390A (en) | 2023-01-10 | 2023-01-10 | Data scheduling method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116302390A true CN116302390A (en) | 2023-06-23 |
Family
ID=86785869
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310034998.6A Pending CN116302390A (en) | 2023-01-10 | 2023-01-10 | Data scheduling method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116302390A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117472530A (en) * | 2023-10-25 | 2024-01-30 | 上海宽睿信息科技有限责任公司 | Centralized management-based data intelligent scheduling method and system |
-
2023
- 2023-01-10 CN CN202310034998.6A patent/CN116302390A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117472530A (en) * | 2023-10-25 | 2024-01-30 | 上海宽睿信息科技有限责任公司 | Centralized management-based data intelligent scheduling method and system |
| CN117472530B (en) * | 2023-10-25 | 2024-04-05 | 上海宽睿信息科技有限责任公司 | Centralized management-based data intelligent scheduling method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106844198B (en) | Distributed dispatching automation test platform and method | |
| US10348766B1 (en) | System and method for managing group policy backup | |
| CN106406993B (en) | A kind of timing task management method and system | |
| CN110941446B (en) | Version release method and device based on multi-environment offline task | |
| US9137106B2 (en) | Systems and methods for private cloud computing | |
| KR101891506B1 (en) | Methods and systems for portably deploying applications on one or more cloud systems | |
| US7383271B2 (en) | Centralized configuration data management for distributed clients | |
| CN101189595B (en) | Solution deployment in a server farm | |
| CN109582466A (en) | A kind of timed task executes method, distributed server cluster and electronic equipment | |
| CN103188088A (en) | Equipment information acquisition system and equipment information acquisition method | |
| US20090144743A1 (en) | Mailbox Configuration Mechanism | |
| US20190245745A1 (en) | Automated creation of test tenants for data center technical issue detection | |
| CN104615486B (en) | For searching for the multi-task scheduling of Extension Software Platform and executing methods, devices and systems | |
| CN116302390A (en) | Data scheduling method and device | |
| CN114035925A (en) | Workflow scheduling method, device and equipment and readable storage medium | |
| CN113434155A (en) | Automatic deployment system in mixed cloud mode | |
| US20090089772A1 (en) | Arrangement for scheduling jobs with rules and events | |
| CN112379989B (en) | Timed task process and queue service process management system and method | |
| CN117076096A (en) | Task flow execution method and device, computer readable medium and electronic equipment | |
| CN114787836A (en) | System and method for remotely executing one or more arbitrarily defined workflows | |
| CN114006815A (en) | Automatic deployment method and device for cloud platform nodes, nodes and storage medium | |
| CN115658791A (en) | Ground application management control system and method | |
| WO2009040171A2 (en) | Method, system and computer program for scheduling execution of jobs driven by events | |
| Agrawal | Kubernetes Deep Dive | |
| Anem et al. | A Primer on Regression Automation Frameworks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |