CN112015534A - Configurated platform scheduling method, system and storage medium - Google Patents
Configurated platform scheduling method, system and storage medium Download PDFInfo
- Publication number
- CN112015534A CN112015534A CN202010879341.6A CN202010879341A CN112015534A CN 112015534 A CN112015534 A CN 112015534A CN 202010879341 A CN202010879341 A CN 202010879341A CN 112015534 A CN112015534 A CN 112015534A
- Authority
- CN
- China
- Prior art keywords
- scheduling
- task
- annotation
- tasks
- platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/485—Task life-cycle, e.g. stopping, restarting, resuming execution
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to big data processing technology, and discloses a configurated platform scheduling method, which comprises the steps of annotating scheduling tasks of a configurated platform, storing the scheduling tasks and corresponding annotation information in a database sqlite in an associated manner, and setting a dependency relationship among the scheduling tasks according to the annotation information; carrying out micro-service scanning on the scheduling task with the annotation and registering; distributing fragments to the scheduling tasks, and determining the association relationship between the server and the scheduling tasks after the fragments are distributed; determining an execution server and an execution mode of the scheduling task according to the incidence relation and the dependency relation between the scheduling tasks; and executing the scheduling task according to the execution server and the execution mode of the scheduling task. The invention also relates to a block chain technology, and the data is stored in the block chain, so that the scheduling system of the configuration platform has great advantages in performance, stability, usability and expansibility.
Description
Technical Field
The present invention relates to big data processing, and in particular, to a method, a system, and a storage medium for scheduling a configured platform.
Background
The scheduling system is a core infrastructure in a big data platform, and the scheduling mode adopted by the current scheduling system is a mode of purely relying on time scheduling, such as crontab. However, the data processing flow has a long dependency chain, so that the problems of unclear dependency among tasks and difficult searching due to scheduling errors exist.
In the prior art, by adopting an elastic-jobmode, the problem of unclear dependence between tasks is improved, and the problem of difficulty in searching errors is solved; but still have the following disadvantages:
1) insufficient support for dependencies between tasks;
2) graphical management interface is not fully functional.
Therefore, a configurable platform scheduling method with rich scheduling function is needed.
Disclosure of Invention
The invention provides a method and a system for scheduling a configurated platform and a computer readable storage medium, which mainly solve the problem of insufficient dependence support among scheduling tasks of the configurated platform.
In order to achieve the above object, the present invention provides a method for scheduling a configured platform, which is applied to an electronic device, and the method includes: annotating scheduling tasks of a configured platform, storing the scheduling tasks and corresponding annotation information in a lightweight database sqlite in an associated manner, and setting a dependency relationship between the scheduling tasks according to the annotation information; performing microservice scanning on the scheduling task with the annotation, and registering in a registration center of the configuration platform according to the type of the annotation of the scheduling task; distributing fragments to the scheduling tasks according to an elastic-jobframe according to the number of the servers in the registration center, and determining the incidence relation between the servers and the scheduling tasks after the fragments are distributed; determining an execution server and an execution mode of the scheduling task according to the incidence relation and the dependency relation between the scheduling tasks; and executing the scheduling task according to the execution server and the execution mode of the scheduling task.
Further, preferably, the annotation of the scheduling task includes task node classification data, task state attribute data and task version attribute data; wherein the task state attributes are divided into operation, to-be-operated, pause, completion or exception; the task node classification comprises a parent node task and a child node task.
Further, preferably, the step of storing the scheduling task and the corresponding annotation information in association with the lightweight database sqlite further includes: and adding a plurality of state data embedding points on the original data embedding points by using a database monitor of the configuration platform, and synchronizing all the data embedding points to the database sqlite. .
Further, preferably, the scheduling task and the corresponding annotation information are stored in a lightweight database sqlite in an associated manner by the following method, a corresponding log file is dynamically added to the scheduling task according to the task, and the scheduling task is hierarchically stored according to the log file in four states of error, war, info and debug; wherein the log files are rolled over in two dimensions of time and size.
Further, preferably, the data of the scheduling task is stored in a block chain; and synchronizing the scheduling task data stored in each lightweight database sqlite into the database of the configuration platform to realize task state query of the scheduling task.
In order to achieve the above object, the present invention further provides a configurated platform scheduling system, which includes a scheduling task annotation unit, a scheduling task registration unit, a scheduling task association unit and a scheduling task execution unit; the scheduling task annotation unit is used for annotating the scheduling tasks of the configured platform, storing the scheduling tasks and corresponding annotation information in a lightweight database sqlite in an associated manner, and setting the dependency relationship among the scheduling tasks according to the annotation information; the scheduling task registration unit is used for performing microservice scanning on the scheduling task with the annotation and registering in a registration center of a configuration platform according to the type of the annotation of the scheduling task; the scheduling task association unit is used for distributing fragments to the scheduling tasks according to an elastic-jobframe according to the number of the servers in the registration center and determining the association relationship between the servers and the scheduling tasks after the fragments are distributed; the scheduling task execution unit is used for determining an execution server and an execution mode of the scheduling task according to the incidence relation and the dependency relation among the scheduling tasks; and executing the scheduling task according to the execution server and the execution mode of the scheduling task.
Further, preferably, the data of the scheduling task is stored in a block chain; the scheduling task annotation unit comprises a scheduling task annotation module and a scheduling task storage module with annotation; the scheduling task annotation module is used for annotating the scheduling task of the configuration platform; the annotation of the scheduling task comprises task node classification, task state attribute and task version attribute; wherein the task state attributes are divided into operation, to-be-operated, pause, completion or exception; the task node classification comprises a father node task and a child node task; and the scheduling task storage module with annotation is used for storing the scheduling task and corresponding annotation information in the lightweight database sqlite and setting the dependency relationship between the scheduling tasks according to the annotation information.
Further, preferably, the scheduling task storage module with the annotation comprises a storage submodule, a dependency relationship setting submodule and a data buried point synchronization submodule; the storage submodule is used for storing the scheduling task and corresponding annotation information in a lightweight database sqlite; the dependency relationship setting submodule is used for setting the dependency relationship among the scheduling tasks according to the annotation information; and the data embedding point synchronization submodule is used for increasing data embedding points in a plurality of states on the original data embedding points by utilizing the database monitor of the configuration platform and synchronizing all the data embedding points to the database sqlite.
To achieve the above object, the present invention also provides an electronic device, including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores a program executable by the at least one processor to enable the at least one processor to perform the configured platform scheduling method as described above.
In addition, to achieve the above object, the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to implement the steps of the above configured platform scheduling method when executed by a processor.
According to the configuration platform scheduling method, the configuration platform scheduling system, the electronic device and the computer readable storage medium, the scheduling task of the configuration platform is annotated, the scheduling task and corresponding annotation information are stored in the lightweight database sqlite in an associated mode, and the dependency relationship among the scheduling tasks is set according to the annotation information; performing micro-service scanning on the scheduling task with the annotation, and registering in a registration center of a configuration platform according to the type of the annotation of the scheduling task; distributing fragments to the scheduling tasks according to an elastic-jobframe according to the number of the servers in the registration center, and determining the incidence relation between the servers and the scheduling tasks after the fragments are distributed; judging an execution server and an execution mode of the scheduling task according to the incidence relation and the dependency relation between the scheduling tasks; and executing the scheduling task according to the execution server and the execution mode of the scheduling task. The functions of the configuration platform are enriched; the beneficial effects are as follows:
1) the management interface provides the tasks of starting, suspending, re-running and terminating the existing tasks, and provides a new task for dynamic addition, so that the usability is greatly improved;
2) the encapsulated framework completes the establishment of tasks in an annotation mode, only data per se needs to be concerned, and the extensibility of scheduling tasks is greatly facilitated;
3) by combining the advantages of the distributed data scheduling of the elastic-joba, the scheduling system of the configuration platform has great advantages in performance, stability, usability and expansibility.
Drawings
FIG. 1 is a flow chart of a preferred embodiment of a method for deploying a deployed platform according to the present invention;
FIG. 2 is a block diagram of a configurable platform scheduling system according to a preferred embodiment of the present invention;
FIG. 3 is a schematic structural diagram of an electronic device according to a preferred embodiment of the invention;
the implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In order to improve the coding efficiency of users, the invention provides a configurated platform scheduling method. FIG. 1 shows a flow of a preferred embodiment of a method for scheduling a configured platform according to the present invention. Referring to fig. 1, the method may be performed by an apparatus, which may be implemented by software and/or hardware.
It should be noted that, in the present invention, a configured platform scheduling method, specifically, the configured platform scheduling method includes steps S110 to S150.
S110, annotating the scheduling tasks of the configuration platform, storing the scheduling tasks and corresponding annotation information in a lightweight database sqlite in an associated manner, and setting the dependency relationship among the scheduling tasks according to the annotation information.
In a specific embodiment, the annotations of the scheduling task include task node classification data, task state attribute data, and task version attribute data; wherein the task state attributes are divided into operation, to-be-operated, pause, completion or exception; the task node classification comprises a parent node task and a child node task.
That is, adding annotations to each scheduling task, wherein the annotations comprise node classification, task state and version number; the task node classification is to add a parent node task annotation and a child node task annotation to increase the association (dependency) relationship between tasks. The task state attribute data and the task version attribute data are historical states and versions of scheduling tasks stored through a lightweight database sqlite, and task state information and version information are initialized; in the specific implementation process, Sqlite adopts a file mode to create a new table for storing version information.
It should be noted that, when the task server is online, the server information will be automatically registered in the registration center, and when the task server is offline, the server status will be automatically updated. That is to say, two attributes of task state and task version are added to the task, and a foundation is provided for adding task dependent functions to the distributed task.
It should be noted that the initial version number is 0, and the version number is incremented by 1 after the execution of one time. The states of the tasks are divided into operation, waiting for operation, suspension, completion or exception.
The specific implementation scenario is as follows: including RUNNING (1, "RUNNING"), READY (2, "to run"), STOP (3, "to run"), OVER (4, "to run"), ERROR (5, "abnormal"); the state of the task during initial registration is READY, the task during execution is RUNNING, the execution is OVER, the state and the version number of each task are registered into a zookeeper during the state change, the initial version number is 0, and 1 is added after the execution.
In a specific implementation process, the step of storing the scheduling task and the corresponding annotation information in the lightweight database sqlite in an associated manner further comprises the steps of adding a plurality of states of data burial points on an original data burial point by using a database listener of the configuration platform, and synchronizing all the data burial points to the database sqlite.
It should be noted that the embedded point is a method for collecting data of an application program such as a website, an App, or a background. Through the embedded points, the generation behaviors of the user in the application can be collected, and then the generated behaviors are used for analyzing and optimizing subsequent experiences of the product, and data support can be provided for the operation of the product. The buried point corresponds to a task listener, and for Elastic-jobs, the listeners are divided into local listeners and distributed listeners. The local monitor only schedules when the node executes the own fragmentation, and the local monitor executes each fragmentation task when the local monitor schedules. The local listener is defined by the elastic joblistener interface. The local listener will execute when the job executes the local slicing task, and if the job is sliced into 6 slices, the listener task will execute 6 times. The distributed listener will execute once when the overall task begins execution and once when the overall task ends execution. The distributed listener is also implemented on the basis of a normal listener.
For each task, the original data burial point is based on the original database listener of the elastic-joba. Two attributes of a task state and a task version are added, so that a listener and a corresponding embedded point of the listener need to be added, and the corresponding control of each state of the task by the listener is further realized.
The specific implementation process is to rewrite the relevant classes and add the process state data to the original data embedding points. The state change not only includes the original starting and ending states, but also changes into RUNNING (1, "RUNNING"), READY (2, "to run"), STOP (3, "pause"), OVER (4, "done"), ERROR (5, "abnormal"), and 5 states.
In a specific embodiment, a scheduling task and corresponding annotation information are stored in a sqlite database in an associated mode through the following method, a corresponding log file is dynamically added to the scheduling task according to the task, and the scheduling task is stored in a grading mode according to the log file in four states of error, war, info and debug; wherein the log files are rolled over in two dimensions of time and size.
That is, in the implementation process, if there are A, B two tasks, a registers the state and version of each node in the zookeeper during the running process. The task B stores the version information of the task A in an execution plan into a sqlite database, before the task B is executed, the version information of the task A in the zookeeper and the version information of the task B stored in the sqlite database are compared, the version information of the task A in the zookeeper is consistent with the version information of the task B stored in the sqlite database, the task B can be normally executed when the state information of the task A in the zookeeper is displayed as OVER, and if not, the task B is suspended.
S120, performing micro-service scanning on the scheduling task with the annotation, and registering in a registration center of the configuration platform according to the type of the annotation of the scheduling task.
Registering as described herein is the registering of tasks with special annotations.
Specifically, the microservice scans all classes with special annotations and completes the registration. During registration, the annotated attribute task name, the number of fragments, the fragment parameters, the execution plan and the dependency relationship are initialized. And selecting and filling the attribute tasks according to actual needs.
The development difficulty of developers is reduced, so that the developers only need to pay attention to the business and do not need to pay attention to the framework.
The implementation scenario is exemplified as follows:
when a new scheduling task is created, a developer adds the @ Job annotation and initializes the relevant attributes of the annotation. @ Job (jobName ═ testJob ", cron ═ 0/10 ═
S130, distributing and slicing the scheduling tasks according to an elastic-jobframe according to the number of the servers in the registration center, and determining the association relationship between the servers and the scheduling tasks after the slicing is distributed.
The task slicing mode is adopted for realization. One task is divided into n independent task items, and the distributed servers execute the respectively distributed sharding items in parallel.
In other words, according to the number of servers in the registration space, the fragmentation condition of the task is determined, and the task is run in parallel or in series according to the plan. That is, the task allocation fragment is determined according to the number of instances in namespace. This strategy relies on an elastic-jobframe.
It should be noted that, the master node elects, the server goes on and off the line, and the total number of the fragments is changed to update the re-fragmentation mark; when the timing task is triggered, if re-fragmentation is needed, the task can be executed only after the fragmentation is finished through the main server fragmentation and blocking in the fragmentation process. If the main server is off-line in the slicing process, the main server is elected first, and then slicing is carried out. In order to maintain the stability of operation of the job, the slicing state is only marked in the operation process, and the operation is not sliced again. Fragmentation may only occur before the next task trigger. And each time of fragmentation is sequenced according to the IP of the server, so that the fragmentation result is ensured not to generate large fluctuation. And a failover function is realized, unallocated fragments are actively captured after execution of a certain server is finished, and available servers are actively searched for executing tasks after a certain server is offline.
And S140, judging an execution server and an execution mode of the scheduling task according to the incidence relation and the dependency relation among the scheduling tasks.
Specifically, in step S130, a task is split into n independent task items, and it is determined which server each independent task item is specifically executed by according to the association relationship between the server and the scheduling task after the distribution of the shards, so that the execution server of the scheduling task after the distribution of the shards can be determined by the association relationship between the server and the scheduling task after the distribution of the shards.
In addition, the dependency relationship between the scheduling tasks, that is, the dependency relationship between the scheduling tasks is determined according to the task node classification (the parent node task annotation and the child node task annotation), that is, the dependency relationship between the scheduling tasks is clear, and the association relationship between the scheduling tasks is also determined. Thus, the execution server and the execution manner of the scheduled task are confirmed.
And S150, executing the scheduling task according to the execution server and the execution mode of the scheduling task.
In a specific embodiment, the data of the scheduling task is stored in a blockchain; and synchronizing the scheduling task data stored in each lightweight database sqlite into the database of the configuration platform, so as to realize task state query of the scheduling task.
Specifically, a log roll function is added through logback; it should be noted that the log roll is a log update. Wherein, logback-core: the base module of the other two modules; logback-class: it is an improved version of log4j, while it fully implements the slf4j API to allow you to easily change to other Logging systems such as log4j or JDK14 Logging; logback-access: the access module is integrated with Servlet containers to provide the function of accessing the log through Http. By adopting logback to increase the log rolling function, the performance is stable, and the running speed is faster.
The original task comprises an original starting state and an original ending state, two attributes of the task state and the task version are added, and the original task is changed into RUNNING (1, "RUNNING"), READY (2, "to be run"), STOP (3, "pause"), OVER (4, "complete"), ERROR (5, "abnormal"), and 5 states; the update pattern of the log also needs to be changed.
Specifically, corresponding log files are dynamically added to the tasks according to the tasks, the logs are rolled according to two dimensions of time and size, and the logs of each task are stored in a grading mode according to error, war, info and debug.
And according to the dependency relationship, registering and storing the task state and the version to a sqlite database, synchronizing to sqlite of other servers, and synchronizing the state and the version to the database. The client creates the node through the client, sets the content, and other servers acquire the state information and the version information in the node.
Tasks can be added, deleted, suspended and rerun in the whole process in an interface mode. Each operation of the interface triggers the above process (register and store task state and version to sqlite database, synchronize to sqlite of other servers, and synchronize state and version to database according to dependency relationship.
In a particular embodiment, the system already has A, B two tasks. The task C is not provided with the @ Job annotation when the system is started, and can be activated at a specific time, parameters can be transmitted to the server side through the page at the moment, the task C is found according to the parameters, other parameters are taken, the task C is activated, and the task C is executed according to the execution plan in the parameters. When the task C is activated, the task is registered according to the parameters, the dependency relationship is established, and the state and the version are managed in the running process.
In conclusion, zookeeper is used as a registration center, and the state of the node is synchronized to the registration center; the method and the system realize that the child node can monitor the task state of the related father node through the monitor and judge whether the current child node carries out data processing or not through the task state; therefore, the dependence of tasks is realized, and the monitoring of a graphical interface is also realized.
The operation of the zookeeper is completed by connecting the zookeeper with the pool. The method comprises the steps of adding nodes, writing state data into the nodes, inquiring the state number of a certain node and the like. And the executor synchronously refreshes the information in the zookeeper to the database and provides a query for historical data. A new management platform is developed in a http request restful mode, and the existing tasks of starting, pausing, rerunning and terminating are supported.
FIG. 2 is a block diagram of a preferred embodiment of a deployed platform scheduling system in accordance with the present invention; as shown with reference to figure 2 of the drawings,
the configuration platform scheduling system 200 includes a scheduling task annotating unit 210, a scheduling task registering unit 220, a scheduling task associating unit 230, and a scheduling task executing unit 240.
And the scheduling task annotation unit 210 is configured to annotate the scheduling tasks of the configured platform, store the scheduling tasks and corresponding annotation information in association with the lightweight database sqlite, and set a dependency relationship between the scheduling tasks according to the annotation information. And the scheduling task registering unit 220 is configured to perform microservice scanning on the scheduling task with the annotation, and register in a registration center of the configuration platform according to the type of the annotation of the scheduling task. And the scheduling task association unit 230 is configured to distribute the scheduling tasks according to an elastic-jobframe according to the number of servers in the registry, and determine an association relationship between the servers and the scheduling tasks after the distribution of the fragments. The scheduling task execution unit 240 is configured to determine an execution server and an execution mode of the scheduling task according to the association relationship and the dependency relationship between the scheduling tasks; and executing the scheduling task according to the execution server and the execution mode of the scheduling task.
The data of the scheduling task is stored in a block chain; the scheduling task annotation unit 210 comprises a scheduling task annotation module 211 and a scheduling task storage module 212 with annotation; the scheduling task annotation module 211 is configured to annotate a scheduling task of the configured platform; the annotation of the scheduling task comprises task node classification, task state attribute and task version attribute; wherein the task state attributes are divided into operation, to-be-operated, pause, completion or exception; the task node classification comprises a father node task and a child node task; and the scheduling task storage module 212 with annotation is used for storing the scheduling task and corresponding annotation information in a lightweight database sqlite in an associated manner, and setting a dependency relationship between the scheduling tasks according to the annotation information.
In a specific embodiment, it is emphasized that the annotated scheduling task storage module 211 includes a storage sub-module 2111, a dependency setting sub-module 2112, and a data sink synchronization sub-module 2113.
The storage submodule 2111 is used for associating and storing the scheduling task and the corresponding annotation information in a lightweight database sqlite; the dependency relationship setting submodule 2112 is configured to set a dependency relationship between scheduling tasks according to the annotation information; and a data embedding point synchronization sub-module 2113, configured to add data embedding points in multiple states to the original data embedding points by using the database listener of the configured platform, and synchronize all the data embedding points to the database sqlite.
Taking an application scene of the vehicle-mounted positioning system as an example, the configuration platform scheduling method based on the image page is utilized to build a server environment. The vehicle-mounted positioning software of each vehicle starts a timing task, positioning information is uploaded to the server every 5s, and a server management interface can operate the timing task of each vehicle; the system server side starts a timing task, and analyzes the uploaded position information of all vehicles every 30 s:
that is, the scheduling task is specifically a positioning task. Firstly annotating a positioning task, storing the positioning task and corresponding annotation information in a lightweight database sqlite in an associated manner, and setting a dependency relationship between the positioning task and a historical task of a main server according to the annotation information; the historical task in the main server is used as a father node task, and the positioning task is used as a child node task.
Performing micro-service scanning on the scheduling task with the annotation, and registering in a registration center of the server according to the type of the annotation of the scheduling task; determining the incidence relation between each server and the positioning task according to the number of the servers in the registration center, namely selecting the server for executing the positioning task; determining an execution server of the positioning task and a historical task of a related main server thereof according to the incidence relation and the dependency relation among the scheduling tasks, and determining an execution mode of the positioning task if the execution server and a father node task thereof are determined; and executing the positioning task according to the execution server and the execution mode of the positioning task, and then making corresponding early warning by the vehicle-mounted positioning system according to the positioning task and a specified rule.
The invention provides a configurated platform scheduling method, which is applied to an electronic device 3.
FIG. 3 illustrates an application environment of a preferred embodiment of the method for deploying platforms in accordance with the invention.
Referring to fig. 3, in the present embodiment, the electronic device 4 may be a terminal device having an arithmetic function, such as a server, a smart phone, a tablet computer, a portable computer, or a desktop computer.
The electronic device 3 includes: a processor 32, a memory 31, a communication bus 33, and a network interface 35.
The memory 31 includes at least one type of readable storage medium. The at least one type of readable storage medium may be a non-volatile storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory 31, and the like. In some embodiments, the readable storage medium may be an internal storage unit of the electronic apparatus 3, such as a hard disk of the electronic apparatus 3. In other embodiments, the readable storage medium may also be an external memory 31 of the electronic device 3, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, provided on the electronic device 3.
In the present embodiment, the readable storage medium of the memory 31 is generally used for storing the configuration platform scheduler 30 and the like installed in the electronic device 3. The memory 31 may also be used to temporarily store data that has been output or is to be output.
A communication bus 33 is used to enable connection communication between these components.
The network interface 34 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface), and is typically used to establish a communication link between the electronic apparatus 3 and other electronic devices.
Fig. 3 only shows the electronic device 3 with components 31-34, but it is to be understood that not all of the shown components are required to be implemented, and that more or fewer components may alternatively be implemented.
Optionally, the electronic device 3 may further include a user interface, which may include an input unit such as a Keyboard (Keyboard), a voice input device such as a microphone (microphone) or other equipment with voice recognition function, a voice output device such as a sound box, a headset, etc., and optionally may also include a standard wired interface, a wireless interface.
Optionally, the electronic device 3 may further comprise a display, which may also be referred to as a display screen or a display unit. In some embodiments, the display device may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an Organic Light-Emitting Diode (OLED) touch device, or the like. The display is used for displaying information processed in the electronic apparatus 3 and for displaying a visualized user interface.
Optionally, the electronic device 3 may further include a Radio Frequency (RF) circuit, a sensor, an audio circuit, and the like, which are not described herein again.
In the apparatus embodiment shown in FIG. 3, a memory 31, which is a type of computer storage medium, may include an operating system, and a configured platform scheduler 30; the processor 32, when executing the configurational platform scheduler 30 stored in the memory 31, implements the following steps: annotating scheduling tasks of a configured platform, storing the scheduling tasks and corresponding annotation information in a lightweight database sqlite in an associated manner, and setting a dependency relationship between the scheduling tasks according to the annotation information; performing microservice scanning on the scheduling task with the annotation, and registering in a registration center of the configuration platform according to the type of the annotation of the scheduling task; distributing fragments to the scheduling tasks according to an elastic-jobframe according to the number of the servers in the registration center, and determining the incidence relation between the servers and the scheduling tasks after the fragments are distributed; determining an execution server and an execution mode of the scheduling task according to the incidence relation and the dependency relation between the scheduling tasks; and executing the scheduling task according to the execution server and the execution mode of the scheduling task. .
In other embodiments, the configured platform scheduler 30 may also be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to accomplish the present invention. The modules referred to herein are a series of computer program segments that perform particular functions. The configuration platform scheduler 30 may be divided into a scheduling task annotating unit 210, a scheduling task registering unit 220, a scheduling task associating unit 230, and a scheduling task executing unit 240.
In addition, the present invention also provides a computer-readable storage medium, which mainly includes a storage data area and a storage program area, wherein the storage data area can store data created according to the use of the block chain node, and the storage program area can store an operating system and an application program required by at least one function, the computer-readable storage medium includes a configured platform scheduler, and the configured platform scheduler implements the operation of the configured platform scheduling method when being executed by a processor.
The embodiments of the computer-readable storage medium of the present invention are substantially the same as the embodiments of the above-mentioned method, system and electronic device for scheduling a configured platform, and are not repeated herein.
In summary, the method, the system, the electronic device and the computer readable storage medium for scheduling a configured platform annotate scheduling tasks of the configured platform, store the scheduling tasks with annotations in a lightweight database sql lite, and set the dependency relationship between the scheduling tasks; performing micro-service scanning on the scheduling task with the annotation, and registering in a registration center of a configuration platform according to the type of the annotation of the scheduling task; distributing fragments to the scheduling tasks according to an elastic-jobframe according to the number of the servers in the registration center, and determining the incidence relation between the servers and the scheduling tasks after the fragments are distributed; after judging the execution server and the execution mode of the scheduling task according to the incidence relation between the server and the scheduling task after the distribution fragmentation and the dependency relation between the scheduling tasks, executing the scheduling task, and enriching the functions of a configuration platform; the management interface provides the functions of starting, suspending, re-running and terminating the existing tasks, and provides the function of dynamically adding new tasks, so that the usability is greatly improved; the packaged framework completes the establishment of tasks in an annotation mode, only data per se needs to be concerned, and the expansibility of scheduling tasks is greatly facilitated; by combining the advantages of the distributed data scheduling of the elastic-joba, the scheduling system of the configuration platform has great advantages in performance, stability, usability and expansibility.
The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, apparatus, article, or method that includes the element.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments. Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) as described above, and includes several programs for enabling a terminal device (e.g., a mobile phone, a computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
The above description is only a preferred embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (10)
1. A method for scheduling a configurated platform is applied to an electronic device, and is characterized by comprising the following steps:
annotating scheduling tasks of a configured platform, storing the scheduling tasks and corresponding annotation information in a lightweight database sqlite in an associated manner, and setting a dependency relationship between the scheduling tasks according to the annotation information;
performing microservice scanning on the scheduling task with the annotation, and registering in a registration center of the configuration platform according to the type of the annotation of the scheduling task;
distributing fragments to the scheduling tasks according to an elastic-jobframe according to the number of the servers in the registration center, and determining the incidence relation between the servers and the scheduling tasks after the fragments are distributed;
determining an execution server and an execution mode of the scheduling task according to the incidence relation and the dependency relation between the scheduling tasks;
and executing the scheduling task according to the execution server and the execution mode of the scheduling task.
2. The method of claim 1, wherein the annotations for the scheduled tasks comprise task node classification data, task state attribute data, and task version attribute data; wherein the task state attributes are divided into operation, to-be-operated, pause, completion or exception; the task node classification comprises a parent node task and a child node task.
3. The method of claim 2, wherein the step of storing the scheduled tasks and corresponding annotation information associations in a lightweight database, sqlite, further comprises:
and adding a plurality of state data embedding points on the original data embedding points by using a database monitor of the configuration platform, and synchronizing all the data embedding points to the database sqlite.
4. The configuration platform scheduling method according to claim 2, wherein the scheduling task and the corresponding annotation information are stored in a lightweight database sqlite in an associated manner, a corresponding log file is dynamically added to the scheduling task according to the task, and the scheduling task is hierarchically stored according to the log file in four states of error, war, info and debug; wherein the log files are rolled over in two dimensions of time and size.
5. The configurable platform scheduling method according to claim 3, wherein the data of the scheduling task is stored in a block chain; and synchronizing the scheduling task data stored in each lightweight database sqlite into the database of the configuration platform to realize task state query of the scheduling task.
6. A configurated platform scheduling system is characterized by comprising a scheduling task annotation unit, a scheduling task registration unit, a scheduling task association unit and a scheduling task execution unit; wherein the content of the first and second substances,
the scheduling task annotation unit is used for annotating the scheduling tasks of the configured platform, storing the scheduling tasks and corresponding annotation information in a lightweight database sqlite in an associated manner, and setting the dependency relationship among the scheduling tasks according to the annotation information;
the scheduling task registration unit is used for performing microservice scanning on the scheduling task with the annotation and registering in a registration center of a configuration platform according to the type of the annotation of the scheduling task;
the scheduling task association unit is used for distributing fragments to the scheduling tasks according to an elastic-jobframe according to the number of the servers in the registration center and determining the association relationship between the servers and the scheduling tasks after the fragments are distributed;
the scheduling task execution unit is used for determining an execution server and an execution mode of the scheduling task according to the incidence relation and the dependency relation among the scheduling tasks; and executing the scheduling task according to the execution server and the execution mode of the scheduling task.
7. The configured platform scheduling system of claim 6 wherein data for the scheduling task is stored in a blockchain;
the scheduling task annotation unit comprises a scheduling task annotation module and a scheduling task storage module with annotation;
the scheduling task annotation module is used for annotating the scheduling task of the configuration platform; the annotation of the scheduling task comprises task node classification, task state attribute and task version attribute; wherein the task state attributes are divided into operation, to-be-operated, pause, completion or exception; the task node classification comprises a father node task and a child node task;
and the scheduling task storage module with annotation is used for storing the scheduling task and corresponding annotation information in the lightweight database sqlite and setting the dependency relationship between the scheduling tasks according to the annotation information.
8. The configuration platform scheduling system according to claim 6, wherein the scheduling task storage module with annotation comprises a storage sub-module, a dependency relationship setting sub-module and a data buried point synchronization sub-module;
the storage submodule is used for storing the scheduling task and corresponding annotation information in a lightweight database sqlite;
the dependency relationship setting submodule is used for setting the dependency relationship among the scheduling tasks according to the annotation information;
and the data embedding point synchronization submodule is used for increasing data embedding points in a plurality of states on the original data embedding points by utilizing the database monitor of the configuration platform and synchronizing all the data embedding points to the database sqlite.
9. An electronic device, comprising: at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores a program executable by the at least one processor to enable the at least one processor to perform the configured platform scheduling method of any of claims 1 to 5.
10. A computer-readable storage medium, storing a computer program, wherein the computer program, when executed by a processor, implements the configured platform scheduling method of any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010879341.6A CN112015534A (en) | 2020-08-27 | 2020-08-27 | Configurated platform scheduling method, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010879341.6A CN112015534A (en) | 2020-08-27 | 2020-08-27 | Configurated platform scheduling method, system and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112015534A true CN112015534A (en) | 2020-12-01 |
Family
ID=73503764
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010879341.6A Pending CN112015534A (en) | 2020-08-27 | 2020-08-27 | Configurated platform scheduling method, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112015534A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114968504A (en) * | 2021-02-26 | 2022-08-30 | 中国联合网络通信集团有限公司 | Distributed task scheduling method and device and storage medium |
-
2020
- 2020-08-27 CN CN202010879341.6A patent/CN112015534A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114968504A (en) * | 2021-02-26 | 2022-08-30 | 中国联合网络通信集团有限公司 | Distributed task scheduling method and device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7271734B2 (en) | Data serialization in distributed event processing systems | |
| CN110832453B (en) | Distributed version control of applications using cloud-based systems | |
| CN108874558B (en) | Message subscription method of distributed transaction, electronic device and readable storage medium | |
| CN111143382B (en) | Data processing method, system and computer readable storage medium | |
| CN108513657B (en) | Data conversion method and backup server | |
| US20100313063A1 (en) | Mitigating reduction in availability level during maintenance of nodes in a cluster | |
| CN111324610A (en) | Data synchronization method and device | |
| CN110247984B (en) | Service processing method, device and storage medium | |
| CN108334514B (en) | Data indexing method and device | |
| CN114385759B (en) | Configuration file synchronization method and device, computer equipment and storage medium | |
| CN109032796B (en) | Data processing method and device | |
| EP2959387B1 (en) | Method and system for providing high availability for state-aware applications | |
| CN110532074A (en) | A kind of method for scheduling task and system of multi-tenant Mode S aaS service cluster environment | |
| CN111324606B (en) | Data slicing method and device | |
| US9886321B2 (en) | Managing distributed analytics on device groups | |
| WO2020224098A1 (en) | Cloud access method and apparatus based on global server load balance, and storage medium | |
| CN111338834B (en) | Data storage method and device | |
| CN111538573A (en) | Asynchronous task processing method and device and computer readable storage medium | |
| CN113760513A (en) | Distributed task scheduling method, device, equipment and medium | |
| CN111784318A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN109739539B (en) | Cross-environment application release method, device, equipment and storage medium | |
| WO2017157111A1 (en) | Method, device and system for preventing memory data loss | |
| CN112015534A (en) | Configurated platform scheduling method, system and storage medium | |
| CN113468143A (en) | Data migration method, system, computing device and storage medium | |
| CN116049142A (en) | Data processing method, device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |