CN110597603A - Scheduling method and system of distributed scheduling tasks - Google Patents
Scheduling method and system of distributed scheduling tasks Download PDFInfo
- Publication number
- CN110597603A CN110597603A CN201910477801.XA CN201910477801A CN110597603A CN 110597603 A CN110597603 A CN 110597603A CN 201910477801 A CN201910477801 A CN 201910477801A CN 110597603 A CN110597603 A CN 110597603A
- Authority
- CN
- China
- Prior art keywords
- scheduling
- task
- database
- log
- trigger
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 218
- 230000002159 abnormal effect Effects 0.000 claims description 15
- 238000004886 process control Methods 0.000 claims description 2
- 238000007726 management method Methods 0.000 description 63
- 238000004458 analytical method Methods 0.000 description 7
- 238000012800 visualization Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 238000010223 real-time analysis Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
Abstract
The application aims to provide a scheduling method and a scheduling system for distributed scheduling tasks. Compared with the prior art, in the method, the scheduling of the scheduling tasks is realized in a mode of combining the database, the scheduling trigger and the progress manager, and the problems that the scheduling tasks on an operating system level can only be executed by a single machine, the computing capacity is limited and single-point risks exist are solved.
Description
Technical Field
The application relates to the technical field of computers, in particular to scheduling technology of scheduling tasks.
Background
The existing scheduling scheme of scheduling tasks is based on an operating system, but scheduling tasks at the operating system level have many defects: the accuracy can only be achieved to minutes, and the task with smaller time accuracy cannot be achieved; scheduling tasks at the operating system level are loaded into the system by way of files, but the way in which files are stored is difficult to manage. Scheduling tasks at the operating system level can only be executed by a single machine, are limited by the computing power of the single machine, and have single-point risks. When the scheduling task at the operating system level meets the separate storage type service of the Internet, the centralized management of the scheduling task of multiple computers cannot be carried out. And the scheduling task of the operating system level only has a simple log, so that problems are not easy to find, and an alarm cannot be given. And fifthly, scheduling tasks at the operating system level can be managed only under a command line, visualization cannot be performed, and management is inconvenient.
Disclosure of Invention
In order to solve the above problems, an object of the present application is to provide a scheduling method and system for distributed scheduling tasks.
According to an aspect of the present application, there is provided a scheduling system for distributed scheduling tasks, the system comprising:
the database is used for acquiring and storing scheduling task data;
the scheduling trigger is used for reading scheduling task data in the database and generating an instant execution task corresponding to the scheduling task;
the process manager is used for starting or stopping the process so as to control the process to execute or finish executing the instant execution task corresponding to the scheduling task generated by the scheduling trigger;
wherein the database and/or scheduling trigger and/or process manager in the system comprises one or more in a distributed arrangement.
Further, the system is characterized in that the system further comprises:
the management background is used for adding scheduling task data;
the database is used for acquiring and storing scheduling task data added by the management background.
Further, the management background is further configured to send a process start instruction or a process stop instruction to the process manager, so that the process manager starts or stops a process.
Further, the management background is further configured to store and present information related to at least any one of a database, a scheduling trigger, or a process manager.
Further, the process manager is further configured to monitor a process, and when an exception occurs in the process, perform corresponding processing.
Further, the process manager is further characterized in that the process manager is further configured to log.
Further, the system is characterized in that the system further comprises:
and the log analyzer is used for receiving the log sent by the process manager, analyzing the log and sending alarm information to the management background when the log is abnormal, wherein the log analyzer in the system comprises one or more distributed logs.
The scheduling trigger is further configured to write a next execution time of the scheduling task into the database, write the generated instant execution task corresponding to the scheduling task into the message queue, and record a log.
According to another aspect of the present application, there is also provided a method for scheduling a distributed scheduling task, where the method includes:
the database acquires and stores scheduling task data;
reading scheduling task data in a database by a scheduling trigger, and generating an instant execution task corresponding to the scheduling task;
and the process manager starts or stops the process to control the process to execute or finish executing the instant execution task corresponding to the scheduling task generated by the scheduling trigger.
Further, characterized in that the method further comprises:
managing background addition task data;
the database acquires and stores the scheduling task data, wherein the acquiring and storing of the scheduling task data comprises acquiring and storing of scheduling task data added by the management background.
Further, characterized in that the method further comprises:
and the management background sends a process starting instruction or a process stopping instruction to the process manager so as to enable the process manager to start or stop the process.
Further, characterized in that the method further comprises:
and the management background stores and presents relevant information of at least any one of the database or the scheduling trigger or the process manager.
Further, characterized in that the method further comprises:
the process manager monitors the process and processes the process accordingly when the process is abnormal.
Further, characterized in that the method further comprises:
the process manager logs about process control.
Further, characterized in that the method further comprises:
and the log analyzer receives the log sent by the process manager, analyzes the log, and sends alarm information to the management background when the log is abnormal.
Further, the reading of the task data in the database and the generation of the scheduling task by the scheduling trigger further comprises:
and the scheduling trigger writes the next execution time of the generated scheduling task into the database and records a log.
Compared with the prior art, in the method, the scheduling of the scheduling tasks is realized in a mode of combining the database, the scheduling trigger and the progress manager, and the problems that the scheduling tasks on an operating system level can only be executed by a single machine, the computing capacity is limited and single-point risks exist are solved.
In addition, in the application, the management background is further used for storing and presenting the relevant information of at least any one of the database, the scheduling trigger or the process manager, and in this way, the user experience is improved and the management is facilitated through background visualization.
In addition, in this application, the system also includes a log analyzer, which is used for receiving the log sent by the process manager and analyzing the log, and when the log is abnormal, sending alarm information to the management background, so as to find and alarm in time, thereby solving the problem in time.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments made with reference to the following drawings:
FIG. 1 illustrates a scheduling system environment diagram of scheduling tasks in accordance with an aspect of the subject application;
FIG. 2 illustrates a flow diagram of a method for scheduling of scheduled tasks in accordance with another aspect of the subject application;
the same or similar reference numbers in the drawings identify the same or similar elements.
Detailed Description
The present invention is described in further detail below with reference to the attached drawing figures.
In a typical configuration of the present application, the network device, the user device, and the terminal device each include one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory, Random Access Memory (RAM), and/or non-volatile memory in a computer-readable medium, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, which include both non-transitory and non-transitory, removable and non-removable media, may implement the information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (transient media), such as modulated data signals and carrier waves.
To further illustrate the technical means and effects adopted by the present application, the following description clearly and completely describes the technical solution of the present application with reference to the accompanying drawings and preferred embodiments.
FIG. 1 is a schematic diagram illustrating an exemplary scheduling system environment for scheduling tasks according to an aspect of the present application, wherein the system comprises:
the database 11 is used for acquiring and storing scheduling task data;
the scheduling trigger 12 is used for reading scheduling task data in the database 11 and generating an instant execution task corresponding to the scheduling task;
the process manager 13 is configured to start or stop a process, so as to control the process to execute or finish executing an instant execution task corresponding to the scheduling task generated by the scheduling trigger 12;
wherein the database 11 and/or the scheduling trigger 12 and/or the process manager 13 in the system comprise one or more in a distributed arrangement.
In this embodiment, the database 11 is configured to obtain and store scheduling task data, wherein the scheduling task data is task data related to scheduling, wherein the scheduling refers to a task being executed at a future time, and the scheduling task specifies a task that can be executed one or more times at the future time. Scheduling tasks are divided into two categories: fixed-point execution, wherein a task is executed only once at a certain future time point; or cyclically, with tasks being executed at intervals. The task to be executed immediately corresponding to the scheduling task is the task to be executed immediately after the scheduling task is generated, and all tasks generated by the scheduling trigger are all tasks to be executed immediately.
The scheduling task data refers to relevant information about scheduling tasks. For example, assuming that the task is a mail alert task performed at 5 points per day, the scheduling task data stored in database 11 includes, but is not limited to: a project identifier for indicating from which project the task came; an action indicating an action to be performed; scheduling, indicating the time setting of task execution; status, status indicating task, enabled/disabled, etc.
Preferably, the system further comprises: a management backend 14 (not shown) for adding scheduling task data; the database 11 is configured to acquire and store scheduling task data added by the management background. In this embodiment, the management background 14 may add the scheduling task data by obtaining an addition operation of the user, for example, the user may add the scheduling task data through an operation interface of the management background 14, or the management background 14 may also automatically add the scheduling task data based on a preset rule, which is not limited herein. The scheduling task data of the management background can come from various business systems, such as a package system, a monitoring system, an order system, a scanning and recording system and the like. The management background 14 is used for recording relevant information of each business system, maintaining the accuracy of the information,
continuing in this embodiment, the scheduling trigger 12 is configured to read scheduling task data in the database 11, and generate an instant execution task corresponding to the scheduling task, where the instant execution task corresponding to the scheduling task includes a task determined based on a scheduling in the task data, and includes task data that is to be executed next time and is equal to the current time.
Specifically, after the scheduling trigger 12 is started, the scheduling task data in the database 11 is read and stored in the memory, and through the storage of the memory, when the scheduling trigger 12 needs to use the scheduling task data, the data does not need to be read from the database 11. Further, the scheduling trigger 12 traverses the memory data, generates the time to be executed next time according to the scheduling, and determines whether the next execution time is equal to the current time, if not, skips over, and if so, adds the task to the queue.
In this embodiment, the scheduling trigger 12 can read only a small amount of database 11 data, there is no time consuming action, the performance is high, and N times can be executed within 1 second, where 1 million < N <1 hundred million, so second level tasks are supported. The format of the schedule definition is compatible with the Crontab format of Linux, one more item than the Crontab is used for defining the second-level task, and the Crontab format is as follows: time-sharing day, month and week; scheduling format: second, time, day, month and week. And writing the generated instant execution task corresponding to the scheduling task into the message queue, and writing the instant execution task corresponding to the scheduling task into the message queue, so that the process can acquire the instant execution task corresponding to the scheduling task from the message queue for execution. Therefore, in the present application, the scheduling trigger 12 is only responsible for generating the immediate execution task corresponding to the scheduling task and is not responsible for executing the immediate execution task, and the generation and the execution are separated from each other and are not limited by the task environment.
Moreover, in this embodiment, the scheduling triggers 12 may also be distributed and horizontally expanded, so that multiple triggers may be deployed, and if the current scheduling trigger 12 goes down, the scheduling trigger may be executed by another scheduling trigger, thereby avoiding a failure risk caused by a single trigger.
Preferably, the scheduling trigger 12 is further configured to write the next execution time of the scheduling task into the database 11 and log the next execution time. The data can be stored by writing the next execution time of the scheduling task into the database 11, and when the scheduling trigger 12 fails, the next execution time of the scheduling task can be read from the database 11, so that data loss is avoided. By recording the log, the next execution time of the scheduling task can be recorded so as to be used for later-stage sharing.
Continuing in this embodiment, the process manager 13 is configured to start or stop a process to control the process to execute or end executing an instant execution task corresponding to the scheduling task generated by the scheduling trigger 12. In this embodiment the process manager 13 would take control of the process to execute the start and stop of the task script to start or stop it from executing the task.
Specifically, when the process is started, the process manager 13 reads the configuration of the database 11, starts the process according to the configuration, and outputs the standard of the process, and redirects the standard error to the file. Wherein, the standard output refers to normal output in the process execution and is defaulted to the terminal; the standard error refers to error information output in the process execution and defaults to the terminal; and then the normal output and error information of the process are all written into the file.
Preferably, the management background 14 is further configured to send a process start instruction or a process stop instruction to the process manager 13, so that the process manager 13 starts or stops a process.
In this embodiment, the starting or stopping of the process may also be controlled by the management background 14, and a corresponding signal is sent to the process according to a start or stop instruction sent by the management background 14.
Preferably, the process manager 13 is further configured to monitor a process, and perform corresponding processing when an exception occurs in the process. For example, a process crashes, starting a new process; if the process has abnormal output, alarming; the process is restarted regularly, namely, the current process is stopped and a new process is started at intervals; the process is started more and redundant processes are stopped; starting few processes and starting lacking processes; a process is initiated by other users, an alarm, etc. The above process exception and the corresponding processing method are only examples, and other process exceptions and corresponding processing methods that may occur in the present or future are also included in the scope of the present application, and are included herein by reference.
In this embodiment, the process manager 13 may be deployed in a multi-machine distributed fashion, and the processes may run on designated machines. If the current process manager 13 is down, second-level switching can be achieved by using load balancing.
Preferably, the process manager 13 is further configured to log the execution status of the process and the occurrence of an exception.
Preferably, the system further comprises: and a log analyzer 15 (not shown) configured to receive the log sent by the process manager 13, analyze the log, and send alarm information to the management background 14 when the log is abnormal, where the log analyzer 15 in the system includes one or more logs in a distributed arrangement.
Specifically, the log analyzer 15 may implement real-time analysis, for example, analyze the received log in a streaming processing manner, where the streaming processing means that after the log is generated, the process manager 13 immediately sends the log to the log analyzer 15, and after the log analyzer 15 receives the log, the log analyzer performs analysis according to a predetermined rule, and after the analysis is completed, the log is stored. The streaming processing is mainly performed after receiving data and before storing the data, and cannot be repeatedly performed. Here, the log analyzer 15 may be deployed in multiple computers and distributed, so that the logs may be stored in a distributed manner.
Specifically, for abnormal alarm, the alarm is divided into real time and non-real time. Real-time alarming is carried out in a streaming processing mode, and alarming is carried out when abnormality is found; the non-real-time alarm is to scan logs regularly, and alarm when the result of the summary analysis exceeds an index value.
Preferably, the management background 14 is further configured to store and present information related to at least one of the database 11, the scheduling trigger 12, or the process manager 13.
The database comprises a server for storing scheduling task data, and can be distributed and horizontally expanded. The scheduling trigger, the process manager or the log analyzer may be a server or a part deployed on a server, and if the scheduling trigger, the process manager or the log analyzer is a part deployed on a server, at least any one of the scheduling trigger, the process manager or the log analyzer may be deployed on one server or separately deployed on different servers.
Thus, the database or scheduling trigger or process management server includes, but is not limited to, a computer, a network host, a single network server, a plurality of network server sets, or a cloud of multiple servers; here, the Cloud is composed of a large number of computers or web servers based on Cloud Computing (Cloud Computing), which is a kind of distributed Computing, one virtual supercomputer consisting of a collection of loosely coupled computers.
The management background 14 may implement server management, specifically, if a scheduling trigger or a process manager is deployed as a part of a device on a server, the management background 14 needs to manage the corresponding server, for example, record related information of the server and a service running on the server.
In addition, the management background 14 also needs to implement process management, record relevant information of the process, such as the affiliated service, maintainer, which servers run on, and the like. Manage the relevant state of the process, start/stop, etc.
In addition, the management background 14 performs scheduling management, manages timing tasks, adjusts task scheduling, manages the status of scheduling tasks, enables/disables, and the like. Therefore, when the user needs to view the information, the user can view the information through the browser interface of the management background 14, and can update the management background 14 through the browser.
Compared with the prior art, in the method, the scheduling of the scheduling tasks is realized in a mode of combining the database, the scheduling trigger and the scheduling manager, and the problems that the scheduling tasks on an operating system level can only be executed by a single machine, the computing capacity is limited and a single point risk exists are solved.
In addition, in the application, the management background is further used for storing and presenting the relevant information of at least any one of the database, the scheduling trigger or the process manager, and in this way, the user experience is improved and the management is facilitated through background visualization.
In addition, in this application, the system also includes a log analyzer, which is used for receiving the log sent by the process manager and analyzing the log, and when the log is abnormal, sending alarm information to the management background, so as to find and alarm in time, thereby solving the problem in time.
Fig. 2 is a flow chart illustrating a scheduling method of scheduling scheduled tasks according to another aspect of the present application, where the method includes:
the S21 database acquires and stores scheduling task data;
s22, reading the scheduling task data in the database by the scheduling trigger, and generating an instant execution task corresponding to the scheduling task;
and S23, the process manager starts or stops the process to control the process to execute or finish executing the instant execution task corresponding to the scheduling task generated by the scheduling trigger.
In this embodiment, in the step S21, the database acquires and stores scheduling task data, wherein the scheduling task data is task data related to scheduling, wherein the scheduling refers to a time in the future when a task is executed, and the scheduling task specifies a task that can be executed one or more times at a future time point. The scheduling is divided into two categories: fixed-point execution, wherein a task is executed only once at a certain future time point; or cyclically, with tasks being executed at intervals. The task to be executed immediately corresponding to the scheduling task is the task to be executed immediately after the scheduling task is generated, and all tasks generated by the scheduling trigger are all tasks to be executed immediately.
For task data, relevant information about the task is meant. For example, assuming a task is to perform a mail alert task at 5 points per day, the scheduling task data stored in the database includes, but is not limited to: an item identification to indicate from which item the task came; an action indicating an action to be performed; scheduling, indicating the time setting of task execution; status, status indicating task, enabled/disabled, etc.
Preferably, the method further comprises: s24 (not shown) managing background addition scheduling task data; wherein the step S21 includes: and the database acquires and stores scheduling task data added by the management background. In this embodiment, the management background may add the scheduling task data by obtaining an addition operation of the user, for example, the user may add the scheduling task data through an operation interface of the management background, or the management background may also automatically add the scheduling task data based on a preset rule, which is not limited herein. The scheduling task data of the management background can come from various business systems, such as a package system, a monitoring system, an order system, a scanning and recording system and the like. The management background is used for recording the relevant information of each service system, maintaining the accuracy of the information,
in this embodiment, in step S22, the scheduling trigger reads the scheduling task data in the database and generates the instant execution task corresponding to the scheduling task, where the instant execution task corresponding to the scheduling task includes a task determined based on the scheduling in the task data, and includes task data that is to be executed next time and is equal to the current time.
Specifically, after the scheduling trigger is started, the scheduling task data in the database is read and stored in the memory, and when the scheduling trigger needs to use the scheduling task data, the data does not need to be read from the database again through the storage of the memory. Furthermore, the scheduling trigger traverses the memory data, generates the time to be executed next time according to the scheduling, and judges whether the next execution time is equal to the current time, if not, the next execution time is skipped over, and if so, the task is added to the queue.
In this embodiment, the scheduling trigger may only read a small amount of preset database data, and there is no time-consuming action, the performance is high, and N times can be executed within 1 second, where 1 million < N <1 hundred million, so it supports second-level tasks. The format of the schedule definition is compatible with the Crontab format of Linux, one more item than the Crontab is used for defining the second-level task, and the Crontab format is as follows: time-sharing day, month and week; scheduling format: second, time, day, month and week. And writing the generated instant execution task corresponding to the scheduling task into the message queue, and writing the instant execution task corresponding to the scheduling task into the message queue, so that the process can acquire the instant execution task corresponding to the scheduling task from the message queue for execution. Therefore, in the present application, the scheduling trigger is only responsible for generating the immediate execution task corresponding to the scheduling task and is not responsible for executing the immediate execution task, and the generation and the execution are separated from each other and are not limited by the task environment.
In addition, in the embodiment, the scheduling triggers can also be distributed and horizontally expanded, so that multiple triggers can be deployed, and if the current scheduling trigger goes down, the scheduling triggers can be executed through other scheduling triggers, thereby avoiding the risk of a single machine.
Preferably, the method further comprises: s25 (not shown), the scheduling trigger writes the next execution time of the generated scheduling task into the database, writes the instant execution task corresponding to the generated scheduling task into the message queue, and records a log. The next execution time of the scheduling task is written into the database, so that the data can be stored, and when the scheduling trigger breaks down, the next execution time of the scheduling task can be read from the database, so that data loss is avoided. In addition, when a plurality of tasks are executed, the scheduling trigger writes the instant execution tasks corresponding to the generated scheduling tasks into the message queue. By recording the log, the generation of the next execution time of the scheduling task can be recorded for later analysis of problems.
In this embodiment, in step S23, the process manager starts or stops a process to control the process to execute or end executing an instant execution task corresponding to the scheduling task generated by the scheduling trigger. In this embodiment the process manager would take control of the process to execute the start and stop of the task script to start or stop it from executing the task.
Specifically, when the process is started, the process manager reads the configuration of the database, starts the process according to the configuration, outputs the standard of the process, and redirects the standard error to the file. Wherein, the standard output refers to normal output in the process execution and is defaulted to the terminal; the standard error refers to error information output in the process execution and defaults to the terminal; and then the normal output and error information of the process are all written into the file.
Preferably, the method further comprises: s26 (not shown), the management background sends a process start instruction or a process stop instruction to the process manager, so that the process manager starts or stops a process.
In this embodiment, the starting or stopping of the process may also be controlled by the management background, and a corresponding signal is sent to the process according to a start or stop instruction sent by the management background.
Preferably, the method further comprises: s27 (not shown), the process manager monitors the process and handles it accordingly when an exception occurs in the process. For example, a process dies, starting a new process; if abnormal input occurs in the process, alarming; restarting the process at regular time, namely killing the current process at intervals and starting a new process; the process is started more, and redundant processes are killed; the process is less started, and the missing process is complemented; the process is initiated by other users, an alarm, etc., and the above process exception and corresponding handling method are only examples, and other existing or future process exceptions and corresponding handling methods are also included in the scope of the present application and are included herein by reference.
In this embodiment, the process manager may be deployed in a multi-machine distributed fashion, and the processes may run on designated machines. If the current process manager is down, second-level switching can be achieved by using load balancing.
Preferably, the method further comprises: s28 (not shown), the process manager logs the execution status of the process and the occurrence of exceptions, etc.
Preferably, the method further comprises: s29 (not shown): and the log analyzer is used for receiving the log sent by the process manager, analyzing the log and sending alarm information to the management background when the log is abnormal, wherein the log analyzer in the system comprises one or more distributed logs.
Specifically, the log analyzer may implement real-time analysis, for example, analyze the received log in a streaming processing manner, where the streaming processing means that the log is generated and then immediately sent to the log analyzer, and after receiving the log, the log analyzer performs analysis according to a predetermined rule, and after the analysis is completed, stores the log. The streaming processing is mainly performed after receiving data and before storing the data, and cannot be repeatedly performed. The log analyzer can be deployed in a plurality of machines and distributed, so that logs can be stored in a distributed mode.
Specifically, for abnormal alarm, the alarm is divided into real time and non-real time. Real-time alarming is carried out in a streaming processing mode, and alarming is carried out when abnormality is found; the non-real-time alarm is to scan logs regularly, and alarm when the result of the summary analysis exceeds an index value.
Preferably, the method further comprises: s30 (not shown), the management back office stores and presents information related to at least one of a database or a scheduling trigger or a process manager.
The database comprises a server for storing task data, and can be distributed and horizontally expanded. The scheduling trigger, the process manager or the log analyzer may be a server or a part deployed on a server, and if the scheduling trigger, the process manager or the log analyzer is a part deployed on a server, at least any one of the scheduling trigger, the process manager or the log analyzer may be deployed on one server or separately deployed on different servers.
Thus, the database or scheduling trigger or process management server includes, but is not limited to, a computer, a network host, a single network server, a plurality of network server sets, or a cloud of multiple servers; here, the Cloud is composed of a large number of computers or web servers based on Cloud Computing (Cloud Computing), which is a kind of distributed Computing, one virtual supercomputer consisting of a collection of loosely coupled computers.
The management background may implement server management, and specifically, if the scheduling trigger or the process manager is deployed as a part of a device on the server, the management background manages the corresponding server, for example, record related information of the server and a service running on the server.
In addition, the management background also needs to realize process management and record relevant information of the process, such as affiliated service, maintainer, which servers run on, and the like. Manage the relevant state of the process, start/stop, etc.
In addition, the management background can also perform scheduling management, manage timing tasks, adjust task scheduling, manage the state of scheduling tasks, start/stop and the like. Therefore, when the user needs to check the information, the information can be checked through a browser interface of the management background, and the management background can be updated through the browser.
Compared with the prior art, in the method, the scheduling of the scheduling tasks is realized in a mode of combining the database, the scheduling trigger and the scheduling manager, and the problems that the scheduling tasks on an operating system level can only be executed by a single machine, the computing capacity is limited and a single point risk exists are solved.
In addition, in the application, the management background is further used for storing and presenting the relevant information of at least any one of the database, the scheduling trigger or the process manager, and in this way, the user experience is improved and the management is facilitated through background visualization.
In addition, in this application, the system also includes a log analyzer, which is used for receiving the log sent by the process manager and analyzing the log, and when the log is abnormal, sending alarm information to the management background, so as to find and alarm in time, thereby solving the problem in time.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is to be understood that the word "comprising" does not exclude other embodiments or steps, and the singular does not exclude the plural.
Claims (16)
1. A distributed scheduling system for scheduled tasks, the system comprising:
the database is used for acquiring and storing scheduling task data;
the scheduling trigger is used for reading scheduling task data in the database and generating an instant execution task corresponding to the scheduling task;
the process manager is used for starting or stopping the process so as to control the process to execute or finish executing the instant execution task corresponding to the scheduling task generated by the scheduling trigger;
wherein the database and/or scheduling trigger and/or process manager in the system comprises one or more in a distributed arrangement.
2. The system of claim 1, further comprising:
the management background is used for adding scheduling task data;
the database is used for acquiring and storing scheduling task data added by the management background.
3. The system of claim 2, wherein the management background is further configured to send a process start instruction or a process stop instruction to the process manager, so that the process manager starts or stops a process.
4. The system of claim 2 or 3, wherein the management background is further configured to store and present information related to at least one of a database or a scheduling trigger or a process manager.
5. The system of any one of claims 1 to 4, wherein the process manager is further configured to monitor a process and to handle when an exception occurs in the process accordingly.
6. The system of any of claims 1 to 5, wherein the process manager is further configured to log.
7. The system of claim 6, further comprising:
and the log analyzer is used for receiving the log sent by the process manager, analyzing the log and sending alarm information to the management background when the log is abnormal, wherein the log analyzer in the system comprises one or more distributed logs.
8. The system according to any one of claims 1 to 7, wherein the scheduling trigger is further configured to write a next execution time of the scheduled task into the database, write a generated instant execution task corresponding to the scheduled task into a message queue, and record a log.
9. A method for scheduling a scheduled task, the method comprising:
the database acquires and stores scheduling task data;
reading scheduling task data in a database by a scheduling trigger, and generating an instant execution task corresponding to the scheduling task;
and the process manager starts or stops the process to control the process to execute or finish executing the instant execution task corresponding to the scheduling task generated by the scheduling trigger.
10. The method of claim 9, further comprising:
adding scheduling task data to a management background;
the database acquires and stores the scheduling task data, wherein the acquiring and storing of the scheduling task data comprises acquiring and storing of scheduling task data added by the management background.
11. The method of claim 10, further comprising:
and the management background sends a process starting instruction or a process stopping instruction to the process manager so as to enable the process manager to start or stop the process.
12. The method according to claim 10 or 11, characterized in that the method further comprises:
and the management background stores and presents relevant information of at least any one of the database or the scheduling trigger or the process manager.
13. The method according to any one of claims 9 to 12, further comprising:
the process manager monitors the process and processes the process accordingly when the process is abnormal.
14. The method according to any one of claims 9 to 13, further comprising:
the process manager logs about process control.
15. The method of claim 14, further comprising:
and the log analyzer receives the log sent by the process manager, analyzes the log, and sends alarm information to the management background when the log is abnormal.
16. The method of any one of claims 9 to 15, wherein the scheduling trigger reading scheduling task data in a database and generating an instant execution task corresponding to the scheduling task further comprises:
and the scheduling trigger writes the next execution time of the scheduling task into the database and records a log.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910477801.XA CN110597603A (en) | 2019-06-03 | 2019-06-03 | Scheduling method and system of distributed scheduling tasks |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910477801.XA CN110597603A (en) | 2019-06-03 | 2019-06-03 | Scheduling method and system of distributed scheduling tasks |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110597603A true CN110597603A (en) | 2019-12-20 |
Family
ID=68852598
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910477801.XA Pending CN110597603A (en) | 2019-06-03 | 2019-06-03 | Scheduling method and system of distributed scheduling tasks |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110597603A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112379877A (en) * | 2020-11-13 | 2021-02-19 | 武汉蓝星科技股份有限公司 | Programming system of embedded end multi-process GUI program |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103197969A (en) * | 2013-03-27 | 2013-07-10 | 百度在线网络技术（北京）有限公司 | Distributed timed task control device and method |
| RU2012156428A (en) * | 2012-12-25 | 2014-06-27 | Закрытое акционерное общество "Лаборатория Касперского" | SYSTEM AND METHOD FOR FAILURE-RESISTANT SCHEDULE TASKS IN DISTRIBUTED MEDIA |
| CN104331327A (en) * | 2014-12-02 | 2015-02-04 | 山东乾云启创信息科技有限公司 | Optimization method and optimization system for task scheduling in large-scale virtualization environment |
| CN106874090A (en) * | 2017-01-23 | 2017-06-20 | 北京思特奇信息技术股份有限公司 | Job scheduling method and system based on cloud system |
| CN108108234A (en) * | 2017-12-06 | 2018-06-01 | 链家网（北京）科技有限公司 | A kind of distributed task management method and system |
-
2019
- 2019-06-03 CN CN201910477801.XA patent/CN110597603A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| RU2012156428A (en) * | 2012-12-25 | 2014-06-27 | Закрытое акционерное общество "Лаборатория Касперского" | SYSTEM AND METHOD FOR FAILURE-RESISTANT SCHEDULE TASKS IN DISTRIBUTED MEDIA |
| CN103197969A (en) * | 2013-03-27 | 2013-07-10 | 百度在线网络技术（北京）有限公司 | Distributed timed task control device and method |
| CN104331327A (en) * | 2014-12-02 | 2015-02-04 | 山东乾云启创信息科技有限公司 | Optimization method and optimization system for task scheduling in large-scale virtualization environment |
| CN106874090A (en) * | 2017-01-23 | 2017-06-20 | 北京思特奇信息技术股份有限公司 | Job scheduling method and system based on cloud system |
| CN108108234A (en) * | 2017-12-06 | 2018-06-01 | 链家网（北京）科技有限公司 | A kind of distributed task management method and system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112379877A (en) * | 2020-11-13 | 2021-02-19 | 武汉蓝星科技股份有限公司 | Programming system of embedded end multi-process GUI program |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112346829B (en) | Method and equipment for task scheduling | |
| CN107451147B (en) | Method and device for dynamically switching kafka clusters | |
| US20210133054A1 (en) | Prioritized transfer of failure event log data | |
| CN102255866A (en) | Method and device for downloading data | |
| CN111339118A (en) | Kubernetes-based resource change history recording method and device | |
| WO2020238860A1 (en) | Distributed file batch processing method and apparatus, and readable storage medium | |
| CN110557291A (en) | Network service monitoring system | |
| CN106649000B (en) | Fault recovery method of real-time processing engine and corresponding server | |
| CN110597603A (en) | Scheduling method and system of distributed scheduling tasks | |
| CN111478792B (en) | Cutover information processing method, system and device | |
| CN110334011B (en) | Method and device for executing test case | |
| CN116841653A (en) | Execution method and device of operation and maintenance job, processor and electronic equipment | |
| CN110019045B (en) | Log floor method and device | |
| CN115617459A (en) | Method, device and equipment for resource scheduling | |
| CN116185706A (en) | Data backup method and device, computing and storage medium and electronic equipment | |
| CN115080309A (en) | Data backup system, method, storage medium, and electronic device | |
| CN115114275A (en) | Data acquisition method, device and medium | |
| CN111600771B (en) | Network resource detection system and method | |
| CN113806438A (en) | Warehousing and preprocessing method and system based on massive BCP data files | |
| CN114327835A (en) | Distributed task scheduling method and device, processor and electronic equipment | |
| KR20130067959A (en) | Method and appratus for processing error of application | |
| CN111866576A (en) | On-line problem processing system and method | |
| CN112463514A (en) | Monitoring method and device for distributed cache cluster | |
| CN112118294B (en) | Request processing method, device and medium based on server cluster | |
| CN116541362A (en) | Log processing method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20191220 |
|
| RJ01 | Rejection of invention patent application after publication |