CN109656690A - Scheduling system, method and storage medium - Google Patents
Scheduling system, method and storage medium Download PDFInfo
- Publication number
- CN109656690A CN109656690A CN201710942824.4A CN201710942824A CN109656690A CN 109656690 A CN109656690 A CN 109656690A CN 201710942824 A CN201710942824 A CN 201710942824A CN 109656690 A CN109656690 A CN 109656690A
- Authority
- CN
- China
- Prior art keywords
- node
- task
- scheduling
- nodes
- slave
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 30
- 230000008878 coupling Effects 0.000 claims description 3
- 238000010168 coupling process Methods 0.000 claims description 3
- 238000005859 coupling reaction Methods 0.000 claims description 3
- 230000004044 response Effects 0.000 claims description 3
- 238000013461 design Methods 0.000 abstract description 9
- 238000004590 computer program Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 8
- 238000012544 monitoring process Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 2
- 230000002688 persistence Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000000712 assembly Effects 0.000 description 1
- 238000000429 assembly Methods 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4812—Task transfer initiation or dispatching by interrupt, e.g. masked
- G06F9/4825—Interrupt from clock, e.g. time of day
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a kind of scheduling system, method and storage mediums.The system includes: node collection, configuration center component and task queue component, node collection is coupled with configuration center component and task queue component respectively, wherein: node collection includes: host node and from node, host node is used for: starting timer, when timer reaches the predetermined time, operation configuration data is obtained from configuration center component, scheduler task is generated according to operation configuration data, task queue component is written into scheduler task, so that scheduler task is executed from node.The embodiment of the present invention abandons the design that traditional multinode all enables timer, using master-slave mode, only allows host node to start timer, and determine what operation when dispatched, is only responsible for executing scheduling from node.The embodiment of the present invention has many advantages, such as that the division of labor is clear, it is efficient to dispatch, can not only reduce the resource overhead of timer, it is ensured that same task will not be dispatched repeatedly.
Description
Technical Field
The present invention relates to the field of network communication technologies, and in particular, to a scheduling system, a scheduling method, and a storage medium.
Background
With the rapid development of internet technology, intelligent services are rapidly developed. In a good intelligent service technical system, an efficient and highly reliable timing scheduling device is needed to support the capabilities of service log acquisition, service monitoring, scaling scheduling and the like, and realize the automatic operation and maintenance capability of the service, the quality analysis capability of the service and the dynamic scaling capability of the service in the service category.
Many existing toolkits and technical frameworks have certain capability of scheduling at regular time, and schedule jobs at regular time mainly according to page configuration requirements. But this approach often suffers from a single point of failure. To avoid single point of failure, a multi-point deployment approach is usually adopted. However, multi-drop deployment introduces the problem that the same task is scheduled once at different nodes. To solve this problem, it is common practice to: firstly inquiring, then judging, then setting and finally executing. Specifically, before the timing scheduling is started, each node firstly designates a persistence unit to inquire whether the scheduling is executed by other nodes, if not, the scheduling is set as executed, and then the scheduling is started to be executed; otherwise execution is aborted.
The applicant finds out through research that: the existing scheme of inquiring, judging, setting and executing at last can realize basic timing scheduling. However, each node is responsible for operations such as timer monitoring, scheduling execution, query, judgment, setting and the like, and the calculation cost is very large. Meanwhile, a persistence unit needs to be additionally opened, and computing resources are wasted.
How to simplify the calculation, reduce the calculation overhead, and implement efficient and highly reliable timing scheduling becomes an urgent technical problem to be solved in the industry.
Disclosure of Invention
In order to solve the problem of high computational overhead and realize efficient and high-reliability timing scheduling, embodiments of the present invention provide a scheduling system, a scheduling method, and a storage medium.
In a first aspect, a scheduling system is provided. The system comprises: a node set, a configuration center component and a task queue component,
the node sets are respectively coupled to the configuration center component and the task queue component,
the node set includes: a master node and a slave node, wherein,
a master node to: and starting a timer, acquiring job configuration data from the configuration center component when the timer reaches preset time, generating a scheduling task according to the job configuration data, and writing the scheduling task into the task queue component so that the scheduling task is executed by the slave node.
In a second aspect, a method for scheduling a system is provided. The method comprises the following steps:
the master node starts a timer;
when the timer reaches preset time, the main node acquires operation configuration data from the configuration center component;
the main node generates a scheduling task according to the operation configuration data;
the master node writes the scheduled task to the task queue component for execution by the slave node.
In a third aspect, a scheduling system is provided. The system comprises:
a memory for storing a program;
a processor for executing a program stored in the memory, the program causing the processor to perform the method described above.
In a fourth aspect, a computer-readable storage medium is provided. The computer readable storage medium has stored therein instructions which, when executed on a computer, cause the computer to perform the method of the above aspects.
In a fifth aspect, a computer program product containing instructions is provided. The product, when run on a computer, causes the computer to perform the method of the aspects described above.
In a sixth aspect, a computer program is provided. The computer program, when run on a computer, causes the computer to perform the methods of the aspects described above.
Therefore, the above embodiment of the present invention abandons the traditional design of starting timers at multiple points, and adopts a master-slave manner, so that only the master node starts the timers and determines when to schedule which job, and the other slave nodes are only responsible for executing scheduling. The embodiment of the invention has clear division of labor and high scheduling efficiency, not only can solve the resource overhead of the traditional timer, but also ensures that the same task cannot be scheduled for many times.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments of the present invention will be briefly described below, and it is obvious that the drawings described below are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 may be an architectural diagram of a scheduling system in accordance with an embodiment of the present invention;
FIG. 2 may be a schematic diagram of a dispatch system startup of an embodiment of the present invention;
FIG. 3 is a schematic diagram of an operational configuration according to one embodiment of the present invention;
FIG. 4 may be a schematic diagram of scheduling jobs according to an embodiment of the present invention;
FIG. 5 may be a flowchart illustrating a method of scheduling according to an embodiment of the invention;
fig. 6 is a schematic diagram of a scheduling system according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
Fig. 1 may be an architecture diagram of a scheduling system according to an embodiment of the present invention.
As shown in fig. 1, the system architecture may include: node assembly 100, configuration center component 200, task queue component 300, load balancer 400. Wherein node assemblies 100 may be coupled to configuration hub component 200, task queue component 300, and load balancer 400, respectively.
The node assembly 100 may include a plurality of nodes, such as 1 master node 10 and a plurality of slave nodes. The plurality of slave nodes are, for example, the slave node 10, the slave node 20, the slave node 30, and the like.
A timer 12 may be provided locally at the master node 10 for timing the scheduling of various tasks.
Each slave node may be a plurality of virtual machines disposed in a physical machine, or may be a physical machine such as a server, a workstation, an operator station, or the like, for example, a physical server with a model x 86. The slave node may also be an internet of things device. The slave node may be installed with various messaging client applications such as instant messaging tools, mailbox clients, social platform software, audio video software, and the like.
The master node and the slave nodes may each comprise a coordinator. For example, the master node 10 may include a coordinator 11, the slave node 20 may include a coordinator 21, and the slave node 30 may include a coordinator 31. The coordinator 11, the coordinator 21, and the coordinator 31 cooperate with each other to form a virtual self-contained coordination center in the node assembly 100 for coordination. The self-contained mode is realized without depending on other components when the coordination application (such as coordination selection of a main node) is applied.
The coordinator 11, the coordinator 21, and the coordinator 31 coordinate with each other, may select 1 master node from a plurality of nodes, and determine the remaining nodes as slave nodes, such that the node set 100 forms a master-slave structure. In this master-slave configuration, the master node and the slave nodes may change roles. For example, when the master node fails, the coordinator 11, the coordinator 21, and the coordinator 31 may reconcile information of the selected node from the nodes, determine the reselected node as the master node, and determine the remaining nodes as the slave nodes. Specifically, the reselection of the master node may be implemented in a voting manner, or in a rotation manner.
User 600 may access load balancer 400 using communication device (e.g., cell phone) 500 and initiate a request to configure job data, such as configure: and the device performs service monitoring request at 9:00-17:00 working days.
Load balancer 400 may be used to receive a request from user 600 to configure job data. In response to the request for configuration job data, load balancer 400 may randomly select any one of the nodes (e.g., master node 10) in the node set, and write the job configuration data to configuration hub component 200 from the selected node. Load balancer 400 may also select any node in the node set by polling or the like. Through the load balancing process of the load balancer 400, each node in the node set can be given an opportunity to perform a write operation defined by the page configuration.
The configuration center component 200 may receive job configuration data written by any node in the node set, store the job configuration data when the job configuration data is successfully written, and feed back a write success message; and feeding back a write failure message when the configuration data of the write operation fails.
The master node 10 may be configured to start the local timer 12, retrieve job configuration data from the configuration center component 200, and add the local timer 12 to the job configuration data. When the local timer 12 reaches a predetermined time, a scheduling task is generated according to the job configuration data, and the scheduling task is written to the task queue component 300 so that the scheduling task is executed by the slave node 20 or the slave node 30.
The slave node 20 or the slave node 30 may obtain the scheduled task (task of 9:00-17:00 service monitoring of weekdays) from the task queue component 300 when their running status is idle and determine whether the scheduled task can be executed (e.g., determine whether there are other tasks for 9:00-17:00 of weekdays or whether there is hardware and software support to monitor services, etc.). When the judgment result is that the scheduling task can be executed, executing the scheduling task; or, when the judgment result is that the execution cannot be performed, discarding the scheduling task; or when the judgment result is that the scheduling task cannot be executed, writing the scheduling task into the task queue component so that the scheduling task can be scheduled again. The design can prevent the task from being discarded at will, and ensure that the task can be smoothly executed by the nodes meeting the requirements.
It should be understood that the number of devices in fig. 1 is merely illustrative. The various nodes (or instances), coordinators, configuration center component 200, task queue component 300, etc. may be adjusted as desired for implementation.
In addition, the system architecture may further include auxiliary devices such as a network and a router. For example, a network may be the medium used to provide communications links between various electronic devices. In particular, the network may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
In this embodiment, the timing scheduling system can be implemented by JDK1.6 +. The bottom layer can realize the configuration of the coordinator according to zab protocol or Raft protocol, and can introduce a queue system (such as Redis queue).
In the present embodiment, all or part of each of the above components and devices may be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the functions or processes according to the embodiments of the present invention are generated in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another computer readable storage medium, for example, where the computer instructions may be transmitted from one node to another website, computer, server, or data center by wire (e.g., coaxial cable, fiber optics, Digital Subscriber Line (DSL)), or wirelessly (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The following embodiments can all apply the system architecture of the present embodiment to perform processes such as timing scheduling. For simplicity of description, the various embodiments may be referred to and cited with respect to each other.
The application scenario of the embodiment may be a scenario of a console (for example, for monitoring a certain task), or a scenario of a proprietary cloud (for example, for a cloud server to learn a training task), and the like. Taking the example of monitoring the data flow of the website a in the scenario of the console as follows, the implementation of the timing scheduling is specifically described:
first, the user 600 directly accesses the load balancer 400 using the cell phone 500, and writes job configuration data regarding "task of monitoring data flow of website a on workday 9:00-17: 00" in the interface of the job configuration service of one node in the node set 100, thereby initiating a scheduled scheduling request to the node set 100.
Then, in response to the timing scheduling request, the coordinator 11, the coordinator 21, and the coordinator 31 can coordinate the selection of the master node 10 from the node set, and the remaining nodes are the slave node 20 and the slave node 30.
Next, the master node 10 starts a timer 12 and generates a scheduling task of "task of monitoring the data stream of site A on weekdays 9:00-17: 00".
Finally, the master node 10 writes the scheduled task to a queue in the task queue component 300. Then, when the slave node 20 is idle, the scheduling task is removed from the queue from the slave node 20 and the data flow of the A site is monitored at 9:00-17:00 on weekdays.
To describe the implementation of the timing scheduling in detail, the following three aspects are described in detail: in the first aspect, a timing scheduling system is started; in a second aspect, a timed job configuration; in a third aspect, jobs are scheduled periodically.
Fig. 2 may be a schematic diagram of a scheduling system startup according to an embodiment of the invention.
Referring to fig. 2, scheduling system startup may include the steps of:
s201, each node starts each coordinator, that is: coordinator 11, coordinator 21, and coordinator 31.
And S202, after the node learns that the coordinator is started successfully, the node starts to participate in service election.
S203, the elected node becomes the master node (e.g., master node 10), and the other nodes become standby nodes (e.g., slave nodes 20 and 30). And the standby nodes participate in election after the main node fails, wherein one standby node becomes a new main node.
S204, the master node 10 starts the timer 12.
When the service is started, each node can be used as a node for load balancing to receive the configuration request of the timing operation. But only one master node in the node cluster will extract tasks periodically. The purpose of this service initiation step is to determine which node in the cluster is the master node and to implement the initialization work for subsequent timing scheduling.
FIG. 3 is a schematic diagram of an operational configuration according to an embodiment of the present invention.
In this embodiment, an Application Programming Interface (API) of the timing scheduling system is peer-to-peer on each node. After each node is started successfully, an interface of the timing operation configuration service is provided for the outside.
Referring to fig. 3, for a user 600, the load balancer 400 in the front can be directly accessed through the handset 500, and any node in the cluster is randomly requested without sensing the node that specifically performs configuration writing. Each node may be responsible for the integrity of the configuration transaction. For example, each node may feed back that the user 600 successfully written the job configuration data to the configuration hub component 200. As another example, each node may roll back failed configurations and feed back the reason for the failure to user 600. This step is non-differentiated for each node, i.e. the master node 10, the slave node 20 and the slave node 30 all have the opportunity to perform a page configuration defined write operation.
For example, when user 600 is routed through load balancer 400 to slave node 20, user 600 adds a new timed scheduling job on slave node 20 and commits the configuration. The slave node 20 will submit this configuration data to the configuration center component 200. In the event that there is no conflict with the federated primary key of the configuration data, the configuration data will be successfully written to configuration hub component 200. The slave node 20 will feed back to the user 600 that the configuration was successful, otherwise the slave node 20 will roll back this operation and the configuration center component 200 will not store this configuration information. The user 600 may get a message from the display page of the node 20 of the failure of the configuration and the reason for the failure.
In this embodiment, writing the configuration of the timing job into the configuration center may facilitate the master node 10 to obtain the timing scheduling job from a unique place (e.g., the configuration center component 200), reduce the time consumed by the master node 10 for searching the timing scheduling job, and enhance the reliability and timeliness of the timing scheduling.
FIG. 4 may be a schematic diagram of scheduling jobs according to an embodiment of the present invention.
Referring to FIG. 4, scheduling a job may include the steps of:
s401, the master node 10 will periodically obtain the configuration data of the timed job from the configuration center component 200. At this point, the master node 10 does not directly parse and execute the schedule of the job, but rather adds these configurations to the local timer 12.
From a fault tolerance perspective, if a master node 10 goes down at a certain time, the coordinator will reselect the master node, and the new master node will re-acquire the configuration from the configuration center component 200, populating the local timer 12. At this point, the master node 10 failure does not affect the contents of the task queue. Therefore, the design of the joint coordinator and the configuration center component of the system avoids the problem of coupling of scheduling and consistency judgment caused by adopting a database lock. This design can be successfully decoupled because only the master node will read the configuration center data and when to execute the job is determined by the master node's local timer.
S402, in the timer 12 of the master node 10, when the scheduling time of a certain job arrives, the master node 10 generates a task for scheduling job configuration, and writes the task into the task queue of the task queue component. Thus, the master node 10 completes the task of issuing for the slave node 20 or the slave node 30 to execute.
The purpose of such master-slave design is: only one "brain" (master node 10) starts a timer, deciding when to schedule something (scheduled job), and the other nodes (slave node 20 or slave node 30) are only responsible for performing the scheduling. This not only saves resource overhead on the timer 12 for snooping, but also ensures that the same task is not scheduled multiple times.
The design of the joint configuration center and the task queue of the system accurately realizes the load balance of each node, ensures that each node cannot be idle under the condition of task accumulation, and avoids the condition that some nodes have high loads and other nodes are always idle.
S403, when the slave node is idle, a task at the top of the task queue may be consumed (extracted) from the task queue. The task queue can adopt a first-in first-out mode to enter tasks into the task queue from the bottom of the task queue in sequence and extract the tasks from the top of the task queue. The design can ensure that each slave node participates in the execution of the scheduling task.
First, the slave node determines whether the task can be executed (the number of retries needs to be less than a threshold). If it is confirmed that the task is executable, the task is immediately executed, otherwise the task is discarded. The task queue no longer stores this task at this time. If the task of the scheduling job is successfully scheduled, the process is successfully finished; otherwise, the slave node adds 1 to the retry number of the task of the scheduling job and writes the retry number into the task queue for other nodes to call. Therefore, the embodiment of the invention can realize the fault tolerance on the scheduling at the cost of the lightest weight.
The design is a result obtained through a great deal of engineering experience, and the existing complex processes (such as consistency judgment, task issuing judgment, load of executed tasks and task execution failure fault tolerance) can be decomposed in an optimal mode. Wherein: the consistency judgment can be realized by the coordinator; the judgment of the issued task can be realized by the decomposition of a local timer of the main node; the load of executing tasks can be realized by a task queue; the fault tolerance of the task execution failure can be judged by each node and realized by adding tasks to the task queue. Therefore, the embodiment of the invention not only can realize simple use, but also can realize the effect of sound function.
In some embodiments, the timing scheduling system may include: the system comprises a node set, a configuration center component and a task queue component, wherein the node set is respectively coupled with the configuration center component and the task queue component. Wherein, the node set may include: a master node and one or more slave nodes. The master node may be configured to start a local timer, obtain job configuration data from the configuration center component, add the local timer to the job configuration data, generate a scheduling task according to the job configuration data when the local timer finishes timing, and write the scheduling task into the task queue component, so that the scheduling task is executed by the slave node.
In some embodiments, the slave node may be configured to, when the running state is idle, obtain a scheduling task from the task queue component, determine whether the scheduling task can be executed, and execute the scheduling task when the determination result indicates that the scheduling task can be executed; or, when the judgment result is that the execution cannot be performed, discarding the scheduling task; or when the judgment result is that the execution cannot be performed, the scheduling task is written into the task queue component.
In some embodiments, any one node in the node set includes a coordinator, and the coordinators in each node of the set may form a self-contained coordination center in the node set. And the slave nodes of each coordinator are used for coordinating the information of the selected nodes in the slave node set, determining the coordinately selected nodes as master nodes, and determining the rest nodes as slave nodes.
In some embodiments, the coordinator may be further configured to reconcile information of the selected node from the set of nodes when the master node fails, determine the reselected node as the master node, and determine the remaining nodes as the slave nodes.
In some embodiments, the configuration center component may be further configured to receive, in advance, job configuration data written by any node in the node set, store the job configuration data when the writing of the job configuration data is successful, and feed back a writing success message; and feeding back a write failure message when the configuration data of the write operation fails.
In some embodiments, the system may further include: a load balancer for coupling with the collection of nodes. The load balancer can also be used for receiving a request of configuring job data of a user in advance, responding to the request of configuring the job data, randomly selecting any node in the node set, and writing the job configuration data into the configuration center component by the selected node.
Fig. 5 may be a flowchart illustrating scheduling according to an embodiment of the present invention.
As shown in fig. 5, the scheduling may include the steps of:
s501, the master node starts a timer.
The timer may be a timer local to the master node.
S502, the main node acquires the job configuration data from the configuration center component.
S503, the main node adds the local timer into the job configuration data.
And S504, when the local timer finishes timing, the main node generates a scheduling task according to the job configuration data.
And S505, the master node writes the scheduling task into the task queue component so that the scheduling task can be executed by the slave node.
In some embodiments, the timing scheduling may further include the steps of:
s506, when the slave node is idle in the running state, the slave node acquires the scheduling task from the task queue component and judges whether the scheduling task can be executed.
S507, when the judgment result is that the scheduling task can be executed, the slave node executes the scheduling task; or, when the judgment result is that the scheduling task cannot be executed, the slave node discards the scheduling task; or when the judging result is that the execution cannot be carried out, the slave node writes the scheduling task into the task queue component.
In some embodiments, the timing scheduling may further include the steps of:
and S508, the coordinator coordinates and selects the nodes from the node set.
And S509, determining the coordinately selected nodes as master nodes, and determining the rest nodes as slave nodes.
In some embodiments, the timing scheduling may further include the steps of:
s510, when the main node fails, all the coordinators reconcile the information of the selected nodes from the node set.
And S511, determining the reselected node as a new master node, and determining the rest nodes as new slave nodes.
In some embodiments, the timing scheduling may further include the steps of:
s512, the configuration center component receives operation configuration data written by any node in the node set in advance.
S513, when the writing of the job configuration data is successful, the configuration center component stores the job configuration data and feeds back a writing success message.
S514, when the configuration data of the writing operation fails, the configuration center component feeds back a writing failure message.
In some embodiments, the timing scheduling may further include the steps of:
s515, the load balancer receives a request for configuring job data from a user in advance.
S516, responding to the request, the load balancer randomly selects any node in the node set, and the selected node writes the operation configuration data into the configuration center component.
In some embodiments, the set of nodes may include: setting a plurality of virtual machines in a physical machine; or a server cluster consisting of a plurality of servers.
In addition, in the case of no conflict, those skilled in the art can flexibly adjust the order of the above operation steps or flexibly combine the above steps according to actual needs. Various implementations are not described again for the sake of brevity. In addition, the contents of the various embodiments may be mutually incorporated by reference.
It should be noted that the system in each embodiment may be used as an execution main body in the method in each embodiment, and may implement a corresponding flow in each method, and for brevity, this aspect is not described again.
Fig. 6 is a schematic diagram of a scheduling system according to an embodiment of the present invention.
As shown in fig. 6, the framework may include a Central Processing Unit (CPU)601, which may perform various operations performed by the embodiment of fig. 5 according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 708 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data necessary for the operation of the system architecture are also stored. The CPU 601, ROM 602, and RAM603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
In particular, according to an embodiment of the present invention, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the invention include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (15)
1. A scheduling system, comprising: a node set, a configuration center component and a task queue component,
the set of nodes are coupled to the configuration hub component and the task queue component, respectively, wherein:
the set of nodes includes: a master node and a slave node, wherein,
the master node is configured to:
a timer is started and the time-out time is started,
obtaining job configuration data from the configuration center component when the timer reaches a predetermined time,
generating a scheduling task according to the job configuration data,
writing the scheduled task to the task queue component for execution by the slave node.
2. The system of claim 1, the slave node to:
obtaining the scheduled task from the task queue component when the running state is idle,
it is determined whether the scheduled task can be executed,
when the judgment result is that the scheduling task can be executed, executing the scheduling task;
or,
when the judgment result is that the scheduling task cannot be executed, discarding the scheduling task;
or,
and when the judgment result is that the scheduling task cannot be executed, writing the scheduling task into the task queue component.
3. The system of claim 2, wherein the master node and the slave node each comprise: a coordinator, any one of the plurality of coordinators, to:
coordinating with other coordinators to select a node from the node set,
determining the coordinately selected node as the master node,
and determining the remaining nodes as the slave nodes.
4. The system of claim 3, wherein: the coordinator is further configured to:
when the main node fails, the coordinator reconciles the selected node from the node set with other coordinators,
determining the reselected node as the master node,
and determining the remaining nodes as the slave nodes.
5. The system of claim 1, wherein: the configuration center component is configured to:
receiving the operation configuration data written by any node in the node set in advance,
when the writing of the operation configuration data is successful, storing the operation configuration data and feeding back a writing success message;
and feeding back a write failure message when the writing of the operation configuration data fails.
6. The system of claim 1, further comprising: a load balancer for coupling with the set of nodes,
the load balancer is used for:
a request of a user to configure job data is received in advance,
randomly selecting any one of the set of nodes in response to the request to configure job data,
and writing the operation configuration data into the configuration center component by the selected node.
7. The system of any of claims 1-6, wherein the set of nodes comprises:
a plurality of virtual machines arranged in one physical machine;
or,
a server cluster consisting of a plurality of servers.
8. A method of scheduling by a scheduling system, the scheduling system being according to any one of claims 1-7, comprising:
the master node starts a timer;
when the timer reaches preset time, the main node acquires operation configuration data from the configuration center component;
the main node generates a scheduling task according to the job configuration data;
the master node writes the scheduling task into the task queue component for execution by the slave node.
9. The method of claim 8, comprising:
when the slave node is idle in the running state, acquiring the scheduling task from the task queue component;
the slave node judges whether the scheduling task can be executed;
when the judging result is that the scheduling task can be executed, the slave node executes the scheduling task;
or,
when the judging result is that the scheduling task cannot be executed, the slave node discards the scheduling task;
or,
and when the judging result is that the scheduling task cannot be executed, the slave node writes the scheduling task into the task queue component.
10. The method of claim 9, comprising:
the coordinator coordinates with other coordinators to select nodes from the node set,
the coordinator determines the node selected by coordination as the main node,
the coordinator determines the remaining nodes as the slave nodes.
11. The method of claim 10, comprising:
when the main node fails, the coordinator and other coordinators reconcile the selected node from the node set;
the coordinator determines the reselected node as the master node;
the coordinator determines the remaining nodes as the slave nodes.
12. The method according to any one of claims 8-11, comprising:
the configuration center component receives the operation configuration data written by any node in the node set in advance;
when the configuration center component successfully writes the operation configuration data, storing the operation configuration data and feeding back a writing success message;
and the configuration center component feeds back a write-in failure message when the writing of the operation configuration data fails.
13. The method according to any one of claims 8-11, comprising:
the load balancer receives a request of configuring operation data of a user in advance;
responding to the request of the configuration operation data by the load balancer, and randomly selecting any node in the node set;
and the load balancer writes the operation configuration data into the configuration center component from the selected node.
14. A scheduling system, comprising:
a memory for storing a program;
a processor for executing a program stored by the memory, the program causing the processor to perform the method of any of claims 8-13.
15. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 8-13.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710942824.4A CN109656690A (en) | 2017-10-11 | 2017-10-11 | Scheduling system, method and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710942824.4A CN109656690A (en) | 2017-10-11 | 2017-10-11 | Scheduling system, method and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109656690A true CN109656690A (en) | 2019-04-19 |
Family
ID=66109605
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710942824.4A Pending CN109656690A (en) | 2017-10-11 | 2017-10-11 | Scheduling system, method and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109656690A (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110427253A (en) * | 2019-07-04 | 2019-11-08 | 中国建设银行股份有限公司 | Robot resource tasks period management-control method and device |
| CN110597608A (en) * | 2019-09-12 | 2019-12-20 | 阿里巴巴集团控股有限公司 | Task processing method and device, distributed system and storage medium |
| CN111913793A (en) * | 2020-07-31 | 2020-11-10 | 同盾控股有限公司 | Distributed task scheduling method, device, node equipment and system |
| CN112311832A (en) * | 2019-07-31 | 2021-02-02 | 杭州萤石软件有限公司 | Task processing method, system and node |
| CN113239038A (en) * | 2021-05-10 | 2021-08-10 | 北京沃东天骏信息技术有限公司 | Data processing method and device |
| CN113254164A (en) * | 2021-07-07 | 2021-08-13 | 北京航天驭星科技有限公司 | Monitoring method and system for task execution state in satellite measurement and control process and measurement and control system |
| CN113419836A (en) * | 2021-07-07 | 2021-09-21 | 深圳市珍爱捷云信息技术有限公司 | Task processing method and device, electronic equipment and computer readable storage medium |
| CN114629806A (en) * | 2022-04-13 | 2022-06-14 | 腾讯科技(成都)有限公司 | Data processing method, data processing apparatus, electronic device, storage medium, and program product |
| CN114625489A (en) * | 2022-03-24 | 2022-06-14 | 阿里巴巴（中国）有限公司 | Access request response method and device and distributed system |
| CN114721831A (en) * | 2022-04-21 | 2022-07-08 | 湖南大学 | Method and system for optimizing real-time performance of Citcoms system |
| CN115840631A (en) * | 2023-01-04 | 2023-03-24 | 中科金瑞(北京)大数据科技有限公司 | RAFT-based high-availability distributed task scheduling method and equipment |
| WO2024020743A1 (en) * | 2022-07-25 | 2024-02-01 | 苏州中科天启遥感科技有限公司 | Master-slave cluster task scheduling method for data production, and application thereof |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5361362A (en) * | 1989-02-24 | 1994-11-01 | At&T Bell Laboratories | Adaptive job scheduling for multiprocessing systems with master and slave processors executing tasks with opposite anticipated execution times respectively |
| CN101022387A (en) * | 2007-03-28 | 2007-08-22 | 中兴通讯股份有限公司 | Communication band width dynamic distributing method in chain type group network |
| CN102521044A (en) * | 2011-12-30 | 2012-06-27 | 北京拓明科技有限公司 | Distributed task scheduling method and system based on messaging middleware |
| CN103647834A (en) * | 2013-12-16 | 2014-03-19 | 上海证券交易所 | System and method used for processing multi-phase distributed task scheduling |
| US20140109100A1 (en) * | 2011-06-28 | 2014-04-17 | Fujitsu Limited | Scheduling method and system |
| CN105447097A (en) * | 2015-11-10 | 2016-03-30 | 北京北信源软件股份有限公司 | Data acquisition method and system |
-
2017
- 2017-10-11 CN CN201710942824.4A patent/CN109656690A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5361362A (en) * | 1989-02-24 | 1994-11-01 | At&T Bell Laboratories | Adaptive job scheduling for multiprocessing systems with master and slave processors executing tasks with opposite anticipated execution times respectively |
| CN101022387A (en) * | 2007-03-28 | 2007-08-22 | 中兴通讯股份有限公司 | Communication band width dynamic distributing method in chain type group network |
| US20140109100A1 (en) * | 2011-06-28 | 2014-04-17 | Fujitsu Limited | Scheduling method and system |
| CN102521044A (en) * | 2011-12-30 | 2012-06-27 | 北京拓明科技有限公司 | Distributed task scheduling method and system based on messaging middleware |
| CN103647834A (en) * | 2013-12-16 | 2014-03-19 | 上海证券交易所 | System and method used for processing multi-phase distributed task scheduling |
| CN105447097A (en) * | 2015-11-10 | 2016-03-30 | 北京北信源软件股份有限公司 | Data acquisition method and system |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110427253A (en) * | 2019-07-04 | 2019-11-08 | 中国建设银行股份有限公司 | Robot resource tasks period management-control method and device |
| CN112311832B (en) * | 2019-07-31 | 2022-08-05 | 杭州萤石软件有限公司 | Task processing method, system and node |
| CN112311832A (en) * | 2019-07-31 | 2021-02-02 | 杭州萤石软件有限公司 | Task processing method, system and node |
| CN110597608A (en) * | 2019-09-12 | 2019-12-20 | 阿里巴巴集团控股有限公司 | Task processing method and device, distributed system and storage medium |
| CN110597608B (en) * | 2019-09-12 | 2023-08-22 | 创新先进技术有限公司 | Task processing method and device, distributed system and storage medium |
| CN111913793A (en) * | 2020-07-31 | 2020-11-10 | 同盾控股有限公司 | Distributed task scheduling method, device, node equipment and system |
| CN113239038A (en) * | 2021-05-10 | 2021-08-10 | 北京沃东天骏信息技术有限公司 | Data processing method and device |
| CN113254164A (en) * | 2021-07-07 | 2021-08-13 | 北京航天驭星科技有限公司 | Monitoring method and system for task execution state in satellite measurement and control process and measurement and control system |
| CN113254164B (en) * | 2021-07-07 | 2021-10-08 | 北京航天驭星科技有限公司 | Monitoring method and system for task execution state in satellite measurement and control process and measurement and control system |
| CN113419836B (en) * | 2021-07-07 | 2023-02-21 | 深圳市珍爱捷云信息技术有限公司 | Task processing method and device, electronic equipment and computer readable storage medium |
| CN113419836A (en) * | 2021-07-07 | 2021-09-21 | 深圳市珍爱捷云信息技术有限公司 | Task processing method and device, electronic equipment and computer readable storage medium |
| CN114625489A (en) * | 2022-03-24 | 2022-06-14 | 阿里巴巴（中国）有限公司 | Access request response method and device and distributed system |
| CN114625489B (en) * | 2022-03-24 | 2024-07-30 | 阿里巴巴（中国）有限公司 | Access request response method, device and distributed system |
| CN114629806A (en) * | 2022-04-13 | 2022-06-14 | 腾讯科技(成都)有限公司 | Data processing method, data processing apparatus, electronic device, storage medium, and program product |
| CN114629806B (en) * | 2022-04-13 | 2023-12-12 | 腾讯科技(成都)有限公司 | Data processing method, device, electronic equipment, storage medium and program product |
| CN114721831A (en) * | 2022-04-21 | 2022-07-08 | 湖南大学 | Method and system for optimizing real-time performance of Citcoms system |
| CN114721831B (en) * | 2022-04-21 | 2024-09-13 | 湖南大学 | Method and system for optimizing CitcomS system real-time performance |
| WO2024020743A1 (en) * | 2022-07-25 | 2024-02-01 | 苏州中科天启遥感科技有限公司 | Master-slave cluster task scheduling method for data production, and application thereof |
| CN115840631A (en) * | 2023-01-04 | 2023-03-24 | 中科金瑞(北京)大数据科技有限公司 | RAFT-based high-availability distributed task scheduling method and equipment |
| CN115840631B (en) * | 2023-01-04 | 2023-05-16 | 中科金瑞(北京)大数据科技有限公司 | RAFT-based high-availability distributed task scheduling method and equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109656690A (en) | Scheduling system, method and storage medium | |
| CN110290180B (en) | Distributed task scheduling method and device, computer equipment and storage medium | |
| CN109327509B (en) | Low-coupling distributed streaming computing system of master/slave architecture | |
| CN106844198B (en) | Distributed dispatching automation test platform and method | |
| US8700931B2 (en) | Method and system for managing power of a mobile device | |
| US8301935B2 (en) | Distributed batch runner | |
| CN112333249B (en) | Business service system and method | |
| CN106817408B (en) | Distributed server cluster scheduling method and device | |
| CN112114950A (en) | Task scheduling method and device and cluster management system | |
| CN109766172B (en) | Asynchronous task scheduling method and device | |
| CN106603689B (en) | Data processing method and device based on distributed publish-subscribe message system | |
| WO2022267646A1 (en) | Pod deployment method and apparatus | |
| CN111597033A (en) | Task scheduling method and device | |
| CN106060189B (en) | A kind of distribution domain name registration system | |
| CN117608825A (en) | Resource management method based on multi-cloud management platform and related equipment | |
| CN113032125A (en) | Job scheduling method, device, computer system and computer-readable storage medium | |
| CN105245408A (en) | Synchronous data monitoring method and device | |
| CN106293911A (en) | Dispatching System, method | |
| CN105373563B (en) | Database switching method and device | |
| CN115576684A (en) | Task processing method and device, electronic equipment and storage medium | |
| CN107294911B (en) | Data packet monitoring method and device, remote procedure call system and equipment | |
| CN105338037A (en) | Dynamic scheduling method and system | |
| CN112711522B (en) | Cloud testing method and system based on docker and electronic equipment | |
| CN112448977A (en) | System, method, apparatus and computer readable medium for assigning tasks | |
| CN115023929A (en) | Data synchronization method, device, system, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190419 |
|
| RJ01 | Rejection of invention patent application after publication |