CN110247954A - A kind of dispatching method and system of distributed task scheduling - Google Patents
A kind of dispatching method and system of distributed task scheduling Download PDFInfo
- Publication number
- CN110247954A CN110247954A CN201910409265.XA CN201910409265A CN110247954A CN 110247954 A CN110247954 A CN 110247954A CN 201910409265 A CN201910409265 A CN 201910409265A CN 110247954 A CN110247954 A CN 110247954A
- Authority
- CN
- China
- Prior art keywords
- task
- node
- zookeeper
- master node
- application server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0893—Assignment of logical groups to network elements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
- H04L41/5041—Network service management, e.g. ensuring proper service fulfilment according to agreements characterised by the time relationship between creation and deployment of a service
- H04L41/5054—Automatic deployment of services triggered by the service manager, e.g. service implementation by automatic configuration of network components
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1031—Controlling of the operation of servers by a load balancer, e.g. adding or removing servers that serve requests
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1034—Reaction to server failures by a load balancer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/104—Peer-to-peer [P2P] networks
- H04L67/1044—Group management mechanisms
- H04L67/1051—Group master selection mechanisms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/51—Discovery or management thereof, e.g. service location protocol [SLP] or web services
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Theoretical Computer Science (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a kind of dispatching method of distributed task scheduling and systems, this method comprises: electing Master node in the cluster of application server by Zookeeper；It will be under the respective directories node in being collected into for task write-in Zookeeper specified task list；Master node starts timed task, reads the task under directory node, obtains task list；Master node starts thread pool, the task in asynchronous allocation task list；Each node of the task into application server cluster after disposing asynchronous allocation by load-balanced server equilibrium is executed.The present invention uses the clustered deploy(ment) framework of load-balanced server, application server and Zookeeper, realize full-range High Availabitity, not only realize the monitoring of the real time health state of real-time task, and ensure that simultaneously integrity service the design of global High Availabitity and can horizontal scalable appearance, while functions such as flexible configuration for also achieving monitor task.
Description
Technical field
The present invention relates to task scheduling technique field, in particular to the dispatching method and system of a kind of distributed task scheduling.
Background technique
With the rapid development of Internet technology, also blowout increases the data of generation therewith, more and more business fields
Scape begins to focus on the real-time of data.Therefore, start to show up prominently based on the business game that data calculate in real time, more and more
Put on line and use, and played an important role in internet area, for example, real-time recommendation, double 11 real-time large-size screen monitors statistics,
Anti- fraud etc. in real time.
Since real-time calculating task usually requires 7*24 operation free of discontinuities, it is directed to the real time health state of real-time task
Monitoring it is particularly important.Simultaneously as the real-time task on line is continuously increased, therefore monitor task dispatches needing support property of system
It can horizontal extension.
Such as Publication No. CN106487597A, a kind of entitled service monitoring system and side based on Zookeeper
The patent of invention of method, the patent use zookeeper as primary clustering, and main function is the performance indicator of capturing service service
The operation information of data and server, while docking real-time Risk Calculation center and alarm platform is realized to running on line
The monitoring of business service.But the global High Availabitity for not solving integrity service designs and can horizontal scalable appearance and monitoring
The problems such as flexible configuration of task.
In conclusion needing to propose the dispatching method and system of a kind of new distributed task scheduling, to solve the above problems.
Summary of the invention
In order to solve problems in the prior art, the embodiment of the invention provides a kind of dispatching method of distributed task scheduling and it is
System, not only realizes the monitoring of the real time health state of real-time task, and ensure that the global High Availabitity of integrity service simultaneously
Design and can horizontal scalable appearance, while functions such as flexible configuration for also achieving monitor task.
To solve said one or multiple technical problems, the technical solution adopted by the present invention is that:
On the one hand, a kind of dispatching method of distributed task scheduling is provided, this method comprises the following steps:
Master node is elected in the cluster of application server by Zookeeper；
Being collected into for task is written under the respective directories node in the specified task list of the Zookeeper；
The Master node starts timed task, reads the task under the directory node, obtains task list；
The Master node starts thread pool, the task in task list described in asynchronous allocation；
The task after disposing asynchronous allocation by load-balanced server equilibrium is into the application server cluster
Each node executed.
Further, the Master node starts thread pool, and the task in task list described in asynchronous allocation includes:
It loses in the thread pool and executes after task in the task list is packaged as sub thread.
Further, the Master node starts thread pool, the task in task list described in asynchronous allocation further include:
The service logic of the task is packaged into http request and is sent to the load-balanced server by the sub thread.
Further, before the Master node starting timed task, the method also includes:
The relevant operation in the specified task list of the Zookeeper is monitored, obtains the more new content of task in real time, and
Update the task list.
Further, the method also includes:
When the Master node delay machine, Zookeeper is re-elected out newly in the cluster of the application server
Master node execute relevant operation.
Further, the method also includes:
The node in the cluster of the application server is increased or decreased by redeploying the application server realizing
Quantity.
On the other hand, a kind of scheduling system of distributed task scheduling is provided, the system comprises:
Host node elects module, for electing Master node in the cluster of application server by Zookeeper；
Task collection module, for being collected into for task to be written to the phase in the specified task list of the Zookeeper
It answers under directory node；
List obtains module, starts timed task for the Master node, reads described under the directory node
Task obtains task list；
Task allocating module starts thread pool for the Master node, appointing in task list described in asynchronous allocation
Business；
Balanced deployment module, for disposing the task after asynchronous allocation by load-balanced server equilibrium to described
Each node in application server cluster is executed.
Further, the task allocating module includes:
Sub thread encapsulation unit, for being lost after being packaged the task in the task list as sub thread into described
It is executed in thread pool.
Further, the task allocating module further include:
Encapsulation unit is requested, the service logic of the task is packaged into http request for the sub thread and is sent to institute
State load-balanced server.
Further, the system also includes:
Catalogue update module obtains in real time for monitoring the relevant operation in the specified task list of the Zookeeper
The more new content of task, and update the task list.
Further, the system also includes:
Module is re-elected, collection of the Zookeeper in the application server when the Master node delay machine is used for
New Master node is re-elected out in group executes relevant operation.
Further, the system also includes:
Scalable molar block, for increasing or decreasing the application server by redeploying the application server realizing
Cluster in node quantity.
Technical solution provided in an embodiment of the present invention has the benefit that
1, the dispatching method and system of distributed task scheduling provided in an embodiment of the present invention solves and appoints including monitoring operation etc.
In scheduling process of being engaged in, the distributed lock and cluster multinode principal and subordinate's election issues of multi-node cluster timed task, and be not limited to
The business processing scene of processing monitoring operation, is equally applicable to other task schedule scenes without dependence operation；
2, the dispatching method and system of distributed task scheduling provided in an embodiment of the present invention is based on Zookeeper frame nature
Reasonable utilization, realize the flexible configuration of task, registration, cancellation, alarm content item, alarm threshold value comprising task and appoint
The functions such as the real-time dynamic adjustment for dispatching cycle of being engaged in；
3, the dispatching method and system of distributed task scheduling provided in an embodiment of the present invention realizes point of real-time calculating task
The scheme of cloth business scenario, the overall situation are designed using HA High Availabitity characteristic, for the growth for monitoring the business scales such as operation or are subtracted
It is few, dynamic may be implemented and do not shut down scalable appearance.
Detailed description of the invention
To describe the technical solutions in the embodiments of the present invention more clearly, make required in being described below to embodiment
Attached drawing is briefly described, it should be apparent that, drawings in the following description are only some embodiments of the invention, for
For those of ordinary skill in the art, without creative efforts, it can also be obtained according to these attached drawings other
Attached drawing.
Fig. 1 is the flow chart of the dispatching method of distributed task scheduling shown according to an exemplary embodiment；
Fig. 2 is the structural schematic diagram of the scheduling system of distributed task scheduling shown according to an exemplary embodiment；
Fig. 3 be the middleware that the scheduling system of distributed task scheduling shown according to an exemplary embodiment specifically uses and
The schematic diagram of deployment architecture.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, below in conjunction with attached in the embodiment of the present invention
Figure, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is only this
Invention a part of the embodiment, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art exist
Every other embodiment obtained under the premise of creative work is not made, shall fall within the protection scope of the present invention.
The main application scenarios of the embodiment of the present invention are that the stream calculation operation run in convection current computing platform is monitored in real time.
Specifically use monitor mode for using timed task (such as 1 minute fixed time interval period), the operating status of timing acquiring task
Collected parameter is carried out being persisted to DB server, and is configured simultaneously according to business monitoring by parameter, specified to business personnel
The abnormal index for needing to alarm carries out Realtime Alerts.The schedule job of the embodiment of the present invention is not limited to above-mentioned monitoring operation and patrols
Volume, other business scenarios equally can be laterally generalized to, here only using monitoring business as description distributed scheduling of the invention
The special case of method and system is illustrated.
Fig. 1 is the flow chart of the dispatching method of distributed task scheduling shown according to an exemplary embodiment, referring to Fig.1 institute
Show, this method comprises the following steps:
Master node is elected in the cluster of application server by Zookeeper.
Specifically, used Zookeeper cluster as distributed type assemblies dispatch coordination service in the embodiment of the present invention, and
The Master node for providing application server cluster elects service, while the distributed lock for also solving cluster timed task is asked
Topic.In addition, Zookeeper cluster equally plays storage job configuration information and monitors the registration center role of operation.
The scale of Zookeeper cluster usually takes the deployment mode of 2N+1 node.
What needs to be explained here is that when the application server in the embodiment of the present invention starts, it is necessary first to be initialized
Operation.The content mainly initialized is the interaction with Zookeeper cluster, and all nodes in application server cluster require
Connection is established with the same Zookeeper cluster, and creates client instance.Then the node of all application servers need to make
It uses and specifies the same directory path as parameter, create Leader Latch example on Zookeeper respectively, and exist simultaneously
LeaderLatchListener monitor is added in Leader Latch example.
Being collected into for task is written under the respective directories node in the specified task list of the Zookeeper.
Specifically, in the embodiment of the present invention, task registration can be set and submit center to carry out the registration of task, update,
The operation such as deletion.When it is implemented, can be by monitoring the task in the inter-related task catalogue that task registration submits center to submit
The operation such as creation or registration, to collect whole mission bit streams, and the whole tasks that will acquire are written to what Zookeeper was specified
Under respective directories node in task list.
The Master node reads the task under the directory node, obtains task list.
Specifically, after some node in Zookeeper election application server cluster is as Master node,
Master node starts timed task, such as triggers the isLeader method of monitor.At this point, Master node can be
The specified ZK catalogue for monitoring job logging information is begun listening in isLeader method, and (i.e. Zookeeper is specified to be appointed
Business catalogue), to collect the whole mission bit streams for needing to monitor.
The Master node starts thread task pool, the task in task list described in asynchronous allocation.
The task after disposing asynchronous allocation by load-balanced server equilibrium is into the application server cluster
Each node executed.
Specifically, load-balanced server in the embodiment of the present invention uses Nginx, Nginx is by all task balances
The cluster for being distributed to application server each child node processing.
As a kind of preferably embodiment, in the embodiment of the present invention, the Master node starts thread pool, and asynchronous point
Include: with the task in the task list
It loses in the thread pool and executes after task in the task list is packaged as sub thread.
Specifically, will start thread pool after Master node obtains task list, using the task in task list as son
Thread is lost in thread pool after being packaged to be executed.What needs to be explained here is that the thread pool in the embodiment of the present invention uses
ScheduledExecutorService。
As a kind of preferably embodiment, in the embodiment of the present invention, the Master node starts thread pool, and asynchronous point
With the task in the task list further include:
The service logic of the task is packaged into http request and is sent to the load-balanced server by the sub thread.
Specifically, task schedule sub thread configures (i.e. service logic) when handling specific service logic, by job parameter
It is packaged into http request, then load balancing is carried out by Nginx service (i.e. load-balanced server), equilibrium assignment takes to application
Each node being engaged in the cluster of device realizes asynchronous process.It, can also be with if the process performance of the node of each application server is different
Other allocation strategies such as distribution are weighted by adjusting the configuration of load-balanced server.
As a kind of preferably embodiment, in the embodiment of the present invention, before the Master node starting timed task, institute
State method further include:
The relevant operation in the specified task list of the Zookeeper is monitored, obtains the more new content of task in real time, and
Update the task list.
Specifically, the relevant operation in task list includes the operations such as creation, update, deletion in the embodiment of the present invention.Prison
The relevant operation in task list for listening Zookeeper specified, the root content information including collecting including task registration etc. are right
Task list is updated.
Below by taking the scheduling of monitor task as an example, the scheme for collecting task registration information is illustrated.Setting is unified first
Mission Monitor path, such as can be set Mission Monitor path be /xxx/jobs, then start a monitor monitor the mesh
Record, for finding/the operation such as creation, update, the deletion of subdirectory of xxx/jobs catalogue.When task submits center to have new task
When (such as job1) is registered, can zookeeper /xxx/jobs catalogue under creation/xxx/jobs/job1, and register corresponding
Monitor control index item.When the monitor discovery of monitoring and dispatching system has newly created subdirectory (/xxx/jobs/job1), it is meant that
There is new operation to joined monitoring queue, monitor can trigger corresponding monitoring movement, can read newly created subdirectory here
Nodal information, pass through parsing node content (Json character string), obtain the monitoring configuration information of corresponding operation.Simultaneously according to work
Industry configuration information creates a specific business processing thread.The thread is put into thread pool simultaneously, with the timing week of configuration
Phase timer-triggered scheduler executes.
When submission center have task (such as job1) it is offline when, zookeeper/xxx/jobs/job1 subdirectory can be deleted
It removes.The monitor discovery (/xxx/jobs/job1) of task scheduling system disappears at this time, shows that job1 does not need to monitor already.
The snoop-operations of monitor triggering at this time are to kill timer-triggered scheduler task.
If system needs in the process of running, the specific task timer-triggered scheduler period to be customized again, at this point it is possible to pass through
More new node/xxx/jobs/jobN configuration information.When the value value that Master node listens to the node updates, meeting
Kill falls original timer-triggered scheduler task, and timed task is restarted in the configuration after reusing change.
As a kind of preferably embodiment, in the embodiment of the present invention, the method also includes:
When the Master node delay machine, Zookeeper is re-elected out newly in the cluster of the application server
Master node execute relevant operation.
Specifically, the characteristic of the High Availabitity of the dispatching method in order to realize distributed task scheduling, in the embodiment of the present invention, when answering
When with Master node delay machine in the cluster of server, Zookeeper, which can be triggered, re-elects the operation of Master node (such as
Triggered using the LeaderLatch mechanism of Zookeeper), after new Mater node is elected, it can re-start initial
Change operation, refreshes the task list for needing to dispatch, the scheduling flow before reentering.What needs to be explained here is that when old
When Master node restores, it can default and schedule job distribution is participated in the identity of slave node.
When Slave node delay machine (such as service processes are lost or network is obstructed) in the cluster of application server, then
The integrated scheduling of task is not influenced, can only reduce the concurrency performance of system.Slave node serve restarts or when network recovery,
It can state before seamless recovery.When ensure that arbitrary node exception in application server above, the High Availabitity of integrity service.
It is further to note that in the embodiment of the present invention, when certain node state exception in Zookeeper cluster
(including leader node or other follower nodes), the ZAB agreement being had by oneself by Zookeeper ensure that the weight of leader
The consistency of new election and clustered node data.Therefore, the node of Zookeeper cluster also meets the feature of High Availabitity.
As a kind of preferably embodiment, in the embodiment of the present invention, the method also includes:
The node in the cluster of the application server is increased or decreased by redeploying the application server realizing
Quantity.
Specifically, can be realized by simply modifying the configuration file of load-balanced server in the embodiment of the present invention
The dilatation of application server cluster and capacity reducing.It can also be made built in load-balanced server by modification configuration file simultaneously
Load balancing is adjusted.
For example, when business game quantity (i.e. task quantity) dramatically increases, if application server cluster process performance
Reach bottleneck, horizontal extension can be carried out by newly-increased application server node.Simplified deployment application service is only needed at this time
The configuration of device and modification load-balanced server.It equally, can also be direct when business game quantity (i.e. task quantity) is less
The mode of offline application server node realizes horizontal capacity reducing.In the embodiment of the present invention, 1 application server node of minimum support
Operation, both having served as the progress task schedule distribution of Master node can also be used as the processing that Work node carries out service logic.This
In it should be noted that if the process performance of each application server node is different, can also be by adjusting load balancing service
The configuration of device is weighted other allocation strategies such as distribution.
Fig. 2 is the structural schematic diagram of the scheduling system of distributed task scheduling shown according to an exemplary embodiment, referring to Fig. 2
Shown, which includes:
Host node elects module, for electing Master node in the cluster of application server by Zookeeper；
Task collection module, for being collected into for task to be written to the phase in the specified task list of the Zookeeper
It answers under directory node；
List obtains module, starts timed task for the Master node, reads described under the directory node
Task obtains task list；
Task allocating module starts thread pool for the Master node, appointing in task list described in asynchronous allocation
Business；
Balanced deployment module, for disposing the task after asynchronous allocation by load-balanced server equilibrium to described
Each node in application server cluster is executed.
Specifically, middleware and deployment architecture that the distributed task dispatching system in the embodiment of the present invention uses are as schemed
Shown in 3, wherein the application server for being responsible for processing business logic includes but is not limited to the Jboss in legend, other similar function
Web container be also able to satisfy demand, such as Tomcat etc..The machine quantity of Jboss cluster is determined by actual portfolio, needs
Meet the service logic that all operations can have been handled in the timed task period.Distributed type assemblies dispatch coordination service uses
Zookeeper cluster provides the Master node election service of application server cluster, while also solving cluster timed task
Distributed lock the problem of.In addition, Zookeeper cluster equally plays storage job configuration information and monitors the note of operation
Volume center role.Load-balanced server uses Nginx, by each sub- section for being distributed to jboss cluster of all task balances
Point processing, the i.e. node of load-balanced server are responsible for the load balancing of application server cluster.It is equal that load can equally be set up
Weigh server cluster, realizes High Availabitity using the mode of VIP+Nginx cluster.What needs to be explained here is that the embodiment of the present invention
In, center can also be submitted by one task registration of setting to carry out the registration of task, update, the operation such as deletion.
As a kind of preferably embodiment, in the embodiment of the present invention, the task allocating module includes:
Sub thread encapsulation unit, for being lost after being packaged the task in the task list as sub thread into described
It is executed in thread pool.
As a kind of preferably embodiment, in the embodiment of the present invention, the task allocating module further include:
Encapsulation unit is requested, the service logic of the task is packaged into http request for the sub thread and is sent to institute
State load-balanced server.
As a kind of preferably embodiment, in the embodiment of the present invention, the system also includes:
Catalogue update module obtains in real time for monitoring the relevant operation in the specified task list of the Zookeeper
The more new content of task, and update the task list.
As a kind of preferably embodiment, in the embodiment of the present invention, the system also includes:
Module is re-elected, collection of the Zookeeper in the application server when the Master node delay machine is used for
New Master node is re-elected out in group executes relevant operation.
As a kind of preferably embodiment, in the embodiment of the present invention, the system also includes:
Scalable molar block, for increasing or decreasing the application server by redeploying the application server realizing
Cluster in node quantity.
In conclusion technical solution provided in an embodiment of the present invention has the benefit that
1, the dispatching method and system of distributed task scheduling provided in an embodiment of the present invention solves and appoints including monitoring operation etc.
In scheduling process of being engaged in, the distributed lock and cluster multinode principal and subordinate's election issues of multi-node cluster timed task, and be not limited to
The business processing scene of processing monitoring operation, is equally applicable to other task schedule scenes without dependence operation；
2, the dispatching method and system of distributed task scheduling provided in an embodiment of the present invention is based on Zookeeper frame nature
Reasonable utilization, realize the flexible configuration of task, registration, cancellation, alarm content item, alarm threshold value comprising task and appoint
The functions such as the real-time dynamic adjustment for dispatching cycle of being engaged in；
3, the dispatching method and system of distributed task scheduling provided in an embodiment of the present invention realizes point of real-time calculating task
The scheme of cloth business scenario, the overall situation are designed using HA High Availabitity characteristic, for the growth for monitoring the business scales such as operation or are subtracted
It is few, dynamic may be implemented and do not shut down scalable appearance.
It should be understood that the scheduling system of distributed task scheduling provided by the above embodiment is in triggering task schedule business
When, only the example of the division of the above functional modules, in practical application, it can according to need and divide above-mentioned function
With being completed by different functional modules, i.e., the internal structure of system is divided into different functional modules, to complete above description
All or part of function.In addition, the scheduling system of distributed task scheduling provided by the above embodiment and the tune of distributed task scheduling
Degree embodiment of the method belongs to same design, i.e. the system is the dispatching method based on the distributed task scheduling, is implemented
Journey is detailed in embodiment of the method, and which is not described herein again.
The foregoing is merely presently preferred embodiments of the present invention, is not intended to limit the invention, it is all in spirit of the invention and
Within principle, any modification, equivalent replacement, improvement and so on be should all be included in the protection scope of the present invention.
Claims (12)
1. a kind of dispatching method of distributed task scheduling, which is characterized in that described method includes following steps:
Master node is elected in the cluster of application server by Zookeeper；
Being collected into for task is written under the respective directories node in the specified task list of the Zookeeper；It is described
Master node starts timed task, reads the task under the directory node, obtains task list；
The Master node starts thread pool, the task in task list described in asynchronous allocation；
The task after disposing asynchronous allocation by load-balanced server equilibrium is each into the application server cluster
Node is executed.
2. the dispatching method of distributed task scheduling according to claim 1, which is characterized in that the Master node starts line
Cheng Chi, the task in task list described in asynchronous allocation include:
It loses in the thread pool and executes after task in the task list is packaged as sub thread.
3. the dispatching method of distributed task scheduling according to claim 2, which is characterized in that the Master node starts line
Cheng Chi, the task in task list described in asynchronous allocation further include:
The service logic of the task is packaged into http request and is sent to the load-balanced server by the sub thread.
4. the dispatching method of distributed task scheduling according to any one of claims 1 to 3, which is characterized in that the Master section
Before point starting timed task, the method also includes:
The relevant operation in the specified task list of the Zookeeper is monitored, obtains the more new content of task in real time, and is updated
The task list.
5. the dispatching method of distributed task scheduling according to any one of claims 1 to 3, which is characterized in that the method is also wrapped
It includes:
When the Master node delay machine, Zookeeper re-elects out new in the cluster of the application server
Master node executes relevant operation.
6. the dispatching method of distributed task scheduling according to any one of claims 1 to 3, which is characterized in that the method is also wrapped
It includes:
The number of the node in the cluster of the application server is increased or decreased by redeploying the application server realizing
Amount.
7. a kind of scheduling system of distributed task scheduling, which is characterized in that the system comprises:
Host node elects module, for electing Master node in the cluster of application server by Zookeeper；
Task collection module, for being collected into for task to be written to the corresponding mesh in the specified task list of the Zookeeper
It records under node；
List obtains module, starts timed task for the Master node, reads the task under the directory node,
Obtain task list；
Task allocating module starts thread pool for the Master node, the task in task list described in asynchronous allocation；
Balanced deployment module, for passing through the task after load-balanced server equilibrium deployment asynchronous allocation to the application
Each node in server cluster is executed.
8. the scheduling system of distributed task scheduling according to claim 7, which is characterized in that the task allocating module packet
It includes:
Sub thread encapsulation unit, for losing after being packaged the task in the task list as sub thread into the thread
It is executed in pond.
9. the scheduling system of distributed task scheduling according to claim 8, which is characterized in that the task allocating module also wraps
It includes:
Encapsulation unit is requested, the service logic of the task is packaged into http request for the sub thread and is sent to described bear
Carry equalization server.
10. the scheduling system of the distributed task scheduling according to claim 7 to 9, which is characterized in that the system also includes:
Catalogue update module obtains task for monitoring the relevant operation in the specified task list of the Zookeeper in real time
More new content, and update the task list.
11. the scheduling system of the distributed task scheduling according to claim 7 to 9, which is characterized in that the system also includes:
Module is re-elected, for when the Master node delay machine, Zookeeper to be in the cluster of the application server
It re-elects out new Master node and executes relevant operation.
12. the scheduling system of the distributed task scheduling according to claim 7 to 9, which is characterized in that the system also includes:
Scalable molar block, for increasing or decreasing the collection of the application server by redeploying the application server realizing
The quantity of node in group.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910409265.XA CN110247954A (en) | 2019-05-15 | 2019-05-15 | A kind of dispatching method and system of distributed task scheduling |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910409265.XA CN110247954A (en) | 2019-05-15 | 2019-05-15 | A kind of dispatching method and system of distributed task scheduling |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110247954A true CN110247954A (en) | 2019-09-17 |
Family
ID=67884191
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910409265.XA Pending CN110247954A (en) | 2019-05-15 | 2019-05-15 | A kind of dispatching method and system of distributed task scheduling |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110247954A (en) |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110572478A (en) * | 2019-09-30 | 2019-12-13 | 重庆紫光华山智安科技有限公司 | Data transmission method and system based on distributed architecture service and FTP service |
| CN110704185A (en) * | 2019-09-20 | 2020-01-17 | 深圳市递四方信息科技有限公司 | Cluster system and method for scheduling fragment timing tasks of cluster system |
| CN110704172A (en) * | 2019-09-20 | 2020-01-17 | 深圳市递四方信息科技有限公司 | Cluster system timing task scheduling method and cluster system |
| CN110737520A (en) * | 2019-09-20 | 2020-01-31 | 深圳市递四方信息科技有限公司 | Method for scheduling and controlling timing task of cluster system and cluster system |
| CN110750343A (en) * | 2019-09-20 | 2020-02-04 | 深圳市递四方信息科技有限公司 | Cluster system timing task scheduling control method and cluster system |
| CN110888719A (en) * | 2019-09-18 | 2020-03-17 | 广州市巨硅信息科技有限公司 | Distributed task scheduling system and method based on web service |
| CN110928662A (en) * | 2019-11-28 | 2020-03-27 | 国网信息通信产业集团有限公司 | Distributed timing task scheduler facing micro-service architecture |
| CN111092754A (en) * | 2019-11-29 | 2020-05-01 | 贝壳技术有限公司 | Real-time access service system and implementation method thereof |
| CN111124806A (en) * | 2019-11-25 | 2020-05-08 | 山东鲁能软件技术有限公司 | Equipment state real-time monitoring method and system based on distributed scheduling task |
| CN111431999A (en) * | 2020-03-23 | 2020-07-17 | 杭州趣维科技有限公司 | Cloud function distributed system based on Paxos algorithm |
| CN111447097A (en) * | 2020-04-20 | 2020-07-24 | 国网甘肃省电力公司信息通信公司 | Cloud platform resource scheduling management method and system |
| CN111722980A (en) * | 2020-06-11 | 2020-09-29 | 咪咕文化科技有限公司 | Data acquisition system and method |
| CN111813513A (en) * | 2020-06-24 | 2020-10-23 | 中国平安人寿保险股份有限公司 | Real-time task scheduling method, device, equipment and medium based on distribution |
| CN112559159A (en) * | 2021-01-05 | 2021-03-26 | 广州华资软件技术有限公司 | Task scheduling method based on distributed deployment |
| CN113051063A (en) * | 2021-06-02 | 2021-06-29 | 北京轻松筹信息技术有限公司 | Task scheduling method and device for distributed tasks and electronic equipment |
| CN113157426A (en) * | 2020-10-26 | 2021-07-23 | 微医云(杭州)控股有限公司 | Task scheduling method, system, equipment and storage medium |
| CN114124959A (en) * | 2021-12-06 | 2022-03-01 | 天地伟业技术有限公司 | Data processing device of cloud streaming media service and cloud streaming media cluster |
| CN115002124A (en) * | 2022-08-01 | 2022-09-02 | 太极计算机股份有限公司 | Distributed task scheduling processing method, device and equipment |
| CN115098253A (en) * | 2022-06-17 | 2022-09-23 | 特赞（上海）信息科技有限公司 | Subtask execution method and system and electronic equipment |
| WO2023134643A1 (en) * | 2022-01-11 | 2023-07-20 | 中兴通讯股份有限公司 | Streaming data processing method and system, node, electronic device, and storage medium |
| CN116980346A (en) * | 2023-09-22 | 2023-10-31 | 新华三技术有限公司 | Container management method and device based on cloud platform |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104834722A (en) * | 2015-05-12 | 2015-08-12 | 网宿科技股份有限公司 | CDN (Content Delivery Network)-based content management system |
| CN106844018A (en) * | 2015-12-07 | 2017-06-13 | 阿里巴巴集团控股有限公司 | A kind of task processing method, apparatus and system |
| CN107193643A (en) * | 2017-03-28 | 2017-09-22 | 中国科学院信息工程研究所 | A kind of highly reliable distributed traffic real-time statistical method and system |
-
2019
- 2019-05-15 CN CN201910409265.XA patent/CN110247954A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104834722A (en) * | 2015-05-12 | 2015-08-12 | 网宿科技股份有限公司 | CDN (Content Delivery Network)-based content management system |
| CN106844018A (en) * | 2015-12-07 | 2017-06-13 | 阿里巴巴集团控股有限公司 | A kind of task processing method, apparatus and system |
| CN107193643A (en) * | 2017-03-28 | 2017-09-22 | 中国科学院信息工程研究所 | A kind of highly reliable distributed traffic real-time statistical method and system |
Cited By (33)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110888719A (en) * | 2019-09-18 | 2020-03-17 | 广州市巨硅信息科技有限公司 | Distributed task scheduling system and method based on web service |
| CN110737520B (en) * | 2019-09-20 | 2024-03-12 | 深圳市递四方信息科技有限公司 | Method for dispatching and controlling timing task by cluster system and cluster system |
| CN110750343B (en) * | 2019-09-20 | 2023-12-12 | 深圳市递四方信息科技有限公司 | Cluster system timing task scheduling control method and cluster system |
| CN110704172A (en) * | 2019-09-20 | 2020-01-17 | 深圳市递四方信息科技有限公司 | Cluster system timing task scheduling method and cluster system |
| CN110750343A (en) * | 2019-09-20 | 2020-02-04 | 深圳市递四方信息科技有限公司 | Cluster system timing task scheduling control method and cluster system |
| CN110704172B (en) * | 2019-09-20 | 2024-03-12 | 深圳市递四方信息科技有限公司 | Cluster system timing task scheduling method and cluster system |
| CN110704185A (en) * | 2019-09-20 | 2020-01-17 | 深圳市递四方信息科技有限公司 | Cluster system and method for scheduling fragment timing tasks of cluster system |
| CN110737520A (en) * | 2019-09-20 | 2020-01-31 | 深圳市递四方信息科技有限公司 | Method for scheduling and controlling timing task of cluster system and cluster system |
| CN110704185B (en) * | 2019-09-20 | 2024-03-22 | 深圳市递四方信息科技有限公司 | Cluster system fragmentation timing task scheduling method and cluster system |
| CN110572478A (en) * | 2019-09-30 | 2019-12-13 | 重庆紫光华山智安科技有限公司 | Data transmission method and system based on distributed architecture service and FTP service |
| CN111124806A (en) * | 2019-11-25 | 2020-05-08 | 山东鲁能软件技术有限公司 | Equipment state real-time monitoring method and system based on distributed scheduling task |
| CN111124806B (en) * | 2019-11-25 | 2023-09-05 | 山东鲁软数字科技有限公司 | Method and system for monitoring equipment state in real time based on distributed scheduling task |
| CN110928662A (en) * | 2019-11-28 | 2020-03-27 | 国网信息通信产业集团有限公司 | Distributed timing task scheduler facing micro-service architecture |
| CN111092754A (en) * | 2019-11-29 | 2020-05-01 | 贝壳技术有限公司 | Real-time access service system and implementation method thereof |
| CN111092754B (en) * | 2019-11-29 | 2022-07-29 | 贝壳技术有限公司 | Real-time access service system and implementation method thereof |
| CN111431999A (en) * | 2020-03-23 | 2020-07-17 | 杭州趣维科技有限公司 | Cloud function distributed system based on Paxos algorithm |
| CN111431999B (en) * | 2020-03-23 | 2022-11-25 | 杭州小影创新科技股份有限公司 | Cloud function distributed system based on Paxos algorithm |
| CN111447097A (en) * | 2020-04-20 | 2020-07-24 | 国网甘肃省电力公司信息通信公司 | Cloud platform resource scheduling management method and system |
| CN111722980B (en) * | 2020-06-11 | 2023-10-20 | 咪咕文化科技有限公司 | Data acquisition system and method |
| CN111722980A (en) * | 2020-06-11 | 2020-09-29 | 咪咕文化科技有限公司 | Data acquisition system and method |
| CN111813513B (en) * | 2020-06-24 | 2024-05-14 | 中国平安人寿保险股份有限公司 | Method, device, equipment and medium for scheduling real-time tasks based on distribution |
| CN111813513A (en) * | 2020-06-24 | 2020-10-23 | 中国平安人寿保险股份有限公司 | Real-time task scheduling method, device, equipment and medium based on distribution |
| CN113157426B (en) * | 2020-10-26 | 2024-04-02 | 微医云(杭州)控股有限公司 | Task scheduling method, system, equipment and storage medium |
| CN113157426A (en) * | 2020-10-26 | 2021-07-23 | 微医云(杭州)控股有限公司 | Task scheduling method, system, equipment and storage medium |
| CN112559159A (en) * | 2021-01-05 | 2021-03-26 | 广州华资软件技术有限公司 | Task scheduling method based on distributed deployment |
| CN113051063B (en) * | 2021-06-02 | 2021-09-17 | 北京轻松筹信息技术有限公司 | Task scheduling method and device for distributed tasks and electronic equipment |
| CN113051063A (en) * | 2021-06-02 | 2021-06-29 | 北京轻松筹信息技术有限公司 | Task scheduling method and device for distributed tasks and electronic equipment |
| CN114124959A (en) * | 2021-12-06 | 2022-03-01 | 天地伟业技术有限公司 | Data processing device of cloud streaming media service and cloud streaming media cluster |
| WO2023134643A1 (en) * | 2022-01-11 | 2023-07-20 | 中兴通讯股份有限公司 | Streaming data processing method and system, node, electronic device, and storage medium |
| CN115098253A (en) * | 2022-06-17 | 2022-09-23 | 特赞（上海）信息科技有限公司 | Subtask execution method and system and electronic equipment |
| CN115002124A (en) * | 2022-08-01 | 2022-09-02 | 太极计算机股份有限公司 | Distributed task scheduling processing method, device and equipment |
| CN116980346B (en) * | 2023-09-22 | 2023-11-28 | 新华三技术有限公司 | Container management method and device based on cloud platform |
| CN116980346A (en) * | 2023-09-22 | 2023-10-31 | 新华三技术有限公司 | Container management method and device based on cloud platform |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110247954A (en) | A kind of dispatching method and system of distributed task scheduling | |
| CN109327509B (en) | Low-coupling distributed streaming computing system of master/slave architecture | |
| CN103092698B (en) | Cloud computing application automatic deployment system and method | |
| CN106888129A (en) | It is a kind of can elastic telescopic distributed service management system and its method | |
| Kamburugamuve et al. | Survey of distributed stream processing for large stream sources | |
| CN105703940A (en) | Multistage dispatching distributed parallel computing-oriented monitoring system and monitoring method | |
| WO2021190360A1 (en) | Virtualized resource scheduling system and method in vehicle diagnostic cloud platform | |
| US8719780B2 (en) | Application server with a protocol-neutral programming model for developing telecommunications-based applications | |
| CN111813570A (en) | Event-driven message interaction method for power Internet of things | |
| CN109857549A (en) | Image processing method, system, equipment and medium based on load balancing | |
| Nastic et al. | Polaris scheduler: Edge sensitive and slo aware workload scheduling in cloud-edge-iot clusters | |
| JP2015537307A (en) | Component-oriented hybrid cloud operating system architecture and communication method thereof | |
| CN105912401A (en) | Distributed data batch processing system and method | |
| JP2012230667A (en) | High-load business process scalability | |
| Kamburugamuve et al. | Survey of distributed stream processing | |
| CN103152393A (en) | Charging method and charging system for cloud computing | |
| Choi et al. | DART: fast and efficient distributed stream processing framework for internet of things | |
| CN102868736A (en) | Design and implementation method of cloud computing monitoring framework, and cloud computing processing equipment | |
| CN103716397B (en) | A kind of service-oriented simulation clock propulsion method | |
| CN103986748A (en) | Method and device for achieving servitization | |
| CN111984505B (en) | Fortune dimension data acquisition device and acquisition method | |
| CN101146127A (en) | A client buffer update method and device in distributed system | |
| CN110059829A (en) | A kind of asynchronous parameters server efficient parallel framework and method | |
| Schwan et al. | Autoflow: Autonomic information flows for critical information systems | |
| CN109828842A (en) | A kind of high-performance data acquisition engine method based on DPDK technological development |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190917 |
|
| RJ01 | Rejection of invention patent application after publication |