CN105872073A - Design method of distributed timed task system based on etcd cluster - Google Patents
Design method of distributed timed task system based on etcd cluster Download PDFInfo
- Publication number
- CN105872073A CN105872073A CN201610287542.0A CN201610287542A CN105872073A CN 105872073 A CN105872073 A CN 105872073A CN 201610287542 A CN201610287542 A CN 201610287542A CN 105872073 A CN105872073 A CN 105872073A
- Authority
- CN
- China
- Prior art keywords
- server
- configuration file
- timed task
- etcd cluster
- timed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/2866—Architectures; Arrangements
- H04L67/30—Profiles
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
- H04L67/1004—Server selection for load balancing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Hardware Redundancy (AREA)
Abstract
The invention belongs to the field of timed task systems, and particularly relates to a design method of a distributed timed task system based on an etcd cluster. Second servers in an execution service cluster complete a timed task through competition by sharing the same configuration file, configuration files can be updated at any time according to service requirements for different timed tasks, new configuration files directly replace configuration files in the etcd cluster, the complexity of updating all the second servers through the configuration files one by one is avoided, the second servers apply for executing timed tasks to the etcd cluster according to the time points of the timed tasks in the configuration files, when the time conditions of the timed tasks executed by the second servers meet the time points of the timed tasks, the second servers apply for executing the timed tasks, and the application is successful when the etcd cluster detects that the timed tasks are not applied for by other second servers. Thus, the high efficiency of the developed distributed timed task system is ensured.
Description
Technical field
The invention belongs to timed task system regions, particularly to a kind of distributed timing of based on etcd cluster task system
The method for designing of system.
Background technology
Timed task system is a kind of according to the business application system specifying strategy periodically to perform task, is service application
A common ingredient in system, is generally used for processing periodic iterative task, and timed task system is widely used
In the field such as data syn-chronization, data prediction.
Traditional timed task system generally, on a main frame, realizes timed task by instruments such as crontab
Scheduling, but increasing along with timed task quantity, the disposal ability of unit has become as the bottleneck of timed task disposal ability, with
Time, one-of-a-kind system there is also the hidden danger of Single Point of Faliure.
Timed task system being expanded to is a kind of method of feasible improvement in a distributed type assemblies, but due to distribution
Formula cluster implements extremely complex property, to such an extent as to distributed timing task system is difficult to develop efficiently, the distribution designed
Stability and the reliability of formula timed task system are all difficult to ensure that.
Summary of the invention
The present invention is in order to overcome above-mentioned the deficiencies in the prior art, it is provided that a kind of distributed timing based on etcd cluster is appointed
The method for designing of business system, it is ensured that the high efficiency of the distributed timing task system of exploitation.
For achieving the above object, present invention employs techniques below measure:
The method for designing of a kind of distributed timing task system based on etcd cluster, comprises the following steps:
S1, according to business demand prepare timed task configuration file；
Described configuration file is stored in the first server in etcd cluster by S2, timed task configuration service, then by described
The store path of described configuration file is sent to perform the second server in service cluster by etcd cluster；
S3, described second server read configuration file according to described store path from first server, and apply institute
State configuration file；
S4, when needing more new configuration file, directly replace the configuration file in etcd cluster with new configuration file；
S5, described second server monitor the renewal of configuration file；
S6, described second server application perform timed task；
S7, when described second server application is to timed task, start to perform described timed task, and update timing and appoint
The application status of business in etcd cluster, increases the description information of current timed task in new configuration file；Otherwise, repeat
Step S5.
Step S5 specifically includes:
S51, described second server monitor the renewal of configuration file by the Watch API in etcd cluster；
The configuration file configuration self that the application of S52, described second server is new, and described in re invocation, Watch API continues
The continuous change monitoring configuration file.
Step S6 specifically includes:
S61, when second server judges that time conditions meets the time point of timed task, described second server application
Perform timed task；
S62, detected described timed task by checkAndSet API in etcd cluster and the most second taken by other
Business device application；
If the described timed task of S63 is applied, then timed task application failure, otherwise, timing is applied for successfully.
Preferably, described configuration file is the configuration file of the overall situation, and the configuration information of described timed task is maintained in joining
Put in file.
Preferably, the form of described configuration file is yaml form.
Further, described configuration file is stored in the first server in etcd cluster by RESTful interface.
Further, described store path is /crontab/config.yaml path.
Further, described description information include the numbering of described second server, second server place Hostname,
During the startup of the network address of second server, the port numbers of second server, the process number of timed task and timed task
Between.
Further, described first server is set to multiple, two-way each other between two described first servers
Letter connects.
Further, each described first server can corresponding multiple described second servers.
The beneficial effects of the present invention is:
1), the second server performed in service cluster of the present invention is by shared same configuration file, employing competition
Mode completes a timed task, for different timed tasks, can according to business demand the most more new configuration file, and
And directly replace the configuration file in etcd cluster with new configuration file, it is to avoid configuration file updates second server one by one
Loaded down with trivial details, it is ensured that the concordance of the configuration file that second server reads, therefore this invention ensures that the distributed of exploitation
The high efficiency of timed task system.
2), described first server be set to multiple, between two described first servers each other two-way communication connect, protect
Demonstrate,prove the first server in etcd cluster can synchronously read configuration file simultaneously；Described second server is according to described
Store path reads configuration file from first server, and applies described configuration file, is effectively guaranteed reading configuration literary composition
The concordance of part.
3), described second server monitored the renewal of configuration file by the Watch API in etcd cluster, described second
The configuration file configuration self that server application is new, and described in re invocation, the change of configuration file is monitored in Watch API continuation,
Guarantee after configuration file updates, second server can self configuration file of change promptly and accurately, make to design point
Cloth timed task system possesses higher accuracy and reliability.
Accompanying drawing explanation
Fig. 1 is the system architecture diagram of the present invention；
Fig. 2 is the system modularization design figure of the present invention.
Reference implication in figure is as follows:
10 timed task configuration service 20 etcd clusters 30 perform service cluster
Detailed description of the invention
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Describe, it is clear that described embodiment is only a part of embodiment of the present invention rather than whole embodiments wholely.Based on
Embodiment in the present invention, it is every other that those of ordinary skill in the art are obtained under not making creative work premise
Embodiment, broadly falls into the scope of protection of the invention.
As it is shown in figure 1, from the point of view of system architecture angle, distributed timing task system is mainly made up of 3 parts, including fixed
Time task configuration service 10, etcd cluster 20, perform service cluster 30, they are respectively used to the configuration of system and system configuration
Update, storage system configuration and timed task state, perform timed task according to the configuration content of system.
As shown in Figure 1, 2, in terms of system architecture and functional realiey two, setting of distributed timing task system is discussed
Meter realizes.
S1, according to business demand prepare timed task configuration file；
Described configuration file is used for describing the details of timed task that distributed timing task system needs to perform, described in join
Putting including at least a timed task list in file, each task includes task names, the timed task of crontab form
Describing, configuration file format can be yaml form, json form, xml form etc., and yaml form is preferably used.
In order to be better understood from configuration file, table 1 is a configuration file simplified, and there is defined two timings and appoints
Business.The entitled mytask1 of task one, performed a mycommand1 every 1 hour；The entitled mytask2 of task two, often
Mycommand2 is performed during it fixing 01:00.
Table 1:
Described configuration file is stored in the first server in etcd cluster 20 by S2, timed task configuration service 10, by institute
State etcd cluster 20 to be sent to the store path of described configuration file perform the second server in service cluster 30, described in deposit
Storage path is /crontab/config.yaml path.
S3, described second server read configuration literary composition according to/crontab/config.yaml path from first server
Part, and apply described configuration file.
Configuration file described above, will create two timed tasks in systems.
S4, can be according to business demand the most more new configuration file, when needing more new configuration file, directly with new configuration
The configuration file in etcd cluster 20 replaced by file.
S5, described second server monitor the renewal of configuration file；
Described second server monitors the renewal of configuration file by the Watch API in etcd cluster 20；Described second
The configuration file configuration self that server application is new, and described in re invocation, the change of configuration file is monitored in Watch API continuation,
Ensure that after configuration file updates, second server can self configuration file of change promptly and accurately.
S6, execution service request perform timed task；
Described second server according to the time point of each timed task in described configuration file to etcd cluster 20 Shen
Please perform timed task, when the time conditions of the second server described timed task of execution meets the time point of timed task,
Described second server application performs timed task, detects described timing by the checkAndSet API in etcd cluster 20
Task is the most by other second server application, if described timed task is applied, then timed task application failure, no
Then, timing is applied for successfully.
Such as task one mytask1 task when 14:00 on the 13rd November in 2015, file path is /crontab/
According to the characteristic of etcd cluster 20, mytask1/2015-11-13-14-00, can ensure that only one of which second server can return
Successfully result.
When the load of some second server is higher, it may be considered that before step S2, wait for a period of time, so may be used
To allow load with relatively low second server priority requisition to task.
S7, when described second server application is to timed task, start to perform described timed task, and update timing and appoint
The application status of business in etcd cluster 10, increases the description information of current timed task in new configuration file；Otherwise, weight
Multiple step S5.
Described description information includes the numbering of described second server, second server place Hostname, second service
The startup time of the network address of device, the port numbers of second server, the process number of timed task and timed task.
Further, described first server is set to multiple, and between two described first servers, two-way communication connects each other
Connect, and the data between described first server carry out synchronized update, be described in obtaining a described first server and join
While putting file, other described first server also synchronizations obtain the data of configuration file.
Further, each described first server can corresponding multiple described second servers.
Claims (10)
1. the method for designing of a distributed timing task system based on etcd cluster, it is characterised in that comprise the following steps:
S1, according to business demand prepare timed task configuration file；
Described configuration file is stored in the first server in etcd cluster (20) by S2, timed task configuration service (10), then by
The store path of described configuration file is sent to perform the second server in service cluster (30) by described etcd cluster (20)；
S3, described second server read configuration file according to described store path from first server, and join described in application
Put file；
S4, when needing more new configuration file, directly replace the configuration file in etcd cluster (20) with new configuration file；
S5, described second server monitor the renewal of configuration file；
S6, described second server application perform timed task；
S7, when described second server application is to timed task, start to perform described timed task, and update timed task
Application status in etcd cluster (20), increases the description information of current timed task in new configuration file；Otherwise, repeat
Step S5.
A kind of method for designing of distributed timing task system based on etcd cluster, its feature
Being, step S5 specifically includes:
S51, described second server monitor the renewal of configuration file by the Watch API in etcd cluster (20)；
The configuration file configuration self that the application of S52, described second server is new, and Watch API continuation prison described in re invocation
Listen the change of configuration file.
A kind of method for designing of distributed timing task system based on etcd cluster, its feature
Being, step S6 specifically includes:
S61, when second server judges that time conditions meets the time point of timed task, described second server application performs
Timed task；
S62, detected described timed task by checkAndSet API in etcd cluster (20) and the most second taken by other
Business device application；
If the described timed task of S63 is applied, then timed task application failure, otherwise, timing is applied for successfully.
A kind of method for designing of distributed timing task system based on etcd cluster, its feature
It is: described configuration file is the configuration file of the overall situation, and the configuration information of described timed task is maintained in configuration file.
A kind of method for designing of distributed timing task system based on etcd cluster, its feature
It is: the form of described configuration file is yaml form.
A kind of method for designing of distributed timing task system based on etcd cluster, its feature
It is: described configuration file is stored in the first server in etcd cluster (20) by RESTful interface.
A kind of method for designing of distributed timing task system based on etcd cluster, its feature
It is: described store path is /crontab/config.yaml path.
A kind of method for designing of distributed timing task system based on etcd cluster, its feature
It is: described description information includes the numbering of described second server, second server place Hostname, second server
The startup time of the network address, the port numbers of second server, the process number of timed task and timed task.
9. the design side of a kind of based on etcd cluster the distributed timing task system as described in any one of claim 1～8
Method, it is characterised in that: described first server is set to multiple, and between two described first servers, two-way communication connects each other
Connect.
A kind of method for designing of distributed timing task system based on etcd cluster, its feature
It is: each described first server can corresponding multiple described second servers.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610287542.0A CN105872073B (en) | 2016-04-28 | 2016-04-28 | A kind of design method of the distributed timing task system based on etcd cluster |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610287542.0A CN105872073B (en) | 2016-04-28 | 2016-04-28 | A kind of design method of the distributed timing task system based on etcd cluster |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105872073A true CN105872073A (en) | 2016-08-17 |
| CN105872073B CN105872073B (en) | 2019-08-02 |
Family
ID=56629061
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610287542.0A Active CN105872073B (en) | 2016-04-28 | 2016-04-28 | A kind of design method of the distributed timing task system based on etcd cluster |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105872073B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106529682A (en) * | 2016-10-28 | 2017-03-22 | 北京奇虎科技有限公司 | Method and apparatus for processing deep learning task in big-data cluster |
| CN107065616A (en) * | 2017-05-08 | 2017-08-18 | 浙江风向标科技有限公司 | Inter-linked controlling method and device, computer-readable recording medium |
| CN109800068A (en) * | 2018-12-20 | 2019-05-24 | 航天信息股份有限公司 | Handle method, system, primary server and the timed task server of timed task |
| CN112597131A (en) * | 2020-12-25 | 2021-04-02 | 广州趣丸网络科技有限公司 | Distributed script control method and related device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101140528A (en) * | 2007-08-31 | 2008-03-12 | 中兴通讯股份有限公司 | Method, device and system for realizing timing tasks load equilibria in cluster |
| CN103647840A (en) * | 2013-12-19 | 2014-03-19 | 深圳市青葡萄科技有限公司 | Distributed management method of symmetric cluster |
| US20140289733A1 (en) * | 2013-03-22 | 2014-09-25 | Palo Alto Research Center Incorporated | System and method for efficient task scheduling in heterogeneous, distributed compute infrastructures via pervasive diagnosis |
| CN104536809A (en) * | 2014-11-26 | 2015-04-22 | 上海瀚之友信息技术服务有限公司 | Distributed timing task scheduling system based on client and server system |
| CN105100259A (en) * | 2015-08-18 | 2015-11-25 | 北京京东尚科信息技术有限公司 | Distributed timed task execution method and system |
-
2016
- 2016-04-28 CN CN201610287542.0A patent/CN105872073B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101140528A (en) * | 2007-08-31 | 2008-03-12 | 中兴通讯股份有限公司 | Method, device and system for realizing timing tasks load equilibria in cluster |
| US20140289733A1 (en) * | 2013-03-22 | 2014-09-25 | Palo Alto Research Center Incorporated | System and method for efficient task scheduling in heterogeneous, distributed compute infrastructures via pervasive diagnosis |
| CN103647840A (en) * | 2013-12-19 | 2014-03-19 | 深圳市青葡萄科技有限公司 | Distributed management method of symmetric cluster |
| CN104536809A (en) * | 2014-11-26 | 2015-04-22 | 上海瀚之友信息技术服务有限公司 | Distributed timing task scheduling system based on client and server system |
| CN105100259A (en) * | 2015-08-18 | 2015-11-25 | 北京京东尚科信息技术有限公司 | Distributed timed task execution method and system |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106529682A (en) * | 2016-10-28 | 2017-03-22 | 北京奇虎科技有限公司 | Method and apparatus for processing deep learning task in big-data cluster |
| CN107065616A (en) * | 2017-05-08 | 2017-08-18 | 浙江风向标科技有限公司 | Inter-linked controlling method and device, computer-readable recording medium |
| CN109800068A (en) * | 2018-12-20 | 2019-05-24 | 航天信息股份有限公司 | Handle method, system, primary server and the timed task server of timed task |
| CN112597131A (en) * | 2020-12-25 | 2021-04-02 | 广州趣丸网络科技有限公司 | Distributed script control method and related device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105872073B (en) | 2019-08-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8301600B1 (en) | Failover recovery in a distributed data store | |
| CN102404390B (en) | Intelligent dynamic load balancing method for high-speed real-time database | |
| CN105100259B (en) | A kind of distributed timing task executing method and system | |
| EP2648114B1 (en) | Method, system, token conreoller and memory database for implementing distribute-type main memory database system | |
| EP2535810A2 (en) | System and method for performing distributed parallel processing tasks in a spot market | |
| US20100023564A1 (en) | Synchronous replication for fault tolerance | |
| US10127077B2 (en) | Event distribution pattern for use with a distributed data grid | |
| US20130132946A1 (en) | Synchronized failover for active-passive applications | |
| US9201747B2 (en) | Real time database system | |
| CN105872073A (en) | Design method of distributed timed task system based on etcd cluster | |
| CN104408071A (en) | Distributive database high-availability method and system based on cluster manager | |
| CN103312809A (en) | Distributed management method for service in cloud platform | |
| US20140059315A1 (en) | Computer system, data management method and data management program | |
| CN114301972B (en) | Hierarchical deployment method and system for blockchain nodes based on cloud edge cooperation | |
| CN102983996A (en) | Dynamic allocation method and system for high-availability cluster resource management | |
| CN105577430A (en) | Node management method of high-end fault-tolerant server | |
| CN102937964A (en) | Intelligent data service method based on distributed system | |
| CN106850269A (en) | A kind of management system of cloud platform | |
| CN105069152A (en) | Data processing method and apparatus | |
| CN112631764A (en) | Task scheduling method and device, computer equipment and computer readable medium | |
| CN104484228A (en) | Distributed parallel task processing system based on Intelli-DSC (Intelligence-Data Service Center) | |
| CN104410511A (en) | Server management method and system | |
| CN105100185A (en) | System and method for adaptively integrating a database state notification service with a distributed transactional middleware machine | |
| CN107800501A (en) | A kind of method for synchronizing time, apparatus and system | |
| CN112231399A (en) | Method and device applied to graph database |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |