CN107590284A - The electoral machinery and system of task manager in distributed reptile system - Google Patents
The electoral machinery and system of task manager in distributed reptile system Download PDFInfo
- Publication number
- CN107590284A CN107590284A CN201710914015.2A CN201710914015A CN107590284A CN 107590284 A CN107590284 A CN 107590284A CN 201710914015 A CN201710914015 A CN 201710914015A CN 107590284 A CN107590284 A CN 107590284A
- Authority
- CN
- China
- Prior art keywords
- distributed
- task manager
- equipment
- task
- message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Computer And Data Communications (AREA)
Abstract
The invention discloses a kind of electoral machinery of task manager in distributed reptile system, methods described comprises the following steps：Distributed apparatus receives or initiated election message, and the election message is used to elect task manager from distributed reptile system；Device parameter by the other equipment in broadcast to distributed reptile system, is received the broadcast message that other equipment is sent, the first optimal distributed apparatus of extraction equipment parameter from broadcast message by distributed apparatus；Distributed apparatus receives the ballot message that other equipment is sent, and the ballot message includes：Poll and the distributed apparatus of ballot, the most distributed apparatus of poll is defined as task manager；If distributed apparatus is task manager, the reptile task of processing locality is distributed into other distributed apparatus.Technical scheme provided by the invention has the advantages of efficiency high.
Description
Technical field
The present invention relates to a kind of election side of task manager in data processing field, more particularly to distributed reptile system
Method and system.
Background technology
(be otherwise known as web crawlers webpage spider, network robot, and among FOAF communities, more frequent is referred to as webpage
Follower), it is a kind of program or script according to certain rule, automatically crawl web message.Other is seldom
The name used also has ant, automatic indexing, simulation program or worm.
Web crawlers is really a kind of application program of network information crawl, and existing web crawlers crawl data volume is big,
The task manager of the distribution of task is randomly assigned, and it may influence the efficiency of task distribution, influences the efficiency of reptile.
The content of the invention
The application provides a kind of electoral machinery of task manager in distributed reptile system.It solves the skill of prior art
The shortcomings that art scheme efficiency is low.
On the one hand, there is provided a kind of distributed reptile method for allocating tasks, methods described comprise the following steps：
Distributed apparatus receives or initiated election message, and the election message is used to elect from distributed reptile system
Task manager；Distributed apparatus by device parameter by the other equipment in broadcast to distributed reptile system,
Receive the broadcast message that other equipment is sent, the first optimal distributed apparatus of extraction equipment parameter from broadcast message；Distribution
Formula equipment receives the ballot message that other equipment is sent, and the ballot message includes：Poll and the distributed apparatus of ballot, by ticket
The most distributed apparatus of number is defined as task manager；If distributed apparatus is task manager, by the reptile of processing locality
Task distributes to other distributed apparatus.
Optionally, methods described also includes：
The distributed apparatus of poll more than second is defined as standby task manager, reduces the task of standby task manager
Handle threshold value.
Optionally, methods described also includes：
Such as task manager failure, start task manager of the standby task manager as distributed system.
Second aspect, there is provided a kind of distributed reptile task distribution system, the system include：Multiple distributed apparatus；
Distributed apparatus, message is elected for receiving or initiating, the election message is used for from distributed reptile system
Elect task manager；By device parameter by the other equipment in broadcast to distributed reptile system, receive
The broadcast message that other equipment is sent, the first optimal distributed apparatus of extraction equipment parameter from broadcast message；Receive other
The ballot message that equipment is sent, the ballot message include：Poll and the distributed apparatus of ballot, by the most distribution of poll
Equipment is defined as task manager；
If distributed apparatus is task manager, the reptile task of processing locality is distributed into other distributed apparatus.
Optionally, the distributed apparatus, it is additionally operable to the distributed apparatus of poll more than second being defined as standby task pipe
Device is managed, reduces the task processing threshold value of standby task manager.
Optionally, the distributed apparatus, be additionally operable to such as task manager failure, start standby task manager as point
The task manager of cloth system.
The third aspect, there is provided a kind of distributed apparatus, including：Processor, wireless transceiver, memory and bus, it is described
Processor, wireless transceiver, memory are connected by bus,
The wireless transceiver, message is elected for receiving or initiating, the election message is used for from distributed reptile system
Task manager is elected in system；
The processor, for device parameter to be set by other in broadcast to distributed reptile system
It is standby, receive the broadcast message that other equipment is sent, the first optimal distributed apparatus of extraction equipment parameter from broadcast message；Connect
The ballot message that other equipment is sent is received, the ballot message includes：Poll and the distributed apparatus of ballot, poll is most
Distributed apparatus is defined as task manager；If distributed apparatus is task manager, the reptile task of processing locality is distributed
Give other distributed apparatus.
Optionally, the processor, for the distributed apparatus of poll more than second to be defined as into standby task manager, drop
The task processing threshold value of low standby task manager.
Optionally, the processor, for such as task manager failure, start standby task manager is as distributed
The task manager of system.
Fourth aspect, there is provided a kind of computer-readable recording medium, it stores the computer journey for electronic data interchange
Sequence, wherein, the computer program causes computer to perform the method that first aspect provides.
It is optimal that technical scheme provided by the invention by the method for election elects parameter from multiple distributed apparatus
Equipment does not handle reptile task as task manager is done when as task manager, and the reptile of processing locality is appointed
Other distributed apparatus are distributed in business, are achieved in that the quick distribution to reptile task, improve efficiency.
Brief description of the drawings
In order to illustrate the technical solution of the embodiments of the present invention more clearly, required use in being described below to embodiment
Accompanying drawing be briefly described, it should be apparent that, drawings in the following description are some embodiments of the present invention, for this area
For those of ordinary skill, on the premise of not paying creative work, other accompanying drawings can also be obtained according to these accompanying drawings.
Fig. 1 is the election of task manager in a kind of distributed reptile system that the first better embodiment of the invention provides
The flow chart of method；
Fig. 2 is the election of task manager in a kind of distributed reptile system that the second better embodiment of the invention provides
The structure chart of system.
Fig. 3 is a kind of hardware structure diagram for distributed apparatus that the second better embodiment of the invention provides.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete
Site preparation describes, it is clear that described embodiment is part of the embodiment of the present invention, rather than whole embodiments.Based on this hair
Embodiment in bright, the every other implementation that those of ordinary skill in the art are obtained under the premise of creative work is not made
Example, belongs to the scope of protection of the invention.
Fig. 1 is refer to, Fig. 1 is task pipe in a kind of distributed reptile system that the first better embodiment of the invention proposes
The electoral machinery of device is managed, this method is as shown in figure 1, comprise the following steps：
Step S101, distributed apparatus receives or initiated election message, and the election message is used for from distributed reptile system
Task manager is elected in system.
Step S102, device parameter is passed through other in broadcast to distributed reptile system by distributed apparatus
Equipment, receive the broadcast message that other equipment is sent, the first optimal distributed apparatus of extraction equipment parameter from broadcast message.
The said equipment parameter can include：Device hardware parameter, for example, the parameter of internal memory, CPU, memory, may be used also certainly
With including some variable elements, for example, reptile task quantity, memory usage, CPU usage etc..
Step S103, distributed apparatus receives the ballot message that other equipment is sent, and the ballot message includes：Poll and
The distributed apparatus of ballot, the most distributed apparatus of poll is defined as task manager.
Step S104, such as distributed apparatus is task manager, and the reptile task of processing locality is distributed into other distributions
Formula equipment.
It is optimal that technical scheme provided by the invention by the method for election elects parameter from multiple distributed apparatus
Equipment does not handle reptile task as task manager is done when as task manager, and the reptile of processing locality is appointed
Other distributed apparatus are distributed in business, are achieved in that the quick distribution to reptile task, improve efficiency.
Optionally, the above method can also include：
The distributed apparatus of poll more than second is defined as standby task manager, reduces the task of standby task manager
Handle threshold value.
Optionally, the above method can also include：
Such as task manager failure, start task manager of the standby task manager as distributed system.
This mode avoids repeatedly election from influencing the course.
Optionally, the above method can also include：
The equipment being directly connected to task manager is defined as the first equipment group, loaded for the first equipment group using first
Balanced algorithm distribution reptile task, is defined as the second equipment group by the equipment being indirectly connected with task management server, for the
For two equipment groups using the algorithm distribution reptile task of the second load balancing, the task threshold of the algorithm of the second load balancing is less than the
The task threshold of one load-balancing algorithm.
It considers that the equipment for being directly connected to, i.e. hop count only have 1, this apart from short, the delay between network
It is few, and it is not easy communication failure occur with task manager, for this kind equipment, it is defined as the first most crucial equipment group, adopts
Realize the distribution of task with the larger load-balancing algorithm of task threshold, from remote equipment, because its hop count is more, distance
Long, the delay between network is short, the failure to communicate easily occurs, so the task quantity of distribution should be lacked.
Optionally, the above method can also include：
Heartbeat message between detection device and task manager, first as do not received the first equipment in setting time
Heartbeat message, the reptile task that the first equipment does not complete is defined as reptile task to be allocated.
Fig. 2 is refer to, Fig. 2 is that a kind of distributed reptile that the second better embodiment of the invention proposes realizes system, should
System as shown in Fig. 2 including：Distributed apparatus 201 and distributed apparatus 201, the task manager are connected with equipment；
Distributed apparatus, message is elected for receiving or initiating, the election message is used for from distributed reptile system
Elect task manager；By device parameter by the other equipment in broadcast to distributed reptile system, receive
The broadcast message that other equipment is sent, the first optimal distributed apparatus of extraction equipment parameter from broadcast message；Receive other
The ballot message that equipment is sent, the ballot message include：Poll and the distributed apparatus of ballot, by the most distribution of poll
Equipment is defined as task manager；
If distributed apparatus is task manager, the reptile task of processing locality is distributed into other distributed apparatus.
Optionally, the distributed apparatus, it is additionally operable to the distributed apparatus of poll more than second being defined as standby task pipe
Device is managed, reduces the task processing threshold value of standby task manager.
Optionally, the distributed apparatus, be additionally operable to such as task manager failure, start standby task manager as point
The task manager of cloth system.
Refering to Fig. 3, Fig. 3 is a kind of distributed apparatus 30, including：Processor 301, wireless transceiver 302, memory 303
With bus 304, wireless transceiver 302 is used for the transceiving data between external equipment.The quantity of processor 301 can be one or
It is multiple.In some embodiments of the present application, processor 301, memory 302 and transceiver 303 can pass through bus 304 or its other party
Formula connects.The step of server 30 can be used for performing Fig. 1.The implication for the term being related on the present embodiment and citing, can
With the embodiment with reference to corresponding to figure 1.Here is omitted.
Wireless transceiver 302, for obtaining the reptile task, obtain the distance of the equipment being connected with the task manager with
And reptile task quantity.
Wherein, store program codes in memory 303.Processor 901 is used to call the program generation stored in memory 903
Code, for performing following operation：
Processor 301, for being that the equipment distributes reptile task according to the distance and reptile task quantity.
It should be noted that processor 301 here can be a treatment element or multiple treatment elements
It is referred to as.For example, the treatment element can be central processing unit (Central Processing Unit, CPU) or spy
Determine integrated circuit (Application Specific Integrated Circuit, ASIC), or be arranged to implement this
Apply for one or more integrated circuits of embodiment, such as：One or more microprocessors (digital singnal
Processor, DSP), or, one or more field programmable gate array (Field Programmable Gate Array,
FPGA)。
Memory 303 can be the general designation of a storage device or multiple memory elements, and for storing and can hold
Parameter, data etc. required for line program code or the operation of application program running gear.And memory 303 can include random storage
Device (RAM), nonvolatile memory (non-volatile memory), such as magnetic disk storage, flash memory can also be included
(Flash) etc..
Bus 304 can be that industry standard architecture (Industry Standard Architecture, ISA) is total
Line, external equipment interconnection (Peripheral Component, PCI) bus or extended industry-standard architecture (Extended
Industry Standard Architecture, EISA) bus etc..The bus can be divided into address bus, data/address bus, control
Bus processed etc..For ease of representing, only represented in Fig. 3 with a thick line, it is not intended that an only bus or a type of
Bus.
The terminal can also include input/output unit, be connected to bus 304, with by bus and the grade of processor 301 its
Its part connects.The input/output unit can provide an inputting interface for operating personnel, so that operating personnel pass through the input
Interface selects item of deploying to ensure effective monitoring and control of illegal activities, and can also be other interfaces, can pass through the external miscellaneous equipment of the interface.
It should be noted that for foregoing each embodiment of the method, in order to be briefly described, therefore it is all expressed as to a system
The combination of actions of row, but those skilled in the art should know, the present invention is not limited by described sequence of movement, because
For according to the present invention, certain some step can use other orders or carry out simultaneously.Secondly, those skilled in the art also should
Know, embodiment described in this description belongs to preferred embodiment, involved action and module not necessarily this hair
Necessary to bright.
In the above-described embodiments, the description to each embodiment all emphasizes particularly on different fields, and is not described in some embodiment
Part, may refer to the associated description of other embodiment.
One of ordinary skill in the art will appreciate that all or part of step in the various methods of above-described embodiment is can
To instruct the hardware of correlation to complete by program, the program can be stored in a computer-readable recording medium, storage
Medium can include：Flash disk, read-only storage (English：Read-Only Memory, referred to as：ROM), random access device (English
Text：Random Access Memory, referred to as：RAM), disk or CD etc..
The content download method and relevant device that are there is provided above the embodiment of the present invention, system are described in detail,
Specific case used herein is set forth to the principle and embodiment of the present invention, and the explanation of above example is simply used
Understand the method and its core concept of the present invention in help；Meanwhile for those of ordinary skill in the art, according to the present invention's
Thought, there will be changes in specific embodiments and applications, in summary, this specification content should not be construed as
Limitation of the present invention.
Claims (10)
1. the electoral machinery of task manager in a kind of distributed reptile system, it is characterised in that methods described includes following step
Suddenly：
Distributed apparatus receives or initiated election message, and the election message is used to elect from distributed reptile system to go out on missions
Manager；
Device parameter by the other equipment in broadcast to distributed reptile system, is received other by distributed apparatus
The broadcast message that equipment is sent, the first optimal distributed apparatus of extraction equipment parameter from broadcast message；
Distributed apparatus receives the ballot message that other equipment is sent, and the ballot message includes：Poll and the distribution of ballot
Equipment, the most distributed apparatus of poll is defined as task manager；
If distributed apparatus is task manager, the reptile task of processing locality is distributed into other distributed apparatus.
2. according to the method for claim 1, it is characterised in that methods described also includes：
The distributed apparatus of poll more than second is defined as standby task manager, reduces the task processing of standby task manager
Threshold value.
3. according to the method for claim 1, it is characterised in that methods described also includes：
Such as task manager failure, start task manager of the standby task manager as distributed system.
4. a kind of distributed reptile task distribution system, it is characterised in that the system includes：Multiple distributed apparatus；
Distributed apparatus, message is elected for receiving or initiating, the election message is used to elect from distributed reptile system
Go out on missions manager；By device parameter by the other equipment in broadcast to distributed reptile system, other are received
The broadcast message that equipment is sent, the first optimal distributed apparatus of extraction equipment parameter from broadcast message；Receive other equipment
The ballot message of transmission, the ballot message include：Poll and the distributed apparatus of ballot, by the most distributed apparatus of poll
It is defined as task manager；
If distributed apparatus is task manager, the reptile task of processing locality is distributed into other distributed apparatus.
5. system according to claim 4, it is characterised in that
The distributed apparatus, it is additionally operable to the distributed apparatus of poll more than second being defined as standby task manager, reduces standby
Threshold value is handled with the task of task manager.
6. according to the method for claim 4, it is characterised in that
The distributed apparatus, it is additionally operable to such as task manager failure, start standby task manager as distributed system
Task manager.
7. a kind of distributed apparatus, including：Processor, wireless transceiver, memory and bus, the processor, wireless receiving and dispatching
Device, memory are connected by bus, it is characterised in that
The wireless transceiver, message is elected for receiving or initiating, the election message is used for from distributed reptile system
Elect task manager；
The processor, for device parameter by the other equipment in broadcast to distributed reptile system, to be connect
Receive the broadcast message that other equipment is sent, the first optimal distributed apparatus of extraction equipment parameter from broadcast message；Receive it
The ballot message that his equipment is sent, the ballot message include：Poll and the distributed apparatus of ballot, by the most distribution of poll
Formula equipment is defined as task manager；If distributed apparatus is task manager, the reptile task of processing locality is distributed to it
His distributed apparatus.
8. server according to claim 7, it is characterised in that the processor, for by the distribution of poll more than second
Formula equipment is defined as standby task manager, reduces the task processing threshold value of standby task manager.
9. server according to claim 7, it is characterised in that the processor, for such as task manager failure, open
Move task manager of the standby task manager as distributed system.
A kind of 10. computer-readable recording medium, it is characterised in that it stores the computer program for electronic data interchange,
Wherein, the computer program causes computer to perform the method as described in claim any one of 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710914015.2A CN107590284A (en) | 2017-09-30 | 2017-09-30 | The electoral machinery and system of task manager in distributed reptile system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710914015.2A CN107590284A (en) | 2017-09-30 | 2017-09-30 | The electoral machinery and system of task manager in distributed reptile system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107590284A true CN107590284A (en) | 2018-01-16 |
Family
ID=61052767
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710914015.2A Pending CN107590284A (en) | 2017-09-30 | 2017-09-30 | The electoral machinery and system of task manager in distributed reptile system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107590284A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105743995A (en) * | 2016-04-05 | 2016-07-06 | 北京轻元科技有限公司 | Transplantable high-available container cluster deploying and managing system and method |
| CN106685724A (en) * | 2017-01-10 | 2017-05-17 | 网宿科技股份有限公司 | Node-server management method and device based on election and system |
| US20170141971A1 (en) * | 2015-11-12 | 2017-05-18 | Hulu, Llc. | Leader Election in Distributed Computer System |
| CN106911728A (en) * | 2015-12-22 | 2017-06-30 | 华为技术服务有限公司 | The choosing method and device of host node in distributed system |
| CN107071009A (en) * | 2017-03-28 | 2017-08-18 | 江苏飞搏软件股份有限公司 | A kind of distributed big data crawler system of load balancing |
| CN107070729A (en) * | 2017-06-13 | 2017-08-18 | 深圳市伊特利网络科技有限公司 | Main equipment elects method and system in network linking |
-
2017
- 2017-09-30 CN CN201710914015.2A patent/CN107590284A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170141971A1 (en) * | 2015-11-12 | 2017-05-18 | Hulu, Llc. | Leader Election in Distributed Computer System |
| CN106911728A (en) * | 2015-12-22 | 2017-06-30 | 华为技术服务有限公司 | The choosing method and device of host node in distributed system |
| CN105743995A (en) * | 2016-04-05 | 2016-07-06 | 北京轻元科技有限公司 | Transplantable high-available container cluster deploying and managing system and method |
| CN106685724A (en) * | 2017-01-10 | 2017-05-17 | 网宿科技股份有限公司 | Node-server management method and device based on election and system |
| CN107071009A (en) * | 2017-03-28 | 2017-08-18 | 江苏飞搏软件股份有限公司 | A kind of distributed big data crawler system of load balancing |
| CN107070729A (en) * | 2017-06-13 | 2017-08-18 | 深圳市伊特利网络科技有限公司 | Main equipment elects method and system in network linking |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106934027A (en) | Distributed reptile realization method and system | |
| CN104503831B (en) | Equipment optimization method and device | |
| CN108121511A (en) | Data processing method, device and equipment in a kind of distributed edge storage system | |
| CN109522314A (en) | Data archiving method and terminal device based on block chain | |
| CN105630419B (en) | A kind of the subregion view sending method and management node of resource pool | |
| CN110263580A (en) | Data processing method, device and block chain node based on block chain | |
| CN107528912A (en) | A kind of automatic method, system and relevant apparatus for realizing distributing real time system | |
| CN107948052A (en) | Information crawler method, apparatus, electronic equipment and system | |
| CN106954043A (en) | The method for allocating tasks and system of cloud service in monitoring system | |
| CN106845869A (en) | The method and system of affiliated enterprise are determined based on big data | |
| CN104267985A (en) | Software loading method and device | |
| CN106952207A (en) | Monitor the electoral machinery and system of cloud platform distributed system | |
| CN107590284A (en) | The electoral machinery and system of task manager in distributed reptile system | |
| CN106851213A (en) | The packet method for allocating tasks and system of cloud platform in monitoring system | |
| CN107562956A (en) | Distributed reptile method for allocating tasks and system | |
| CN108021639A (en) | Information-pushing method, device, server and storage medium | |
| CN107679233A (en) | Distributed reptile method for allocating tasks and system | |
| CN107249187A (en) | The terminal management method and system of countermark | |
| CN107545004A (en) | The method, apparatus and search engine system of a kind of search | |
| CN105978853A (en) | Session scheduling method, scheduling server and terminal | |
| CN107800789A (en) | The distribution method and system of task manager in distributed reptile system | |
| CN112073505A (en) | Method for unloading on cloud server, control device and storage medium | |
| CN106873470A (en) | The statistics and distribution method and system of coil winding machine | |
| CN106874515A (en) | Network information grasping means and system | |
| CN107085523A (en) | The realization method and system that terminal app attends to anything else |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180116 |