CN104580476B - The method and apparatus for choosing node in a distributed system - Google Patents
The method and apparatus for choosing node in a distributed system Download PDFInfo
- Publication number
- CN104580476B CN104580476B CN201510016624.7A CN201510016624A CN104580476B CN 104580476 B CN104580476 B CN 104580476B CN 201510016624 A CN201510016624 A CN 201510016624A CN 104580476 B CN104580476 B CN 104580476B
- Authority
- CN
- China
- Prior art keywords
- nodes
- node
- none
- candidate
- distributed system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 17
- 239000012634 fragment Substances 0.000 claims description 6
- 230000015654 memory Effects 0.000 abstract description 11
- 238000010586 diagram Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
- H04L65/40—Support for services or applications
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Multi Processors (AREA)
Abstract
The present invention provides a kind of method and apparatus for choosing node in a distributed system, contributes to by improving the memory size of None nodes under relatively low workload and cost, to improve the performance of entire PrestoDB clusters.This method includes：In the distributed system, using specified a part of node as candidate None nodes；In the case where needing the data slice by multiple Source nodes to converge to a node, a node is selected in the None nodes of the candidate, then the data slice of the multiple Source nodes is converged to the node of selection.
Description
Technical field
The present invention relates to field of computer technology, particularly a kind of method for choosing node in a distributed system and
Device.
Background technology
Along with the rise of big data, the business datum amount of Internet company rises year by year, therefore major Internet company
All carry out big data technology in inside, and be directed to core business system and build data warehouse, data warehouse is divided into two at present
Type：Off-line data warehouse and real-time data warehouse.
The representative products in off-line data warehouse are exactly hive, the product since bottom Computational frame is MapReduce,
It is suitable for the off-line analysis of super large data set and calculating, for the relatively high data analysis of requirement of real-time and calculates simultaneously uncomfortable
It closes.
The representative products of real-time data warehouse are PrestoDB, which is developed by FaceBook, uses PipeLine
Distributed data calculate and transmission mode, the analysis and calculating of big data disclosure satisfy that within 100ms-20m, meet
The requirement of real-time data analysis and calculating.
Since PrestoDB is a distributed computing framework based on memory, carry out data analysis and calculating when
It waits, PrestoDB will need the data analyzed and calculated to be divided into data slice and each data slice is read PrestoDB's first
In memory in Source nodes, the data in each Source node memories are then converged into a None section by network
In point or multiple Fixed nodes, specifically converges to None nodes or Fixed nodes is related to the type of aggregate function,
Such as：If including order by sentences in inquiry, with regard to needing to carry out whole sequence to all results, therefore it is each
Data in Source node memories just need to converge in a None node, then carry out whole sequence；If in inquiry
Include group by sentences, then just need to be grouped all results, therefore in each Source node memories
Data just need to converge in multiple Fixed nodes, to be grouped.
PrestoDB is that a node is randomly selected from entire cluster as None nodes, specific PrestoDB at present
The Algorithms of Selecting of various nodes is as shown in Figure 1, Fig. 1 is the stream for choosing node in PrestoDB clusters according in the prior art
The schematic diagram of journey.As shown in Figure 1, first determining whether the type for the node for needing to choose, saved if you need to choose None nodes or Fixed
Point, then randomly select in the cluster；If you need to choose Source nodes, first judge whether to need in such a way that hardware perceives, if
It is then to be chosen according to data locality, otherwise randomly selects multiple nodes as Source nodes.Here hardware perceives
Refer to the position perceived where data to be treated, locality refers to the preferential node selected where data as working node.
Just it is exactly the node where data to be treated because if assignment node, data can be reduced and carry out network
The required time is transmitted, the calculating task required time can be reduced.So hardware perception can be used in some cases
Mode chooses node by locality principle.
As soon as it can therefore be seen that if a node is selected as None nodes, then the requirement to its memory size
It is bigger.It, just must be to all nodes in cluster to ensure that the analysis of PrestoDB big data quantities is smoothed out with what is calculated
Carry out memory upgrade, make each node can be competent at when being selected as None nodes calculating require, this upgrade job amount and
Cost is all bigger.
Invention content
In view of this, the present invention provides a kind of method and apparatus for choosing node in a distributed system, by only improving
The memory size of None nodes, to improve the performance of entire PrestoDB clusters under relatively low workload and cost.
To achieve the above object, according to an aspect of the invention, there is provided a kind of choosing node in a distributed system
Method.
The present invention's chooses in the method for node in a distributed system, and distributed system is PrestoDB clusters, the party
Method includes：In the distributed system, using specified a part of node as candidate None nodes；Need will be multiple
In the case that the data slice of Source nodes converges to a node, a node is selected in the None nodes of the candidate,
Then the data slice of the multiple Source nodes is converged to the node of selection.
Optionally, after using a part of node specified as candidate None nodes, further include：Need will be multiple
In the case that the data slice of Source nodes converges to multiple Fixed nodes, the None for currently whether allowing the candidate is judged
Node saves if so, randomly selecting multiple nodes in the distributed system as Fixed as candidate Fixed nodes
Otherwise point randomly selects multiple nodes as Fixed nodes except None nodes candidate described in the distributed system.
Optionally, after using a part of node specified as candidate None nodes, further include：It is needing fragment
Data be saved in Source nodes in the case of, judge currently whether allow the None nodes of candidate as candidate's
Source nodes, if so, randomly selecting multiple nodes in the distributed system as Source nodes, otherwise described
Multiple nodes are randomly selected except candidate None nodes described in distributed system as Source nodes.
Optionally, multiple nodes are randomly selected in the distributed system includes as the step of Source nodes：
In the case of currently employed hardware perceptive mode, multiple node conducts are chosen according to locality principle in the distributed system
Source nodes.
Optionally, multiple node conducts are randomly selected except None nodes candidate described in the distributed system
The step of Source nodes includes：In the case of currently employed hardware perceptive mode, waited described in the distributed system
Multiple nodes are chosen as Source nodes according to locality principle except the None nodes of choosing.
According to another aspect of the present invention, a kind of device for choosing node in a distributed system is provided.
For the device for choosing node in a distributed system of the present invention, distributed system is PrestoDB clusters, should
Device includes：Configuration module, for recording an appointed part as candidate None nodes in the distributed system
Node；None node selecting modules, for the case where needing the data slice by multiple Source nodes to converge to a node
Under, select a node as None nodes in the None nodes of the candidate.
Optionally, further include Fixed node selecting modules, for needing to converge the data slice of multiple Source nodes
In the case of multiple Fixed nodes, judge currently whether to allow the None nodes of the candidate to be saved as candidate Fixed
Point, if so, randomly selecting multiple nodes in the distributed system as Fixed nodes, otherwise in the distributed system
Multiple nodes are randomly selected except candidate None nodes described in system as Fixed nodes.
Optionally, further include Source node selecting modules, for needing the data by fragment to be saved in Source sections
In the case of point, the None nodes for currently whether allowing the candidate are judged as candidate Source nodes, if so, in institute
It states and randomly selects multiple nodes in distributed system as Source nodes, it is otherwise candidate described in the distributed system
Multiple nodes are randomly selected except None nodes as Source nodes.
Optionally, the Source nodes selecting module is additionally operable in the case of currently employed hardware perceptive mode,
Multiple nodes are chosen except candidate None nodes described in the distributed system according to locality principle to save as Source
Point.
Optionally, the Source nodes selecting module is additionally operable in the case of currently employed hardware perceptive mode,
Multiple nodes are chosen except candidate None nodes described in the distributed system according to locality principle to save as Source
Point.
According to the technique and scheme of the present invention, a part of node is specified to be saved as candidate None in PrestoDB clusters
Point can carry out memory upgrade to the node of the range in this way to limit within limits the selection of None nodes
And dilatation, it is allowed to competent calculate and requires.This mode is not necessarily to carry out memory upgrade to all nodes of entire PrestoDB clusters
Dilatation, therefore the workload of upgrade expanding is relatively low, and the performance of entire PrestoDB clusters can be improved.
Description of the drawings
Attached drawing does not constitute inappropriate limitation of the present invention for more fully understanding the present invention.Wherein：
Fig. 1 is schematic diagram according to the ... of the embodiment of the present invention；
Fig. 2 is the schematic diagram of the method according to the ... of the embodiment of the present invention for choosing node in a distributed system；
Fig. 3 is the signal of the main modular of the device according to the ... of the embodiment of the present invention for choosing node in a distributed system
Figure.
Specific implementation mode
It explains to the exemplary embodiment of the present invention below in conjunction with attached drawing, including the various of the embodiment of the present invention
Details should think them only exemplary to help understanding.Therefore, those of ordinary skill in the art should recognize
It arrives, various changes and modifications can be made to the embodiments described herein, without departing from scope and spirit of the present invention.Together
The description to known function and structure is omitted for clarity and conciseness in sample in following description.
In the scheme of the embodiment of the present invention, a part of node in specified PrestoDB clusters is as candidate's in advance
None nodes are just selected from this part of nodes when needing to select None nodes.Configuration item can also be set, to then from
It selects None nodes still to randomly choose None nodes in PrestoDB clusters in this part of nodes to be configured.
When PrestoDB starts, which is parsed, according to the configuration information in configuration item, structure one is by corresponding
A lists of the IP-Port to composition, and used when distributing None nodes.Its configuration specification is for example：None convergence sections
Point=IP address 1：Port 1；IP address 2：Port 2.It specifies IP address and is used as time for two nodes of address 1 and address 2
The None nodes of choosing, port are respectively port 1 and port 2., can also be for whether allowing above-mentioned candidate's in configuration item
Whether None nodes are configured as candidate Fixed nodes, for allowing above-mentioned candidate None nodes as candidate
Source nodes also configured.In this way, when needing to select node, flow can carry out as shown in Figure 2.Fig. 2 is basis
The schematic diagram of the method for choosing node in a distributed system of the embodiment of the present invention.This method can be by PrestoDB
Coordinator nodes execute.
Step S21：Judgement needs the type of the node of selection.It is saved in Source nodes needing the data by fragment
In the case of, it is Source nodes to need the node of selection, enters step S24.Need to data slice carry out convergence processing when according to
The type of aggregate function determines the type for the node for needing selection, when that need to select None nodes, enters step S22；It is needing to select
When selecting Fixed nodes, S23 is entered step.
Step S22：Judge whether None nodes are chosen from specified range.The judgement is carried out according to above-mentioned configuration item.
If so, one node of selection is no as None nodes (step S221) from the candidate None nodes recorded in configuration item
A node is then randomly selected as None nodes (step S222).
Step S23：Determine whether candidate None nodes as candidate Fixed nodes.If so, can be random
Multiple nodes are chosen as Fixed nodes (step S231), otherwise except candidate None nodes in a distributed system with
Machine chooses multiple nodes as Fixed nodes.
Step S24：Judge whether in such a way that hardware perceives, if so, entering step S241, otherwise enters step
S242。
Step S241：Determine whether candidate None nodes as candidate Source nodes.If so, can be by
According to locality principle choose multiple nodes as Source nodes (step S2411), otherwise candidate in a distributed system
Multiple nodes are chosen as Source nodes (step S2412) according to locality principle except None nodes.
Step S242：Determine whether candidate None nodes as candidate Source nodes.If so, can be with
Machine chooses multiple nodes as Source nodes (step S2421), otherwise the None nodes of candidate in a distributed system it
Multiple nodes are randomly selected outside as Source nodes (step S2422).
Fig. 3 is the signal of the main modular of the device according to the ... of the embodiment of the present invention for choosing node in a distributed system
Figure.As shown in figure 3, the device 30 for choosing node in a distributed system of the embodiment of the present invention includes mainly 31 He of configuration module
None nodes selecting module 32.Configuration module 31 is used to record the appointed None nodes as candidate in distributed system
A part of node.None nodes selecting module 32 is for needing the data slice by multiple Source nodes to converge to a node
In the case of, select a node as None nodes in candidate None nodes.
Device 30 can also include further include Fixed node selecting module (not shown)s, for need will be multiple
In the case that the data slice of Source nodes converges to multiple Fixed nodes, judge currently whether allow candidate None nodes
Otherwise exist if so, randomly selecting multiple nodes in a distributed system as Fixed nodes as candidate Fixed nodes
Multiple nodes are randomly selected as Fixed nodes except candidate None nodes in distributed system.
Device 30 can also include Source node selecting module (not shown)s, for needing the data of fragment
In the case of being saved in Source nodes, the Source nodes for currently whether allowing candidate None nodes as candidate are judged,
It is otherwise candidate in a distributed system if so, randomly select multiple nodes in a distributed system as Source nodes
Multiple nodes are randomly selected except None nodes as Source nodes.
Source node selecting modules can be additionally used in the case of currently employed hardware perceptive mode, in distributed system
Multiple nodes are chosen as Source nodes according to locality principle except the None nodes of middle candidate.Source nodes select mould
Block is additionally operable in the case of currently employed hardware perceptive mode, in a distributed system according to this except candidate None nodes
Ground principle chooses multiple nodes as Source nodes.
Technical solution according to the ... of the embodiment of the present invention specifies a part of node as candidate's in PrestoDB clusters
None nodes in this way can be in the progress of the node of the range to limit within limits the selection of None nodes
Upgrading and dilatation are deposited, competent calculate is allowed to and requires.This mode is not necessarily in all nodes progress to entire PrestoDB clusters
Upgrade expanding is deposited, therefore the workload of upgrade expanding is relatively low, and the performance of entire PrestoDB clusters can be improved.
Above-mentioned specific implementation mode, does not constitute limiting the scope of the invention.Those skilled in the art should be bright
It is white, design requirement and other factors are depended on, various modifications, combination, sub-portfolio and replacement can occur.It is any
Modifications, equivalent substitutions and improvements made by within the spirit and principles in the present invention etc., should be included in the scope of the present invention
Within.
Claims (6)
1. a kind of method for choosing node in a distributed system, the distributed system are PrestoDB clusters, feature exists
In this method includes：
In the distributed system, using specified a part of node as candidate None nodes；
In the case where needing the data slice by multiple Source nodes to converge to a node, in the None nodes of the candidate
The data slice of the multiple Source nodes, is then converged to the node of selection by one node of middle selection；
Wherein, after using a part of node specified as candidate None nodes, further include：
In the case where needing the data slice by multiple Source nodes to converge to multiple Fixed nodes, whether judgement is currently permitted
Perhaps the None nodes of the described candidate are as candidate Fixed nodes, if so, being randomly selected in the distributed system multiple
Otherwise node randomly selects multiple sections as Fixed nodes except None nodes candidate described in the distributed system
Point is used as Fixed nodes；
Wherein, after using a part of node specified as candidate None nodes, further include：
In the case where needing the data by fragment to be saved in Source nodes, the None for currently whether allowing the candidate is judged
Node is as candidate Source nodes, if so, randomly selecting multiple nodes in the distributed system as Source
Otherwise node randomly selects multiple nodes and is saved as Source except None nodes candidate described in the distributed system
Point.
2. according to the method described in claim 1, making it is characterized in that, randomly selecting multiple nodes in the distributed system
Include for the step of Source nodes：In the case of currently employed hardware perceptive mode, in the distributed system according to
Locality principle chooses multiple nodes as Source nodes.
3. method according to claim 1 or 2, which is characterized in that the candidate None described in the distributed system
Multiple nodes are randomly selected except node includes as the step of Source nodes：The currently employed hardware perceptive mode the case where
Under, except the candidate None nodes described in the distributed system multiple node conducts are chosen according to locality principle
Source nodes.
4. a kind of device for choosing node in a distributed system, the distributed system are PrestoDB clusters, feature exists
In the device includes：
Configuration module, for recording appointed a part of node as candidate None nodes in the distributed system；
None node selecting modules, for the case where needing the data slice by multiple Source nodes to converge to a node
Under, select a node as None nodes in the None nodes of the candidate；
Wherein, further include Fixed node selecting modules, for need the data slice by multiple Source nodes to converge to multiple
In the case of Fixed nodes, the None nodes for currently whether allowing the candidate are judged as candidate Fixed nodes, if so,
Multiple nodes are then randomly selected in the distributed system as Fixed nodes, otherwise described in the distributed system
Multiple nodes are randomly selected as Fixed nodes except candidate None nodes；
Wherein, further include Source node selecting modules, in the feelings for needing the data by fragment to be saved in Source nodes
Under condition, the None nodes for currently whether allowing the candidate are judged as candidate Source nodes, if so, in the distribution
Multiple nodes are randomly selected in formula system as Source nodes, otherwise None candidate described in the distributed system is saved
Multiple nodes are randomly selected as Source nodes except point.
5. device according to claim 4, which is characterized in that the Source nodes selecting module is additionally operable to adopt currently
In the case of with hardware perceptive mode, multiple nodes are chosen as Source according to locality principle in the distributed system
Node.
6. device according to claim 4 or 5, which is characterized in that the Source nodes selecting module is additionally operable to working as
In the case of preceding use hardware perceptive mode, according to locality except the candidate None nodes described in the distributed system
Principle chooses multiple nodes as Source nodes.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510016624.7A CN104580476B (en) | 2015-01-13 | 2015-01-13 | The method and apparatus for choosing node in a distributed system |
| PCT/CN2016/070551 WO2016112831A1 (en) | 2015-01-13 | 2016-01-11 | Method and device of selecting distributed system node |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510016624.7A CN104580476B (en) | 2015-01-13 | 2015-01-13 | The method and apparatus for choosing node in a distributed system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104580476A CN104580476A (en) | 2015-04-29 |
| CN104580476B true CN104580476B (en) | 2018-09-14 |
Family
ID=53095633
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510016624.7A Active CN104580476B (en) | 2015-01-13 | 2015-01-13 | The method and apparatus for choosing node in a distributed system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN104580476B (en) |
| WO (1) | WO2016112831A1 (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104580476B (en) * | 2015-01-13 | 2018-09-14 | 北京京东尚科信息技术有限公司 | The method and apparatus for choosing node in a distributed system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101102225A (en) * | 2007-07-26 | 2008-01-09 | 北京航空航天大学 | Management method of wireless sensor network nodes |
| US7710884B2 (en) * | 2006-09-01 | 2010-05-04 | International Business Machines Corporation | Methods and system for dynamic reallocation of data processing resources for efficient processing of sensor data in a distributed network |
| CN101924777A (en) * | 2009-06-17 | 2010-12-22 | 中国移动通信集团公司 | Method, system and equipment for searching active nodes in P2P streaming media system |
| CN103188161A (en) * | 2011-12-30 | 2013-07-03 | 中国移动通信集团公司 | Method and system of distributed data loading scheduling |
| CN104168332A (en) * | 2014-09-01 | 2014-11-26 | 广东电网公司信息中心 | Load balance and node state monitoring method in high performance computing |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100242042A1 (en) * | 2006-03-13 | 2010-09-23 | Nikhil Bansal | Method and apparatus for scheduling work in a stream-oriented computer system |
| CN102572809A (en) * | 2010-12-27 | 2012-07-11 | 中国移动通信集团公司 | Method, system and equipment for selecting gateway nodes |
| CN104580476B (en) * | 2015-01-13 | 2018-09-14 | 北京京东尚科信息技术有限公司 | The method and apparatus for choosing node in a distributed system |
-
2015
- 2015-01-13 CN CN201510016624.7A patent/CN104580476B/en active Active
-
2016
- 2016-01-11 WO PCT/CN2016/070551 patent/WO2016112831A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7710884B2 (en) * | 2006-09-01 | 2010-05-04 | International Business Machines Corporation | Methods and system for dynamic reallocation of data processing resources for efficient processing of sensor data in a distributed network |
| CN101102225A (en) * | 2007-07-26 | 2008-01-09 | 北京航空航天大学 | Management method of wireless sensor network nodes |
| CN101924777A (en) * | 2009-06-17 | 2010-12-22 | 中国移动通信集团公司 | Method, system and equipment for searching active nodes in P2P streaming media system |
| CN103188161A (en) * | 2011-12-30 | 2013-07-03 | 中国移动通信集团公司 | Method and system of distributed data loading scheduling |
| CN104168332A (en) * | 2014-09-01 | 2014-11-26 | 广东电网公司信息中心 | Load balance and node state monitoring method in high performance computing |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2016112831A1 (en) | 2016-07-21 |
| CN104580476A (en) | 2015-04-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6559215B2 (en) | System, method for executing database query and computer readable recording medium | |
| US10404787B1 (en) | Scalable distributed data streaming computations across multiple data processing clusters | |
| US10366111B1 (en) | Scalable distributed computations utilizing multiple distinct computational frameworks | |
| EP3080720B1 (en) | Social-driven recaching of accessible objects | |
| WO2019061132A1 (en) | Hybrid file system architecture, file storage, dynamic migration, and application thereof | |
| WO2014082052A4 (en) | Scaling computing clusters | |
| KR20190043446A (en) | Workflow engine framework | |
| CN101116313B (en) | Determining highest workloads for nodes in an overlay network | |
| US20190114200A1 (en) | Workflow engine framework | |
| Heintz et al. | Cross-phase optimization in mapreduce | |
| US10776404B2 (en) | Scalable distributed computations utilizing multiple distinct computational frameworks | |
| Li et al. | MapReduce delay scheduling with deadline constraint | |
| CN103942197A (en) | Data monitoring processing method and device | |
| Bosque et al. | A load index and load balancing algorithm for heterogeneous clusters | |
| CN105635323A (en) | Data transmission control method and device | |
| CN105760212A (en) | Data redistribution method and device based on vessels | |
| CN104580476B (en) | The method and apparatus for choosing node in a distributed system | |
| US20160357806A1 (en) | Server-based management for querying eventually-consistent database | |
| Huang et al. | Intelligent task migration with deep Qlearning in multi‐access edge computing | |
| WO2021115082A1 (en) | Job scheduling method and job scheduling apparatus | |
| US10691700B1 (en) | Table replica allocation in a replicated storage system | |
| KR20180085633A (en) | Method and apparatus for processing query | |
| CN103577424B (en) | The realization method and system of distributed data base view | |
| Liang et al. | Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey | |
| Costa Filho et al. | An adaptive replica placement approach for distributed key‐value stores |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |