WO2016112831A1 - 在分布式系统中选取节点的方法和装置 - Google Patents

在分布式系统中选取节点的方法和装置 Download PDF

Info

Publication number
WO2016112831A1
WO2016112831A1 PCT/CN2016/070551 CN2016070551W WO2016112831A1 WO 2016112831 A1 WO2016112831 A1 WO 2016112831A1 CN 2016070551 W CN2016070551 W CN 2016070551W WO 2016112831 A1 WO2016112831 A1 WO 2016112831A1
Authority
WO
WIPO (PCT)
Prior art keywords
node
nodes
candidate
none
distributed system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2016/070551
Other languages
English (en)
French (fr)
Inventor
吕信
郭李明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Original Assignee
Beijing Jingdong Century Trading Co Ltd
Beijing Jingdong Shangke Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Century Trading Co Ltd, Beijing Jingdong Shangke Information Technology Co Ltd filed Critical Beijing Jingdong Century Trading Co Ltd
Publication of WO2016112831A1 publication Critical patent/WO2016112831A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/40—Support for services or applications

Definitions

  • the present invention relates to the field of computer technologies, and in particular, to a method and apparatus for selecting nodes in a distributed system.
  • the representative product of the offline data warehouse is hive. Because the underlying computing framework is MapReduce, it is suitable for off-line analysis and calculation of very large data sets. It is not suitable for data analysis and calculation with high real-time requirements.
  • PrestoDB The representative product of real-time data warehouse is PrestoDB. This product is developed by FaceBook and adopts PipeLine's distributed data calculation and transmission mode. It can meet the analysis and calculation of big data within 100ms-20m, and meets real-time data analysis and calculation. Requirements.
  • PrestoDB is a memory-based distributed computing framework
  • PrestoDB first divides the data that needs to be analyzed and calculated into data slices and reads each piece of data into the memory of the Source node of PrestoDB during data analysis and calculation.
  • the data in the memory of each Source node is aggregated through the network to a None node or a plurality of Fixed nodes, specifically, the aggregation to the None node or the Fixed node is related to the type of the aggregate function, for example, if the query contains an order By statement, then all the results need to be sorted as a whole, so the data in the memory of each Source node needs to be aggregated into a None node, and then the overall order; if the query contains a group by statement, then you need to The results are grouped, so within each Source node The data in the store needs to be aggregated into multiple Fixed nodes for grouping.
  • FIG. 1 is a schematic diagram of a process of selecting nodes in a PrestoDB cluster according to the prior art. As shown in Figure 1, first determine the type of node to be selected. If you need to select a None node or a Fixed node, you can select it randomly in the cluster. If you need to select the Source node, first determine whether you need to use hardware-aware mode. If yes, It is selected according to the data locality, otherwise multiple nodes are randomly selected as the Source node.
  • the hardware sensing here refers to the location where the data that needs to be processed is located, and the locality refers to the node where the data is preferentially selected as the working node. Because if the assigned working node is just the node where the data to be processed is located, the time required for the data to be transmitted by the network can be reduced, and the time required for the computing task can be reduced. Therefore, in some cases, hardware-aware mode can be used to select nodes according to the locality principle.
  • the present invention provides a method and apparatus for selecting nodes in a distributed system, which improves the performance of the entire PrestoDB cluster at a relatively low workload and cost by only increasing the memory capacity of the None node.
  • a method of selecting a node in a distributed system is provided.
  • the distributed system is a PrestoDB cluster
  • the method includes: in the distributed system, a specified part a node as a candidate None node; in the case where a data slice of a plurality of Source nodes needs to be aggregated to one node, one node among the candidate None nodes is selected, and then the data pieces of the multiple Source nodes are aggregated to The node selected.
  • the method further includes: determining whether the candidate None node is currently allowed if the data pieces of the multiple Source nodes need to be aggregated to multiple Fixed nodes. As a candidate Fixed node, if yes, a plurality of nodes are randomly selected as the Fixed node in the distributed system, otherwise, multiple nodes are randomly selected as the Fixed node in the distributed system except the candidate None node. .
  • the method further includes: determining whether the candidate None node is currently allowed as the candidate Source node if the data of the fragment needs to be saved to the Source node. If yes, multiple nodes are randomly selected as the Source node in the distributed system. Otherwise, multiple nodes are randomly selected as the Source node in the distributed system except the candidate None node.
  • the step of randomly selecting multiple nodes as the Source node in the distributed system includes: selecting multiple nodes as the source according to the locality principle in the distributed system when the hardware sensing mode is currently adopted. node.
  • the step of randomly selecting multiple nodes as the Source node outside the candidate None node in the distributed system includes: in the case of currently adopting a hardware-aware manner, in the distributed system In addition to the candidate None node, multiple nodes are selected as Source nodes according to the locality principle.
  • an apparatus for selecting a node in a distributed system is provided.
  • the distributed system is a PrestoDB cluster
  • the device includes: a configuration module, configured to record a part of a designated node of the distributed node in the distributed system; and a None node selection module, configured to aggregate data pieces of the plurality of Source nodes to In the case of one node, one of the candidate None nodes is selected as the None node.
  • the method further includes a fixed node selection module, configured to determine whether the candidate neighbor node is currently allowed as a candidate fixed node if the data pieces of the multiple source nodes need to be aggregated to multiple fixed nodes, if Then, a plurality of nodes are randomly selected as the Fixed node in the distributed system, otherwise, multiple nodes are randomly selected as the Fixed node in the distributed system except the candidate None node.
  • a fixed node selection module configured to determine whether the candidate neighbor node is currently allowed as a candidate fixed node if the data pieces of the multiple source nodes need to be aggregated to multiple fixed nodes, if Then, a plurality of nodes are randomly selected as the Fixed node in the distributed system, otherwise, multiple nodes are randomly selected as the Fixed node in the distributed system except the candidate None node.
  • the method further includes a source node selection module, configured to determine whether the candidate None node is currently allowed as a candidate Source node if the data of the fragment needs to be saved to the Source node, and if yes, A plurality of nodes are randomly selected as the Source node in the distributed system. Otherwise, multiple nodes are randomly selected as the Source node in the distributed system except the candidate None node.
  • a source node selection module configured to determine whether the candidate None node is currently allowed as a candidate Source node if the data of the fragment needs to be saved to the Source node, and if yes, A plurality of nodes are randomly selected as the Source node in the distributed system. Otherwise, multiple nodes are randomly selected as the Source node in the distributed system except the candidate None node.
  • the source node selection module is further configured to select multiple nodes as source nodes according to the locality principle in the distributed system except that the candidate neighbor nodes are currently employed in the distributed system.
  • the source node selection module is further configured to select multiple nodes as source nodes according to the locality principle in the distributed system except that the candidate neighbor nodes are currently employed in the distributed system.
  • a part of the nodes are designated as candidate None nodes in the PrestoDB cluster, so that the selection of the None nodes is limited to a certain range, so that the nodes in the range can be upgraded and expanded to make the computing competent. Claim.
  • This method does not require memory upgrade and expansion for all nodes in the entire PrestoDB cluster. Therefore, the workload of upgrading and expanding is relatively low, and the performance of the entire PrestoDB cluster can be improved.
  • Figure 1 is a schematic view of an embodiment of the present invention
  • FIG. 2 is a schematic diagram of a method of selecting a node in a distributed system according to an embodiment of the present invention
  • FIG. 3 is a schematic diagram of the main modules of an apparatus for selecting nodes in a distributed system in accordance with an embodiment of the present invention.
  • a part of nodes in the PrestoDB cluster are designated as candidate None nodes in advance, and are selected from the partial nodes when the None node needs to be selected. You can also set the configuration item to configure whether to select the None node from this part of the node or randomly select the None node in the PrestoDB cluster.
  • FIG. 2 is a schematic diagram of a method of selecting nodes in a distributed system in accordance with an embodiment of the present invention. This method can be performed by a Coordinator node in PrestoDB.
  • Step S21 Determine the type of the node that needs to be selected.
  • the node to be selected is the Source node, and the process proceeds to step S24.
  • the type of the node to be selected is determined according to the type of the aggregate function.
  • the process proceeds to step S22: when the fixed node needs to be selected, the process proceeds to step S23.
  • Step S22 Determine whether the None node is selected from the specified range. This judgment is made based on the above configuration items. If yes, one node is selected as a None node from the candidate None nodes recorded in the configuration item (step S221), otherwise one node is randomly selected as the None node (step S222).
  • Step S23 It is judged whether the candidate None node is allowed as the candidate Fixed node. If so, a plurality of nodes may be randomly selected as the Fixed node (step S231). Otherwise, a plurality of nodes are randomly selected as the Fixed nodes in addition to the candidate None nodes in the distributed system.
  • Step S24 determining whether the hardware sensing mode is adopted, and if yes, proceeding to step S241, otherwise proceeding to step S242.
  • Step S241 Determine whether the candidate None node is allowed as the candidate Source node. If yes, a plurality of nodes may be selected as the Source node according to the locality principle (step S2411). Otherwise, a plurality of nodes are selected as the Source node according to the locality principle in addition to the candidate None nodes in the distributed system (step S2412).
  • Step S242 Determine whether the candidate None node is allowed as the candidate Source node. If so, a plurality of nodes may be randomly selected as the Source node (step S2421), otherwise a plurality of nodes are randomly selected as the Source node in addition to the candidate None nodes in the distributed system (step S2422).
  • the apparatus 30 for selecting a node in a distributed system mainly includes a configuration module 31 and a None node selection module 32.
  • the configuration module 31 is configured to record a part of the designated node of the None node in the distributed system.
  • the None node selection module 32 is configured to select one node among the candidate None nodes as a None node in the case where it is required to aggregate data pieces of a plurality of Source nodes to one node.
  • the apparatus 30 may further include a fixed node selection module (not shown) for determining whether the candidate None node is currently allowed to be used if the data pieces of the multiple Source nodes need to be aggregated to the plurality of Fixed nodes.
  • the candidate Fixed node if yes, randomly selects multiple nodes as the Fixed node in the distributed system, otherwise, multiple nodes are randomly selected as the Fixed node in addition to the candidate None nodes in the distributed system.
  • the device 30 may further include a Source node selection module (not shown) for determining whether the candidate None node is currently allowed as the candidate Source node if the data of the fragment needs to be saved to the Source node, and if so, Then, multiple nodes are randomly selected as the Source node in the distributed system. Otherwise, multiple nodes are randomly selected as the Source node in addition to the candidate None nodes in the distributed system.
  • a Source node selection module (not shown) for determining whether the candidate None node is currently allowed as the candidate Source node if the data of the fragment needs to be saved to the Source node, and if so, Then, multiple nodes are randomly selected as the Source node in the distributed system. Otherwise, multiple nodes are randomly selected as the Source node in addition to the candidate None nodes in the distributed system.
  • the Source node selection module can also be used to select multiple nodes as Source nodes according to the locality principle in addition to the candidate None nodes in the distributed system in the current hardware-aware manner.
  • the Source node selection module is further configured to select multiple nodes as Source nodes according to the locality principle in addition to the candidate None nodes in the distributed system in the current hardware-aware manner.
  • a part of the node is designated as a candidate None node in the PrestoDB cluster, so that the selection of the None node is limited to a certain range, so that the memory of the range of nodes can be upgraded and expanded. Competency calculation requirements.
  • This method does not require memory upgrade and expansion for all nodes of the entire PrestoDB cluster. Therefore, the workload of upgrading and expanding is relatively low, and the performance of the entire PrestoDB cluster can be improved.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)
  • Multi Processors (AREA)

Abstract

本发明提供一种在分布式系统中选取节点的方法和装置，有助于在比较低的工作量和成本下通过提高None节点的内存容量，从而提高整个PrestoDB集群的性能。该方法包括：在所述分布式系统中，将指定的一部分节点作为候选的None节点；在需要将多个Source节点的数据片汇聚到一个节点的情况下，在所述候选的None节点中选择一个节点，然后将所述多个Source节点的数据片汇聚到选择的节点。

Description

在分布式系统中选取节点的方法和装置 技术领域
本发明涉及计算机技术领域，特别地涉及一种在分布式系统中选取节点的方法和装置。
背景技术
伴随着大数据的兴起，互联网公司的业务数据量逐年上升，因此各大互联网公司都在内部推行大数据技术，并且针对于核心业务系统建设数据仓库，目前数据仓库分为两种类型：离线数据仓库和实时数据仓库。
离线数据仓库的代表产品就是hive，该产品由于底层计算框架是MapReduce，因此其适合于超大数据集的离线分析和计算，对于实时性要求比较高的数据分析和计算并不适合。
实时数据仓库的代表产品是PrestoDB，该产品由FaceBook开发，采用了PipeLine的分布式数据计算和传输模式，对于大数据的分析和计算能够满足在100ms-20m之内，满足了实时数据分析和计算的要求。
由于PrestoDB是一个基于内存的分布式计算框架，在进行数据分析和计算的时候，PrestoDB首先将需要分析和计算的数据分为数据片并将每个数据片读取到PrestoDB的Source节点中的内存中，然后将每个Source节点内存中的数据通过网络汇聚到一个None节点或者多个Fixed节点中，具体是汇聚到None节点还是Fixed节点与聚合函数的类型相关，例如：如果查询中包含有order by语句，那么就需要对所有的结果进行整体排序，因此各个Source节点内存中的数据就需要汇聚到一个None节点中，然后进行整体排序；如果查询中包含有group by语句，那么就需要对所有的结果进行分组，因此各个Source节点内 存中的数据就需要汇聚到多个Fixed节点中，从而进行分组。
目前PrestoDB是从整个集群中随机选取一个节点作为None节点的，具体PrestoDB各种节点的选取算法如图1所示，图1是根据现有技术中的在PrestoDB集群中选取节点的流程的示意图。如图1所示，首先判断需要选取的节点的类型，如需选取None节点或Fixed节点，则在集群中随机选取；如需选取Source节点，先判断是否需要采用硬件感知的方式，若是，则根据数据本地性来选取，否则随机选取多个节点作为Source节点。这里的硬件感知是指感知需要处理的数据所在的位置，本地性是指优先选择数据所在的节点作为工作节点。因为如果分配的工作节点，刚好就是需要处理的数据所在的节点，就能减少数据进行网络传输所需要的时间，能够减少计算任务所需要的时间。所以在一些情况下可采用硬件感知方式，按本地性原则选取节点。
因此可以看出，如果一个节点被选择作为None节点，那么对其内存容量的要求就比较大。要想保证PrestoDB大数据量分析与计算的顺利进行，就必须对集群中的所有节点进行内存升级，使各个节点在被选择为None节点时都能胜任计算要求，这种升级工作量和成本都比较大。
发明内容
行鉴于此，本发明提供一种在分布式系统中选取节点的方法和装置，通过只提高None节点的内存容量，从而在比较低的工作量和成本下提高整个PrestoDB集群的性能。
为实现上述目的，根据本发明的一个方面，提供了一种在分布式系统中选取节点的方法。
本发明的在分布式系统中选取节点的方法中，分布式系统为PrestoDB集群，该方法包括：在所述分布式系统中，将指定的一部分 节点作为候选的None节点；在需要将多个Source节点的数据片汇聚到一个节点的情况下，在所述候选的None节点中选择一个节点，然后将所述多个Source节点的数据片汇聚到选择的节点。
可选地，在将指定的一部分节点作为候选的None节点之后，还包括：在需要将多个Source节点的数据片汇聚到多个Fixed节点的情况下，判断当前是否允许所述候选的None节点作为候选的Fixed节点，若是，则在所述分布式系统中中随机选取多个节点作为Fixed节点，否则在所述分布式系统中所述候选的None节点之外随机选取多个节点作为Fixed节点。
可选地，在将指定的一部分节点作为候选的None节点之后，还包括：在需要将分片的数据保存到Source节点的情况下，判断当前是否允许所述候选的None节点作为候选的Source节点，若是，则在所述分布式系统中随机选取多个节点作为Source节点，否则在所述分布式系统中所述候选的None节点之外随机选取多个节点作为Source节点。
可选地，在所述分布式系统中随机选取多个节点作为Source节点的步骤包括：在当前采用硬件感知方式的情况下，在所述分布式系统中按照本地性原则选取多个节点作为Source节点。
可选地，在所述分布式系统中所述候选的None节点之外随机选取多个节点作为Source节点的步骤包括：在当前采用硬件感知方式的情况下，在所述分布式系统中所述候选的None节点之外按照本地性原则选取多个节点作为Source节点。
根据本发明的另一方面，提供了一种在分布式系统中选取节点的装置。
对于本发明的在分布式系统中选取节点的装置，分布式系统为 PrestoDB集群，该装置包括：配置模块，用于记录所述分布式系统中被指定的作为候选的None节点的一部分节点；None节点选择模块，用于在需要将多个Source节点的数据片汇聚到一个节点的情况下，在所述候选的None节点中选择一个节点作为None节点。
可选地，还包括Fixed节点选择模块，用于在需要将多个Source节点的数据片汇聚到多个Fixed节点的情况下，判断当前是否允许所述候选的None节点作为候选的Fixed节点，若是，则在所述分布式系统中随机选取多个节点作为Fixed节点，否则在所述分布式系统中所述候选的None节点之外随机选取多个节点作为Fixed节点。
可选地，还包括Source节点选择模块，用于在需要将分片的数据保存到Source节点的情况下，判断当前是否允许所述候选的None节点作为候选的Source节点，若是，则在所述分布式系统中随机选取多个节点作为Source节点，否则在所述分布式系统中所述候选的None节点之外随机选取多个节点作为Source节点。
可选地，所述Source节点选择模块还用于在当前采用硬件感知方式的情况下，在所述分布式系统中所述候选的None节点之外按照本地性原则选取多个节点作为Source节点。
可选地，所述Source节点选择模块还用于在当前采用硬件感知方式的情况下，在所述分布式系统中所述候选的None节点之外按照本地性原则选取多个节点作为Source节点。
根据本发明的技术方案，在PrestoDB集群中指定一部分节点作为候选的None节点，从而将None节点的选取限定在一定范围之内，这样可以对该范围的节点进行内存升级和扩容，使之胜任计算要求。这种方式无需对整个PrestoDB集群的所有节点进行内存升级扩容，因此升级扩容的工作量比较低，并且能够提高整个PrestoDB集群的性能。
附图说明
附图用于更好地理解本发明，不构成对本发明的不当限定。其中：
图1是根据本发明实施例的示意图；
图2是根据本发明实施例的在分布式系统中选取节点的方法的示意图；
图3是根据本发明实施例的在分布式系统中选取节点的装置的主要模块的示意图。
具体实施方式
以下结合附图对本发明的示范性实施例做出说明，其中包括本发明实施例的各种细节以助于理解，应当将它们认为仅仅是示范性的。因此，本领域普通技术人员应当认识到，可以对这里描述的实施例做出各种改变和修改，而不会背离本发明的范围和精神。同样，为了清楚和简明，以下的描述中省略了对公知功能和结构的描述。
在本发明实施例的方案中，事先指定PrestoDB集群中的一部分节点作为候选的None节点，在需要选择None节点时就从这一部分节点中选择。也可以设置配置项，对于是从这一部分节点中选择None节点还是在PrestoDB集群中随机选择None节点进行配置。在PrestoDB启动的时候，对该配置项进行解析，根据配置项中的配置信息，构建一个由对应的IP-Port对组成的一个列表，并在分配None节点时进行使用。其配置规范例如None汇聚节点＝IP地址1：端口1；IP地址2：端口2。即指定了IP地址为地址1和地址2的两个节点作为候选的None节点，端口分别为端口1和端口2。在配置项中，还可以对于是否允许上述的候选的None节点作为候选的Fixed节点进行配置，对于是否允许上述的候选的None节点作为候选的Source节点也进行配置。这样，在需要选择节点时，可按图2所示流程来进行。图2是根据本发明实施例的在分布式系统中选取节点的方法的示意图。该方法可由PrestoDB中的Coordinator节点来执行。
步骤S21：判断需要选择的节点的类型。在需要将分片的数据保存到Source节点的情况下，需选择的节点是Source节点，进入步骤S24。在需要对数据片进行汇聚处理时根据聚合函数的类型来确定需选择的节点的类型，在需选择None节点时，进入步骤S22：在需选择Fixed节点时，进入步骤S23。
步骤S22：判断None节点是否从指定范围中选取。该判断根据上述的配置项进行。若是，则从配置项中记录的候选的None节点中选取一个节点作为None节点(步骤S221)，否则随机选取一个节点作为None节点(步骤S222)。
步骤S23：判断是否允许候选的None节点作为候选的Fixed节点。若是，则可以随机选取多个节点作为Fixed节点(步骤S231)，否则在分布式系统中的候选的None节点之外随机选取多个节点作为Fixed节点。
步骤S24：判断是否采用硬件感知的方式，若是，进入步骤S241，否则进入步骤S242。
步骤S241：判断是否允许候选的None节点作为候选的Source节点。若是，则可以按照本地性原则选取多个节点作为Source节点(步骤S2411)，否则在分布式系统中的候选的None节点之外按照本地性原则选取多个节点作为Source节点(步骤S2412)。
步骤S242：判断是否允许候选的None节点作为候选的Source节点。若是，则可以随机选取多个节点作为Source节点(步骤S2421)，否则在分布式系统中的候选的None节点之外随机选取多个节点作为Source节点(步骤S2422)。
图3是根据本发明实施例的在分布式系统中选取节点的装置的主要模块的示意图。如图3所示，本发明实施例的在分布式系统中选取节点的装置30主要包括配置模块31和None节点选择模块32。配置模块31用于记录分布式系统中被指定的作为候选的None节点的一部分节点。None节点选择模块32用于在需要将多个Source节点的数据片汇聚到一个节点的情况下，在候选的None节点中选择一个节点作为None节点。
装置30还可以包括还包括Fixed节点选择模块(图中未示出)，用于在需要将多个Source节点的数据片汇聚到多个Fixed节点的情况下，判断当前是否允许候选的None节点作为候选的Fixed节点，若是，则在分布式系统中随机选取多个节点作为Fixed节点，否则在分布式系统中候选的None节点之外随机选取多个节点作为Fixed节点。
装置30还可以包括Source节点选择模块(图中未示出)，用于在需要将分片的数据保存到Source节点的情况下，判断当前是否允许候选的None节点作为候选的Source节点，若是，则在分布式系统中随机选取多个节点作为Source节点，否则在分布式系统中候选的None节点之外随机选取多个节点作为Source节点。
Source节点选择模块还可用于在当前采用硬件感知方式的情况下，在分布式系统中候选的None节点之外按照本地性原则选取多个节点作为Source节点。Source节点选择模块还用于在当前采用硬件感知方式的情况下，在分布式系统中候选的None节点之外按照本地性原则选取多个节点作为Source节点。
根据本发明实施例的技术方案，在PrestoDB集群中指定一部分节点作为候选的None节点，从而将None节点的选取限定在一定范围之内，这样可以对该范围的节点进行内存升级和扩容，使之胜任计算要求。这种方式无需对整个PrestoDB集群的所有节点进行内存升级扩容， 因此升级扩容的工作量比较低，并且能够提高整个PrestoDB集群的性能。
上述具体实施方式，并不构成对本发明保护范围的限制。本领域技术人员应该明白的是，取决于设计要求和其他因素，可以发生各种各样的修改、组合、子组合和替代。任何在本发明的精神和原则之内所作的修改、等同替换和改进等，均应包含在本发明保护范围之内。

Claims (10)

  1. 一种在分布式系统中选取节点的方法，所述分布式系统为PrestoDB集群，该方法包括：
    在所述分布式系统中，将指定的一部分节点作为候选的None节点；
    在需要将多个Source节点的数据片汇聚到一个节点的情况下，在所述候选的None节点中选择一个节点，然后将所述多个Source节点的数据片汇聚到选择的节点。
  2. 根据权利要求1所述的方法，其中，在将指定的一部分节点作为候选的None节点之后，还包括：
    在需要将多个Source节点的数据片汇聚到多个Fixed节点的情况下，判断当前是否允许所述候选的None节点作为候选的Fixed节点，若是，则在所述分布式系统中随机选取多个节点作为Fixed节点，否则在所述分布式系统中所述候选的None节点之外随机选取多个节点作为Fixed节点。
  3. 根据权利要求1所述的方法，其中，在将指定的一部分节点作为候选的None节点之后，还包括：
    在需要将分片的数据保存到Source节点的情况下，判断当前是否允许所述候选的None节点作为候选的Source节点，若是，则在所述分布式系统中随机选取多个节点作为Source节点，否则在所述分布式系统中所述候选的None节点之外随机选取多个节点作为Source节点。
  4. 根据权利要求3所述的方法，其中，在所述分布式系统中随机选取多个节点作为Source节点的步骤包括：在当前采用硬件感知方式的情况下，在所述分布式系统中按照本地性原则选取多个节点作为Source节点。
  5. 根据权利要求3或4所述的方法，其中，在所述分布式系统中 所述候选的None节点之外随机选取多个节点作为Source节点的步骤包括：在当前采用硬件感知方式的情况下，在所述分布式系统中所述候选的None节点之外按照本地性原则选取多个节点作为Source节点。
  6. 一种在分布式系统中选取节点的装置，所述分布式系统为PrestoDB集群，该装置包括：
    配置模块，用于记录所述分布式系统中被指定的作为候选的None节点的一部分节点；
    None节点选择模块，用于在需要将多个Source节点的数据片汇聚到一个节点的情况下，在所述候选的None节点中选择一个节点作为None节点。
  7. 根据权利要求6所述的装置，其中，还包括Fixed节点选择模块，用于在需要将多个Source节点的数据片汇聚到多个Fixed节点的情况下，判断当前是否允许所述候选的None节点作为候选的Fixed节点，若是，则在所述分布式系统中随机选取多个节点作为Fixed节点，否则在所述分布式系统中所述候选的None节点之外随机选取多个节点作为Fixed节点。
  8. 根据权利要求6所述的装置，其中，还包括Source节点选择模块，用于在需要将分片的数据保存到Source节点的情况下，判断当前是否允许所述候选的None节点作为候选的Source节点，若是，则在所述分布式系统中随机选取多个节点作为Source节点，否则在所述分布式系统中所述候选的None节点之外随机选取多个节点作为Source节点。
  9. 根据权利要求8所述的装置，其中，所述Source节点选择模块还用于在当前采用硬件感知方式的情况下，在所述分布式系统中所述候选的None节点之外按照本地性原则选取多个节点作为Source节点。
  10. 根据权利要求8或9所述的装置，其中，所述Source节点选择模块还用于在当前采用硬件感知方式的情况下，在所述分布式系统中所述候选的None节点之外按照本地性原则选取多个节点作为Source节点。
PCT/CN2016/070551 2015-01-13 2016-01-11 在分布式系统中选取节点的方法和装置 Ceased WO2016112831A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510016624.7 2015-01-13
CN201510016624.7A CN104580476B (zh) 2015-01-13 2015-01-13 在分布式系统中选取节点的方法和装置

Publications (1)

Publication Number Publication Date
WO2016112831A1 true WO2016112831A1 (zh) 2016-07-21

Family

ID=53095633

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/070551 Ceased WO2016112831A1 (zh) 2015-01-13 2016-01-11 在分布式系统中选取节点的方法和装置

Country Status (2)

Country Link
CN (1) CN104580476B (zh)
WO (1) WO2016112831A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104580476B (zh) * 2015-01-13 2018-09-14 北京京东尚科信息技术有限公司 在分布式系统中选取节点的方法和装置

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100242042A1 (en) * 2006-03-13 2010-09-23 Nikhil Bansal Method and apparatus for scheduling work in a stream-oriented computer system
CN102572809A (zh) * 2010-12-27 2012-07-11 中国移动通信集团公司 一种选择网关节点的方法、系统和设备
CN104580476A (zh) * 2015-01-13 2015-04-29 北京京东尚科信息技术有限公司 在分布式系统中选取节点的方法和装置

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7710884B2 (en) * 2006-09-01 2010-05-04 International Business Machines Corporation Methods and system for dynamic reallocation of data processing resources for efficient processing of sensor data in a distributed network
CN100568823C (zh) * 2007-07-26 2009-12-09 北京航空航天大学 无线传感器网络节点管理方法
CN101924777B (zh) * 2009-06-17 2013-03-06 中国移动通信集团公司 在p2p流媒体系统中查找活动节点的方法、系统及设备
CN103188161B (zh) * 2011-12-30 2017-11-21 中国移动通信集团公司 分布式数据加载调度方法与系统
CN104168332A (zh) * 2014-09-01 2014-11-26 广东电网公司信息中心 高性能计算中负载均衡与节点状态监控方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100242042A1 (en) * 2006-03-13 2010-09-23 Nikhil Bansal Method and apparatus for scheduling work in a stream-oriented computer system
CN102572809A (zh) * 2010-12-27 2012-07-11 中国移动通信集团公司 一种选择网关节点的方法、系统和设备
CN104580476A (zh) * 2015-01-13 2015-04-29 北京京东尚科信息技术有限公司 在分布式系统中选取节点的方法和装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
CHEN, CHUN;: "The Application And Practice Of Presto In Meituan", PROGRAMMER, 30 June 2014 (2014-06-30), pages 98 - 102 *

Also Published As

Publication number Publication date
CN104580476A (zh) 2015-04-29
CN104580476B (zh) 2018-09-14

Similar Documents

Publication Publication Date Title
Hu et al. Time-and cost-efficient task scheduling across geo-distributed data centers
Nasir et al. When two choices are not enough: Balancing at scale in distributed stream processing
CN103150263B (zh) 分级存储方法
US9870270B2 (en) Realizing graph processing based on the mapreduce architecture
US11423082B2 (en) Methods and apparatus for subgraph matching in big data analysis
US20170124151A1 (en) Optimization of continuous queries in hybrid database and stream processing systems
US10866970B1 (en) Range query capacity allocation
CN103106249A (zh) 一种基于Cassandra的数据并行处理系统
US20190356530A1 (en) Moving nodes in a distributed system
CN111083179B (zh) 物联网云平台、基于物联网云平台的设备交互方法及装置
US9330158B1 (en) Range query capacity allocation
CN103377269B (zh) 传感器数据定位方法和装置
Labouseur et al. Scalable and Robust Management of Dynamic Graph Data.
CN103823846A (zh) 一种基于图论的大数据存储及查询方法
JP2016048536A (ja) クラスタコンピューティングシステム用のマスタデバイス、スレーブデバイスおよびそのコンピューティング方法
US12259828B2 (en) Forwarding incoming IO to SCM namespaces
US11568320B2 (en) Handling system-characteristics drift in machine learning applications
Dai et al. Research and implementation of big data preprocessing system based on Hadoop
WO2017206562A1 (zh) 一种数据表的处理方法、装置及系统
CN103823881B (zh) 分布式数据库的性能优化的方法及装置
CN113886036A (zh) 用于优化分布式系统集群配置的方法和系统
Yang et al. Automatic and Scalable Data Replication Manager in Distributed Computation and Storage Infrastructure of Cyber-Physical Systems.
US10841374B2 (en) Dynamic sharding for state-based processing
WO2016112831A1 (zh) 在分布式系统中选取节点的方法和装置
CN106716400A (zh) 一种数据表的分区管理方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16737057

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 07/11/2017)

122 Ep: pct application non-entry in european phase

Ref document number: 16737057

Country of ref document: EP

Kind code of ref document: A1