WO2013106993A1 - Capacity expansion method and device and data access method and device - Google Patents

Capacity expansion method and device and data access method and device Download PDF

Info

Publication number
WO2013106993A1
WO2013106993A1 PCT/CN2012/070464 CN2012070464W WO2013106993A1 WO 2013106993 A1 WO2013106993 A1 WO 2013106993A1 CN 2012070464 W CN2012070464 W CN 2012070464W WO 2013106993 A1 WO2013106993 A1 WO 2013106993A1
Authority
WO
WIPO (PCT)
Prior art keywords
node
virtual
physical
data
nodes
Prior art date
Application number
PCT/CN2012/070464
Other languages
French (fr)
Chinese (zh)
Inventor
印雨
潘孝忠
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Priority to CN201280000231.1A priority Critical patent/CN103348653B/en
Priority to PCT/CN2012/070464 priority patent/WO2013106993A1/en
Publication of WO2013106993A1 publication Critical patent/WO2013106993A1/en

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/104Peer-to-peer [P2P] networks

Definitions

  • P2P (Pear to Pear) storage devices based on DHT (Distributed Hash Table) technology have high scalability, support large-scale data storage, and can better cope with human data in the era of network information explosion. Ultra-large storage and management challenges. However, with the growth of storage data, the capacity of storage devices needs to be expanded to meet the growing demand for data storage.
  • DHT Distributed Hash Table
  • the storage device When the storage device is initialized, the storage device is allocated a specified number of virtual nodes in advance, and a mapping relationship between the physical node and the virtual node is established, so that there are multiple virtual nodes on one physical node.
  • the storage device When the storage device is expanded, that is, when the physical node increases, the virtual node is dynamically migrated to the new physical node. For example, before the capacity expansion, each physical node assumes 2N virtual nodes. After the capacity expansion, the physical nodes are doubled. If the load is balanced, each physical node generally assumes N virtual nodes. During the system expansion process, the total number of virtual nodes is always constant.
  • the total number of virtual nodes is preset and remains constant during the system expansion process.
  • a large number of virtual nodes are usually set in advance. Before the system is expanded, the number of virtual nodes corresponding to each physical node is large. When the number of data copies is constant, the physical nodes correspond. The more virtual nodes, the worse the reliability of the system; and, although the total number of pre-set virtual nodes is relatively large, as the system continues to expand, it will eventually expand to the point where no virtual nodes can be assigned to new physical nodes, making the system unable to Capacity expansion, system scalability is poor. Summary of the invention
  • an embodiment of the present invention provides a method and a storage device for capacity expansion.
  • the technical solution is as follows:
  • a method of expanding a volume comprising:
  • n is less than the preset value
  • the data managed by the virtual node and the new virtual node obtained by the splitting is determined, and the data that is not managed by the virtual node and the new virtual node that is split is deleted.
  • a determining module configured to determine the number of virtual nodes according to the number of physical nodes before the expansion, so that the physical nodes before each expansion correspond to n virtual nodes, where n is less than a preset value;
  • the splitting module is used to expand the physical node when the system is expanded, and some physical nodes appear corresponding to one virtual node.
  • the virtual node before the expansion is split, and a new virtual node is generated, so that the client is based on the split virtual node.
  • Quantity redefining the virtual node of the data map;
  • a migration module configured to migrate part of the virtual node to the newly added physical node
  • the deletion module is configured to determine data managed by the virtual node and the new virtual node that is split by the virtual node, and delete the data that is not managed by the virtual node and the new virtual node that is split by the virtual node. Based on the foregoing expansion method, a method for accessing data is also provided, where the method includes:
  • a client device, the client device includes:
  • An access module configured to determine a virtual node of the data mapping to be accessed, determining, according to the first correspondence, a first physical node where the data is located, and requesting access data from the first physical node;
  • the update module is further configured to: when the virtual node migrates, update a correspondence between the physical node and the virtual node to obtain a second correspondence;
  • the access module is further configured to determine, according to the second correspondence, a second physical node where the data is located, and request access data from the second physical node.
  • the virtual node is dynamically migrated by the virtual node splitting during the capacity expansion, which avoids the reliability problem caused by pre-setting a large number of virtual nodes, and improves the storage device.
  • the reliability while avoiding the problem of non-expansion caused by the fact that no virtual nodes can be allocated, improves the scalability of the storage device.
  • a corresponding access data scheme is provided.
  • Embodiment 1 is a flowchart of a method for expanding capacity provided by Embodiment 1 of the present invention
  • FIG. 4 is a schematic diagram of a partition of a storage device after splitting according to Embodiment 2 of the present invention.
  • this embodiment provides a method for expanding a capacity, which may be performed by a storage device, where the method includes:
  • a physical node When a system is expanded, a physical node is newly added, and a part of the physical node corresponds to a virtual node.
  • a new virtual node is generated, and the client re-determines the virtual node of the data mapping according to the number of the virtual nodes after the splitting;
  • each virtual node splits a new virtual node Pi splits P (i+N), that is, P0 splits P10, PI splits Pl l, and so on, P9 splits P19, and the total number of virtual nodes after splitting is 2N 20.
  • the virtual node corresponding to each physical node is doubled. Take Figure 3 as an example. After the split, each physical node corresponds to four virtual nodes. The corresponding relationship between the physical node and the virtual node after splitting is as follows:
  • the virtual nodes corresponding to each physical node before the splitting are evenly distributed, the virtual nodes corresponding to each physical node are still balanced after the splitting. Therefore, the read and write requests to the client do not cause data hotspot problems.
  • Hash represents a hash operation
  • key represents the identity of the data
  • mod represents a modulo operation
  • N is the number of virtual nodes before the split.
  • 2N represents the number of virtual nodes after the split.
  • the migration policy is not specifically limited, and any strategy for enabling the virtual nodes corresponding to the physical nodes to be substantially averaged is adapted to the embodiment. For example, calculate the average of the number of virtual nodes corresponding to each physical node, and the physical section The virtual node corresponding to the virtual node number exceeding the average value is migrated to the physical node whose corresponding virtual node number is smaller than the average value.
  • any strategy for enabling the virtual nodes corresponding to the physical nodes to be substantially averaged is adapted to the embodiment. For example, calculate the average of the number of virtual nodes corresponding to each physical node, and the physical section The virtual node corresponding to the virtual node number exceeding the average value is migrated to the physical node whose corresponding virtual node number is smaller than the average value.
  • two new physical nodes are added, and a total of seven physical nodes are added.
  • each physical node corresponds to an average of 20/7 «2. 85 logics.
  • Nodes that is, some physical nodes correspond to three virtual nodes, and some physical nodes correspond to two virtual nodes.
  • a total of five virtual nodes are migrated to the newly added physical node, and one physical node and one virtual node are migrated.
  • the corresponding relationship can be: Physical node 1: P0, P5, P10
  • the client device re-determines the virtual node of the data mapping to be written, determines a physical node where the data is located according to the first correspondence, and requests the physical node to write data.
  • step 202 For the virtual node splitting process, refer to step 202, and details are not described herein again.
  • the physical node 6 redirects the request to the physical node 1 after receiving the request from the client device to read the data.

Abstract

Embodiments of the present invention relate to the technical field of data storage, and provide a capacity expansion method and device and a data access method and device. The method comprises: determining, according to the number of physical nodes before capacity expansion, the number of virtual nodes, so that each physical node before capacity expansion corresponds to n virtual nodes, n being smaller than a preset value; when physical nodes are added during capacity expansion of the system and a part of the physical nodes each correspond to one virtual node, the virtual nodes before capacity expansion being split to generate new virtual nodes, so that the client re-determines, according to the number of the virtual nodes after splitting, the virtual nodes for data mapping; migrating a part of the virtual nodes to the newly added physical nodes; and determining data managed by the virtual nodes and the newly obtained virtual nodes after splitting, and respectively deleting data not managed by the virtual nodes and the newly obtained virtual nodes after splitting. By means of the above solution, the present invention improves the reliability and the expandability of a storage device.

Description

扩容的方法和设备、 以及访问数据的方法和设备 技术领域  Method and device for expanding capacity, and method and device for accessing data
本发明涉及数据存储领域, 特别涉及一种扩容的方法和设备、 以及访问数据的方法和 设备。  The present invention relates to the field of data storage, and in particular, to a method and device for expanding a capacity, and a method and device for accessing data.
 Say
背景技术 Background technique
基于 DHT (Distributed Hash Table, 分布式哈希表) 技术的 P2P (Pear to Pear, 点 对点) 存储设备拥有很高的扩展性, 支持大规模的数据存储, 可以较好地应对网络信息爆 炸时代人类数据超大规模的存储和管理的挑战。 但是书随着存储数据的大量增长, 存储设备 的容量需要不断扩大, 才能满足数据存储不断增长的需求。  P2P (Pear to Pear) storage devices based on DHT (Distributed Hash Table) technology have high scalability, support large-scale data storage, and can better cope with human data in the era of network information explosion. Ultra-large storage and management challenges. However, with the growth of storage data, the capacity of storage devices needs to be expanded to meet the growing demand for data storage.
在存储设备初始化时, 预先给存储设备分配指定个数的虚拟节点, 并建立物理节点和 虚拟节点的映射关系, 使得一个物理节点上有多个虚拟节点。 在存储设备扩容时, 即物理 节点增加时, 动态的迁移虚拟节点到新的物理节点上。 例如, 扩容前, 每个物理节点承担 2N个虚拟节点, 扩容后, 物理节点增加一倍, 如果负载均衡, 每个物理节点一般会承担 N 个虚拟节点。 在系统扩容过程中, 虚拟节点的总数始终保持恒定不变。  When the storage device is initialized, the storage device is allocated a specified number of virtual nodes in advance, and a mapping relationship between the physical node and the virtual node is established, so that there are multiple virtual nodes on one physical node. When the storage device is expanded, that is, when the physical node increases, the virtual node is dynamically migrated to the new physical node. For example, before the capacity expansion, each physical node assumes 2N virtual nodes. After the capacity expansion, the physical nodes are doubled. If the load is balanced, each physical node generally assumes N virtual nodes. During the system expansion process, the total number of virtual nodes is always constant.
在实现本发明的过程中, 发明人发现现有技术至少存在以下问题:  In the process of implementing the present invention, the inventors have found that the prior art has at least the following problems:
现有的扩容方案, 虚拟节点的总数预先设定后, 在系统扩容过程中始终保持恒定不变。 为了保证系统的可扩展性, 通常预先设定很大数量的虚拟节点, 在系统扩容前会使每个物 理节点对应的虚拟节点数量较多, 在数据副本数量一定的情况下, 物理节点对应的虚拟节 点越多, 系统的可靠性越差; 并且, 虽然预先设定的虚拟节点总数比较大, 但是随着系统 不断扩容, 最终会扩展到没有虚拟节点可以分配给新的物理节点, 使系统无法扩容, 系统 的可扩展性差。 发明内容  In the existing expansion plan, the total number of virtual nodes is preset and remains constant during the system expansion process. In order to ensure the scalability of the system, a large number of virtual nodes are usually set in advance. Before the system is expanded, the number of virtual nodes corresponding to each physical node is large. When the number of data copies is constant, the physical nodes correspond. The more virtual nodes, the worse the reliability of the system; and, although the total number of pre-set virtual nodes is relatively large, as the system continues to expand, it will eventually expand to the point where no virtual nodes can be assigned to new physical nodes, making the system unable to Capacity expansion, system scalability is poor. Summary of the invention
为了提高存储设备的可靠性和可扩展性, 本发明实施例提供了一种扩容的方法和存储 设备。 所述技术方案如下:  In order to improve the reliability and scalability of the storage device, an embodiment of the present invention provides a method and a storage device for capacity expansion. The technical solution is as follows:
一种扩容的方法, 所述方法包括:  A method of expanding a volume, the method comprising:
根据扩容前的物理节点的数量, 确定虚拟节点的数量, 使每个扩容前的物理节点对应 n 个虚拟节点, n小于预设值; Determine the number of virtual nodes based on the number of physical nodes before the expansion, and make the physical nodes before each expansion corresponding to n. Virtual nodes, n is less than the preset value;
当系统扩容时新增加了物理节点、 且部分物理节点出现对应一个虚拟节点的情况时, 扩容前的虚拟节点分裂, 产生新的虚拟节点, 使客户端根据分裂后的虚拟节点的数量, 重 新确定数据映射的虚拟节点;  When a physical node is added to the system and a physical node corresponds to a virtual node, the virtual node before the expansion is split, and a new virtual node is generated, so that the client re-determines according to the number of the virtual nodes after the split. Virtual node of data mapping;
迁移部分虚拟节点到新增加的物理节点;  Migrate some virtual nodes to newly added physical nodes;
确定虚拟节点和其分裂得到的新的虚拟节点分别管理的数据, 分别删除虚拟节点和其 分裂得到的新的虚拟节点上不属于自身管理的数据。  The data managed by the virtual node and the new virtual node obtained by the splitting is determined, and the data that is not managed by the virtual node and the new virtual node that is split is deleted.
一种存储设备, 所述存储设备包括:  A storage device, the storage device includes:
确定模块, 用于根据扩容前的物理节点的数量, 确定虚拟节点的数量, 使每个扩容前 的物理节点对应 n个虚拟节点, n小于预设值;  a determining module, configured to determine the number of virtual nodes according to the number of physical nodes before the expansion, so that the physical nodes before each expansion correspond to n virtual nodes, where n is less than a preset value;
分裂模块, 用于当系统扩容时新增加了物理节点、 且部分物理节点出现对应一个虚拟 节点的情况时, 扩容前的虚拟节点分裂, 产生新的虚拟节点, 使客户端根据分裂后的虚拟 节点的数量, 重新确定数据映射的虚拟节点;  The splitting module is used to expand the physical node when the system is expanded, and some physical nodes appear corresponding to one virtual node. The virtual node before the expansion is split, and a new virtual node is generated, so that the client is based on the split virtual node. Quantity, redefining the virtual node of the data map;
迁移模块, 用于迁移部分虚拟节点到新增加的物理节点;  a migration module, configured to migrate part of the virtual node to the newly added physical node;
删除模块, 用于确定虚拟节点和其分裂得到的新的虚拟节点分别管理的数据, 分别删 除虚拟节点和其分裂得到的新的虚拟节点上不属于自身管理的数据。 基于上述扩容方法, 还提供了一种访问数据的方法, 所述方法包括:  The deletion module is configured to determine data managed by the virtual node and the new virtual node that is split by the virtual node, and delete the data that is not managed by the virtual node and the new virtual node that is split by the virtual node. Based on the foregoing expansion method, a method for accessing data is also provided, where the method includes:
当虚拟节点分裂时, 更新物理节点和虚拟节点的对应关系, 得到第一对应关系; 确定欲访问的数据映射的虚拟节点, 根据所述第一对应关系, 确定所述数据所在的第 一物理节点, 向所述第一物理节点请求访问数据;  When the virtual node is split, the corresponding relationship between the physical node and the virtual node is updated, and the first correspondence is obtained. The virtual node that determines the data mapping to be accessed is determined according to the first correspondence, and the first physical node where the data is located is determined. Requiring access data to the first physical node;
当虚拟节点迁移时, 更新物理节点和虚拟节点的对应关系, 得到第二对应关系; 根据所述第二对应关系, 确定所述数据所在的第二物理节点, 向所述第二物理节点请 求访问数据。  Updating the correspondence between the physical node and the virtual node to obtain a second correspondence relationship when the virtual node is migrated; determining, according to the second correspondence, the second physical node where the data is located, and requesting access to the second physical node data.
一种客户端设备, 所述客户端设备包括:  A client device, the client device includes:
更新模块, 用于当虚拟节点分裂时, 更新物理节点和虚拟节点的对应关系, 得到第一 对应关系;  An update module, configured to update a correspondence between a physical node and a virtual node when the virtual node is split, to obtain a first correspondence relationship;
访问模块, 用于确定欲访问的数据映射的虚拟节点, 根据所述第一对应关系, 确定所 述数据所在的第一物理节点, 向所述第一物理节点请求访问数据;  An access module, configured to determine a virtual node of the data mapping to be accessed, determining, according to the first correspondence, a first physical node where the data is located, and requesting access data from the first physical node;
所述更新模块, 还用于当虚拟节点迁移时, 更新物理节点和虚拟节点的对应关系, 得 到第二对应关系; 所述访问模块, 还用于根据所述第二对应关系, 确定所述数据所在的第二物理节点, 向所述第二物理节点请求访问数据。 The update module is further configured to: when the virtual node migrates, update a correspondence between the physical node and the virtual node to obtain a second correspondence; The access module is further configured to determine, according to the second correspondence, a second physical node where the data is located, and request access data from the second physical node.
本发明实施例提供的技术方案的有益效果是:  The beneficial effects of the technical solutions provided by the embodiments of the present invention are:
在每个物理节点对应较少虚拟节点的情况下, 扩容时通过虚拟节点分裂的方式, 动态 的迁移虚拟节点, 避免了预先设定很大数量的虚拟节点引起的可靠性问题, 提高了存储设 备的可靠性, 同时避免了没有虚拟节点可以分配引起的无法扩容问题, 提高了存储设备的 可扩展性。 并且, 基于上述扩容方法, 提供一种相应的访问数据方案。 附图说明  In the case that each physical node corresponds to a small number of virtual nodes, the virtual node is dynamically migrated by the virtual node splitting during the capacity expansion, which avoids the reliability problem caused by pre-setting a large number of virtual nodes, and improves the storage device. The reliability, while avoiding the problem of non-expansion caused by the fact that no virtual nodes can be allocated, improves the scalability of the storage device. And, based on the foregoing expansion method, a corresponding access data scheme is provided. DRAWINGS
为了更清楚地说明本发明实施例中的技术方案, 下面将对实施例描述中所需要使用的 附图作简单地介绍, 显而易见地, 下面描述中的附图仅仅是本发明的一些实施例, 对于本 领域普通技术人员来讲, 在不付出创造性劳动的前提下, 还可以根据这些附图获得其他的 附图。  In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly described. It is obvious that the drawings in the following description are only some embodiments of the present invention. Other drawings may also be obtained from those of ordinary skill in the art in view of the drawings.
图 1是本发明实施例 1提供的扩容的方法流程图;  1 is a flowchart of a method for expanding capacity provided by Embodiment 1 of the present invention;
图 2是本发明实施例 2提供的扩容的方法流程图;  2 is a flowchart of a method for expanding capacity provided by Embodiment 2 of the present invention;
图 3是本发明实施例 2提供的分裂前存储设备分区示意图;  3 is a schematic diagram of a partition of a pre-split memory device according to Embodiment 2 of the present invention;
图 4是本发明实施例 2提供的分裂后存储设备分区示意图;  4 is a schematic diagram of a partition of a storage device after splitting according to Embodiment 2 of the present invention;
图 5是本发明实施例 3提供的访问数据的方法流程图;  5 is a flowchart of a method for accessing data provided by Embodiment 3 of the present invention;
图 6是本发明实施例 3提供的客户端设备写入数据过程示意图;  6 is a schematic diagram of a process of writing data by a client device according to Embodiment 3 of the present invention;
图 7是本发明实施例 3提供的客户端设备读取数据过程示意图;  7 is a schematic diagram of a process of reading data by a client device according to Embodiment 3 of the present invention;
图 8是本发明实施例 4提供的存储设备结构示意图;  8 is a schematic structural diagram of a storage device according to Embodiment 4 of the present invention;
图 9是本发明实施例 5提供客户端设备结构示意图。 具体实施方式  FIG. 9 is a schematic structural diagram of a client device according to Embodiment 5 of the present invention. detailed description
为使本发明的目的、 技术方案和优点更加清楚, 下面将结合附图对本发明实施方式作 进一步地详细描述。  The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
实施例 1  Example 1
参见图 1,本实施例提供了一种扩容的方法,该方法可以由存储设备执行,该方法包括: Referring to FIG. 1, this embodiment provides a method for expanding a capacity, which may be performed by a storage device, where the method includes:
101: 根据扩容前的物理节点的数量, 确定虚拟节点的数量, 使每个扩容前的物理节点 对应 n个虚拟节点, n小于预设值; 101: Determine, according to the number of physical nodes before the expansion, the number of virtual nodes, so that the physical nodes before each expansion correspond to n virtual nodes, where n is smaller than a preset value;
102: 当系统扩容时新增加了物理节点、 且部分物理节点出现对应一个虚拟节点的情况 时, 扩容前的虚拟节点分裂, 产生新的虚拟节点, 使客户端根据分裂后的虚拟节点的数量, 重新确定数据映射的虚拟节点; 102: When a system is expanded, a physical node is newly added, and a part of the physical node corresponds to a virtual node. When the virtual node is expanded before the expansion, a new virtual node is generated, and the client re-determines the virtual node of the data mapping according to the number of the virtual nodes after the splitting;
103: 迁移部分虚拟节点到新增加的物理节点;  103: Migrate part of the virtual node to the newly added physical node;
104 :确定虚拟节点和其分裂得到的新的虚拟节点分别管理的数据, 分别删除虚拟节点 和其分裂得到的新的虚拟节点上不属于自身管理的数据。  104: Determine data respectively managed by the virtual node and the new virtual node that is split by the virtual node, and delete the data that is not managed by the virtual node and the new virtual node that is split by the virtual node.
本实施例在每个物理节点对应较少虚拟节点的情况下, 扩容时通过虚拟节点分裂的方 式, 动态的迁移虚拟节点, 避免了预先设定很大数量的虚拟节点引起的可靠性问题, 提高 了存储设备的可靠性, 同时避免了没有虚拟节点可以分配引起的无法扩容问题, 提高了存 储设备的可扩展性。 实施例 2  In this embodiment, when each physical node corresponds to a small number of virtual nodes, the virtual node is dynamically migrated by the virtual node splitting during the capacity expansion, thereby avoiding the reliability problem caused by presetting a large number of virtual nodes, and improving The reliability of the storage device is avoided, and the problem of non-expansion caused by the fact that no virtual node can be allocated is avoided, and the scalability of the storage device is improved. Example 2
参见图 2,本实施例提供了一种扩容的方法,该方法可以由存储设备执行,该方法包括: 201: 根据扩容前的物理节点的数量, 确定虚拟节点的数量, 使每个扩容前的物理节点 对应 n个虚拟节点, n小于预设值;  Referring to FIG. 2, this embodiment provides a method for expanding a capacity, which may be performed by a storage device. The method includes: 201: Determine the number of virtual nodes according to the number of physical nodes before the expansion, so that each expansion is performed. The physical node corresponds to n virtual nodes, and n is smaller than a preset value;
其中, 预设值为现有技术中每个物理节点对应的虚拟节点的数量。 本实施例中 n 的典 型值可以为 2或 3, 该典型值通常远小于现有技术中每个物理节点对应的虚拟节点的数量。 参见图 3, 以 5个物理节点为例, 共有 10个虚拟节点, 分别为 P0、 Pl、 P2、 P3、 P4、 P5、 P6、 P7、 P8、 P9, 每个物理节点对应 2 个虚拟节点, 物理节点和虚拟节点的对应关系具体 如下:  The default value is the number of virtual nodes corresponding to each physical node in the prior art. The typical value of n in this embodiment may be 2 or 3. The typical value is usually much smaller than the number of virtual nodes corresponding to each physical node in the prior art. Referring to Figure 3, taking 5 physical nodes as an example, there are 10 virtual nodes, which are P0, Pl, P2, P3, P4, P5, P6, P7, P8, and P9, and each physical node corresponds to 2 virtual nodes. The correspondence between the physical node and the virtual node is as follows:
物理节点 1 : P0、 P5  Physical node 1 : P0, P5
物理节点 2: Pl、 P6  Physical node 2: Pl, P6
物理节点 3: P2、 P7  Physical node 3: P2, P7
物理节点 4: P3、 P8  Physical node 4: P3, P8
物理节点 5 : P4、 P9  Physical node 5 : P4, P9
当每份数据有 M个数据备份 (或称副本) 时, 一个物理节点发生故障时, 并发度可以 达到 n X (M-1 ) , 即故障物理节点上的 n个虚拟节点可以并发的从 (M-1 ) 个备份节点上恢 复出数据。 通过减少每个物理节点对应的虚拟节点的数量, 可以提高系统的可靠性。 这是 因为, 系统的可靠性和每个物理节点对应的虚拟节点的数量有严重的依赖关系, 一个物理 节点发生故障, 该故障物理节点对应的所有虚拟节点的副本所在物理节点如果同时发生故 障, 会导致系统不可用。 因此, 减少每个物理节点对应的虚拟节点的数量, 减小并发度, 可以提高系统的可靠性。 202: 当系统扩容时新增加了物理节点、 且部分物理节点出现对应一个虚拟节点的情况 时, 扩容前的虚拟节点分裂, 产生新的虚拟节点; When each data has M data backups (or copies), when a physical node fails, the concurrency can reach n X (M-1 ), that is, the n virtual nodes on the failed physical node can concurrently ( Data is recovered on M-1) backup nodes. By reducing the number of virtual nodes corresponding to each physical node, the reliability of the system can be improved. This is because the reliability of the system has a serious dependence on the number of virtual nodes corresponding to each physical node. If a physical node fails, the physical node of the replica of all the virtual nodes corresponding to the failed physical node fails at the same time. Will cause the system to be unavailable. Therefore, reducing the number of virtual nodes corresponding to each physical node and reducing the degree of concurrency can improve the reliability of the system. 202: When a physical node is newly added when the system is expanded, and a part of the physical node corresponds to one virtual node, the virtual node before the expansion is split, and a new virtual node is generated;
参见图 4所示的分裂示意图, 设虚拟节点为 Pi, 分裂前 i=0、 1、 2、 …、 9, 分裂前虚 拟节点的总数 N=10, 每个虚拟节点分裂出一个新的虚拟节点, Pi分裂出 P (i+N), 即 P0分 裂出 P10, PI分裂出 Pl l, 依次类推, P9分裂出 P19, 分裂后虚拟节点的总数为 2N=20。 分 裂后, 每个物理节点对应的虚拟节点增加一倍, 以图 3 为例, 分裂后每个物理节点对应 4 个虚拟节点, 分裂后物理节点和虚拟节点的对应关系具体如下:  Referring to the split diagram shown in Figure 4, let the virtual node be Pi, before the split i = 0, 1, 2, ..., 9, the total number of virtual nodes before splitting N = 10, each virtual node splits a new virtual node Pi splits P (i+N), that is, P0 splits P10, PI splits Pl l, and so on, P9 splits P19, and the total number of virtual nodes after splitting is 2N=20. After the splitting, the virtual node corresponding to each physical node is doubled. Take Figure 3 as an example. After the split, each physical node corresponds to four virtual nodes. The corresponding relationship between the physical node and the virtual node after splitting is as follows:
物理节点 1 : P0、 P5、 P10、 P15  Physical node 1 : P0, P5, P10, P15
物理节点 2: Pl、 P6、 Pl l、 P16  Physical node 2: Pl, P6, Pl l, P16
物理节点 3: P2、 P7、 P12、 P17  Physical node 3: P2, P7, P12, P17
物理节点 4: P3、 P8、 P13、 P18  Physical node 4: P3, P8, P13, P18
物理节点 5: P4、 P9、 P14、 P19  Physical node 5: P4, P9, P14, P19
需要说明的是, 每个虚拟节点与其分裂出的新的虚拟节点共同指向同一份物理节点上 的数据, 即 Pi与 P (i+N) 共同指向同一份物理节点上的数据, 例如, P0与 P10共同指向同 一份物理节点上的数据, P1与 P11共同指向同一份物理节点上的数据等。  It should be noted that each virtual node and its split new virtual node point to the same physical node data, that is, Pi and P (i+N) jointly point to the data on the same physical node, for example, P0 and P10 points to the same physical node, and P1 and P11 point to the same physical node.
由于分裂前每个物理节点对应的虚拟节点是均衡分布的, 分裂后每个物理节点对应的 虚拟节点仍然是均衡分布的, 因此, 对客户端的读写请求不会造成数据热点问题。  Since the virtual nodes corresponding to each physical node before the splitting are evenly distributed, the virtual nodes corresponding to each physical node are still balanced after the splitting. Therefore, the read and write requests to the client do not cause data hotspot problems.
203: 根据分裂后的虚拟节点的数量, 重新确定数据映射的虚拟节点;  203: Re-determine the virtual node of the data mapping according to the number of the virtual nodes after the splitting;
具体的, 对数据的标识进行哈希运算得到哈希值, 将哈希值对分裂后的虚拟节点总数 取模得到模值, 根据模值确定该数据对应的虚拟节点。 公式表示如下:  Specifically, the hash of the data is hashed to obtain a hash value, and the hash value is modulo the total number of the split virtual nodes to obtain a modulus value, and the virtual node corresponding to the data is determined according to the modulus value. The formula is expressed as follows:
Hash (key) mod (2N) ,  Hash (key) mod (2N) ,
其中, Hash表示哈希运算, key表示数据的标识, mod表示取模运算, N为分裂前虚拟 节点的数量, 相应的, 2N表示分裂后虚拟节点的数量。  Hash represents a hash operation, key represents the identity of the data, mod represents a modulo operation, and N is the number of virtual nodes before the split. Correspondingly, 2N represents the number of virtual nodes after the split.
由于虚拟节点与其分裂得到的新的虚拟节点指向同一份数据, 因此, 上述重新映射过 程中, 不会造成数据的指向错误。 例如, Hash (abc) mod (N) =0, 即分裂前 key为 abc的数据 被映射到虚拟节点 P0, Hash (abc) mod (2N) =10, 即分裂后 key为 abc的数据被映射到虚拟节 点 P10, 由于虚拟节点 P10是由虚拟节点 P0分裂得到, 虚拟节点 P10和虚拟节点 P0指向同 一份数据, 因此, 在重新映射过程中不会造成数据的指向错误。  Since the virtual node and the new virtual node obtained by the split point to the same data, the above remapping process does not cause data pointing errors. For example, Hash (abc) mod (N) =0, that is, the data before the split key abc is mapped to the virtual node P0, Hash (abc) mod (2N) = 10, that is, the data after the split key abc is mapped to The virtual node P10, since the virtual node P10 is split by the virtual node P0, the virtual node P10 and the virtual node P0 point to the same data, and therefore, the data pointing error is not caused in the remapping process.
204: 迁移部分虚拟节点到新增加的物理节点;  204: Migrate part of the virtual node to the newly added physical node;
本实施例对迁移策略不做具体限定, 任何使物理节点对应的虚拟节点能够大致平均的 策略都适应于本实施例。 例如, 计算每个物理节点对应虚拟节点数量的平均值, 将物理节 点对应虚拟节点数量超过该平均值的虚拟节点迁移到对应虚拟节点数量小于该平均值的物 理节点上。 下面结合一具体实例进行说明。 In this embodiment, the migration policy is not specifically limited, and any strategy for enabling the virtual nodes corresponding to the physical nodes to be substantially averaged is adapted to the embodiment. For example, calculate the average of the number of virtual nodes corresponding to each physical node, and the physical section The virtual node corresponding to the virtual node number exceeding the average value is migrated to the physical node whose corresponding virtual node number is smaller than the average value. The following description will be made in conjunction with a specific example.
分裂后每个物理节点平均对应 20/5=4个虚拟节点, 系统扩容时新增加 2个物理节点, 共 7个物理节点, 扩容后每个物理节点平均对应 20/7 «2. 85个逻辑节点, 即部分物理节点 对应 3个虚拟节点, 部分物理节点对应 2个虚拟节点, 为了减少数据迁移量, 一共迁移 5 个虚拟节点到新增加的物理节点上, 迁移后物理节点和虚拟节点的一种对应关系可以为: 物理节点 1 : P0、 P5、 P10  After splitting, each physical node corresponds to an average of 20/5=4 virtual nodes. When the system is expanded, two new physical nodes are added, and a total of seven physical nodes are added. After expansion, each physical node corresponds to an average of 20/7 «2. 85 logics. Nodes, that is, some physical nodes correspond to three virtual nodes, and some physical nodes correspond to two virtual nodes. To reduce the amount of data migration, a total of five virtual nodes are migrated to the newly added physical node, and one physical node and one virtual node are migrated. The corresponding relationship can be: Physical node 1: P0, P5, P10
物理节点 2: Pl、 P6、 P16  Physical node 2: Pl, P6, P16
物理节点 3: P2、 P12、 P17  Physical node 3: P2, P12, P17
物理节点 4: P8、 P13、 P18  Physical node 4: P8, P13, P18
物理节点 5: P4、 P14、 P19  Physical node 5: P4, P14, P19
物理节点 6: P15, Pl l 、 P7  Physical node 6: P15, Pl l, P7
物理节点 7: P3、 P9  Physical node 7: P3, P9
205: 确定虚拟节点和其分裂得到的新的虚拟节点分别管理的数据, 分别删除虚拟节点 和其分裂得到的新的虚拟节点上不属于自身管理的数据。  205: Determine data respectively managed by the virtual node and the new virtual node that is split by the virtual node, and delete data that is not managed by the virtual node and the new virtual node that is split by the virtual node.
由于虚拟节点和其分裂得到的新的虚拟节点指向同一份数据, 相当于虚拟节点和其分 裂得到的新的虚拟节点上各有一半的数据是无效的, 因此, 迁移后删除虚拟节点和其分裂 得到的新的虚拟节点上不属于自身管理的数据, 可以提升磁盘的利用率。  Since the virtual node and the new virtual node obtained by its split point to the same data, half of the data on the virtual node and the new virtual node that is split is invalid. Therefore, the virtual node and its split are deleted after the migration. The obtained new virtual node does not belong to its own management data, which can improve the utilization of the disk.
本实施例在每个物理节点对应较少虚拟节点的情况下, 扩容时通过虚拟节点分裂的方 式, 动态的迁移虚拟节点, 避免了预先设定很大数量的虚拟节点引起的可靠性问题, 提高 了存储设备的可靠性, 同时避免了没有虚拟节点可以分配引起的无法扩容问题, 提高了存 储设备的可扩展性。 实施例 3  In this embodiment, when each physical node corresponds to a small number of virtual nodes, the virtual node is dynamically migrated by the virtual node splitting during the capacity expansion, thereby avoiding the reliability problem caused by presetting a large number of virtual nodes, and improving The reliability of the storage device is avoided, and the problem of non-expansion caused by the fact that no virtual node can be allocated is avoided, and the scalability of the storage device is improved. Example 3
基于本发明实施例提供的扩容方法, 参见图 5, 本实施例提供了一种访问数据的方法, 该方法可以由客户端设备执行, 该方法包括:  The method for accessing data according to the embodiment of the present invention is as follows: Referring to FIG. 5, the embodiment provides a method for accessing data, and the method may be performed by a client device, where the method includes:
301: 当虚拟节点分裂时, 更新物理节点和虚拟节点的对应关系, 得到第一对应关系; 301: When the virtual node is split, update the correspondence between the physical node and the virtual node to obtain a first correspondence.
302: 确定欲访问的数据映射的虚拟节点, 根据第一对应关系, 确定该数据所在的第一 物理节点, 向该第一物理节点请求访问数据; 302: Determine a virtual node of the data mapping to be accessed, determine, according to the first correspondence, a first physical node where the data is located, and request access data from the first physical node;
303: 当虚拟节点迁移时, 更新物理节点和虚拟节点的对应关系, 得到第二对应关系; 303: When the virtual node migrates, update the correspondence between the physical node and the virtual node to obtain a second correspondence.
304 :根据第二对应关系, 确定该数据所在的第二物理节点, 向该第二物理节点请求访 问数据。 304: Determine, according to the second correspondence, the second physical node where the data is located, and request the second physical node to visit Ask the data.
在步骤 304 向该第二物理节点请求访问数据之后, 如果该数据还没有从第一物理节点 上迁移到第二物理节点, 第二物理节点从第一物理节点获取该数据。  After requesting access to the second physical node at step 304, if the data has not been migrated from the first physical node to the second physical node, the second physical node acquires the data from the first physical node.
在步骤 301之前, 该方法进一步还包括:  Before step 301, the method further includes:
确定欲访问的数据映射的虚拟节点, 根据第三对应关系, 确定该数据所在的第一物理 节点, 向该第一物理节点请求访问数据, 第三对应关系是虚拟节点分裂之前, 物理节点和 虚拟节点的对应关系。  Determining a virtual node of the data mapping to be accessed, determining, according to the third correspondence, the first physical node where the data is located, requesting access data from the first physical node, where the third correspondence relationship is before the virtual node is split, the physical node and the virtual node The correspondence of nodes.
下面以客户端设备写入数据过程 (put ) 和读取数据过程 (get ) 为例进行说明。  The following takes the client device write data process (put) and the read data process (get) as an example.
参见图 6 所示的客户端设备写入数据过程, 该过程包括分裂前、 分裂后、 以及迁移虚 拟节点到新增加的物理节点的整个数据写入过程。 例如各阶段物理节点和虚拟节点的对应 关系分别如下:  See the client device write data process shown in Figure 6, which includes pre-split, split, and the entire data write process of migrating the virtual node to the newly added physical node. For example, the correspondence between physical nodes and virtual nodes in each phase is as follows:
分裂前 (第三对应关系) 分裂后 (第- -对应关系) 迁移后 (第二对应关系) 物理节点 1 : P0、 P5 物理节点 1 : P0、 P5、 P10、 P15 物理节点 1 : P0、 P5、 P10 物理节点 2: Pl、 P6 物理节点 2: Pl、 P6、 Pl l、 P16 物理节点 2: Pl、 P6、 P16 物理节点 3: P2、 P7 物理节点 3: P2、 P7、 P12、 P17 物理节点 3: P2、 P12、 P17 物理节点 4: P3、 P8 物理节点 4: P3、 P8、 P13、 P18 物理节点 4: P8、 P13、 P18 物理节点 5: P4、 P9 物理节点 5: P4、 P9、 P14、 P19 物理节点 5: P4、 P14、 P19 物理节点 6: P15、 Pl l、 P7 物理节点 7: P3、 P9 401: 客户端设备确定欲写入的数据映射的虚拟节点, 根据第三对应关系, 确定该数据 所在的物理节点, 向该物理节点请求写入数据; Before splitting (third correspondence) After splitting (first-corresponding relationship) After migration (second correspondence) Physical node 1: P0, P5 Physical node 1: P0, P5, P10, P15 Physical node 1: P0, P5 P10 Physical node 2: Pl, P6 Physical node 2: Pl, P6, Pl l, P16 Physical node 2: Pl, P6, P16 Physical node 3: P2, P7 Physical node 3: P2, P7, P12, P17 Physical node 3: P2, P12, P17 Physical Node 4: P3, P8 Physical Node 4: P3, P8, P13, P18 Physical Node 4: P8, P13, P18 Physical Node 5: P4, P9 Physical Node 5: P4, P9, P14 P19 Physical node 5: P4, P14, P19 Physical node 6: P15, Pl l, P7 Physical node 7: P3, P9 401: The client device determines the virtual node of the data mapping to be written, according to the third correspondence, Determining the physical node where the data is located, requesting write data to the physical node;
其中, 第三对应关系是虚拟节点分裂之前, 物理节点和虚拟节点的对应关系。  The third correspondence relationship is a correspondence between the physical node and the virtual node before the virtual node is split.
设欲写入的数据 key=abc, value=l l l。 根据公式 Hash (key) mod (N), 确定该数据映射 的虚拟节点。 设 Hash (abc) =15, Hash (abc) mod (10) =5, 即该数据映射到虚拟节点 P5, 根据 第三对应关系可知, 请求写入的数据(key=abc, value=l l l)所在的物理节点为物理节点 1。 Set the data to be written key=abc, value=lll. Determine the virtual node of the data map according to the formula Hash (key) mod (N). Let Hash (abc) =15, Hash (abc) mod (10) =5, that is, the data is mapped to the virtual node P5, according to the third correspondence, the data requested to be written (key= a bc, value=lll) The physical node is physical node 1.
402: 虚拟节点分裂, 客户端设备更新物理节点和虚拟节点的对应关系, 得到第一对应 关系;  402: The virtual node is split, and the client device updates the correspondence between the physical node and the virtual node to obtain a first correspondence.
其中, 虚拟节点分裂过程参见步骤 202, 这里不再赘述。  For the virtual node splitting process, refer to step 202, and details are not described herein again.
403: 客户端设备重新确定欲写入的数据映射的虚拟节点, 根据第一对应关系, 确定该 数据所在的物理节点, 向该物理节点请求写入数据;  403: The client device re-determines the virtual node of the data mapping to be written, determines a physical node where the data is located according to the first correspondence, and requests the physical node to write data.
根据公式 Hash (key) mod (2N), 确定分裂后该数据所在的物理节点。 设 Hash (abc) =15, Hash (abc) mod (20) =15, 即该数据映射到虚拟节点 P15, 根据第一对应关系可知, 请求写入 的数据(key=abc, value=l l l)所在的物理节点仍然为物理节点 1。 同时由于 P15是 P5分裂 得到的, 所以客户端设备的数据写到了正确的物理节点上的正确的虚拟节点上。 According to the formula Hash (key) mod (2N), the physical node where the data is located after the split is determined. Let Hash (abc) = 15, Hash (abc) mod (20) = 15, that is, the data is mapped to the virtual node P15. According to the first correspondence, the physical node where the data requested to be written (key=abc, value=lll) is still the physical node 1 . At the same time, since P15 is obtained by splitting P5, the data of the client device is written to the correct virtual node on the correct physical node.
404: 虚拟节点迁移, 客户端设备更新物理节点和虚拟节点的对应关系, 得到第二对应 关系;  404: The virtual node is migrated, and the client device updates the correspondence between the physical node and the virtual node to obtain a second correspondence.
其中, 虚拟节点迁移过程参见步骤 204, 这里不再赘述。 例如, 将虚拟节点 P15由物理 节点 1迁移到物理节点 6。  For the virtual node migration process, refer to step 204, and details are not described herein. For example, virtual node P15 is migrated from physical node 1 to physical node 6.
405: 客户端设备根据第二对应关系, 确定该数据所在的物理节点, 向该物理节点请求 写入数据。  405: The client device determines, according to the second correspondence, the physical node where the data is located, and requests the physical node to write data.
请求写入的数据(key=abc, value=l l l)映射到虚拟节点 P15, 根据迁移后物理节点和虚 拟节点的对应关系可知, 虚拟节点 P15在物理节点 6上, 向物理节点 6请求写入数据。 参见图 7 所示的客户端设备读取数据过程, 该过程包括分裂前、 分裂后、 以及迁移虚 拟节点到新增加的物理节点的整个数据读取过程。 设客户端设备读取数据过程中各阶段物 理节点和虚拟节点的对应关系与客户端设备写入数据过程中的相同。  The data requested to be written (key=abc, value=lll) is mapped to the virtual node P15. According to the correspondence between the physical node and the virtual node after the migration, the virtual node P15 requests the physical node 6 to write data to the physical node 6. . See the client device read data process shown in Figure 7, which includes pre-splitting, splitting, and the entire data reading process of migrating the virtual node to the newly added physical node. It is assumed that the correspondence between the physical node and the virtual node in each stage of the process of reading data by the client device is the same as that in the process of writing data by the client device.
501: 客户端设备确定欲读取的数据映射的虚拟节点, 根据第三对应关系, 确定该数据 所在的物理节点, 向该物理节点请求读取数据;  501: The client device determines a virtual node of the data mapping to be read, determines a physical node where the data is located according to the third correspondence, and requests the physical node to read the data.
设欲读取的数据 key=abc, value=l l l。 根据公式 Hash (key) mod (N), 确定该数据映射 的虚拟节点。 设 Hash (abc) mod (10) =5, 即该数据映射到虚拟节点 P5, 根据第三对应关系可 知, 请求读取的数据(key=abc, value=l l l)所在的物理节点为物理节点 1。  Set the data to be read key=abc, value=l l l. Determine the virtual node of the data map according to the formula Hash (key) mod (N). Let Hash (abc) mod (10) = 5, that is, the data is mapped to the virtual node P5. According to the third correspondence, the physical node where the data requested to be read (key=abc, value=lll) is the physical node 1 .
502: 虚拟节点分裂, 客户端设备更新物理节点和虚拟节点的对应关系, 得到第一对应 关系;  502: The virtual node is split, and the client device updates the correspondence between the physical node and the virtual node to obtain a first correspondence.
其中, 虚拟节点分裂过程参见步骤 202, 这里不再赘述。  For the virtual node splitting process, refer to step 202, and details are not described herein again.
503: 客户端设备重新确定欲读取的数据映射的虚拟节点, 根据第一对应关系, 确定该 数据所在的物理节点, 向该物理节点请求读取数据;  503: The client device re-determines the virtual node of the data mapping to be read, determines a physical node where the data is located according to the first correspondence, and requests the physical node to read the data.
根据公式 Hash (key) mod (2N) , 确定分裂后该数据所在的物理节点。 设 Hash (abc) mod (20) =15, 即该数据映射到虚拟节点 P15, 根据第一对应关系可知, 请求读取 的数据(key=abc, value=l l l)所在的物理节点仍然为物理节点 1。 同时由于 P15是 P5分裂 得到的, 所以客户端设备可以正确的读取到数据。  According to the formula Hash (key) mod (2N) , the physical node where the data is located after the split is determined. Let Hash (abc) mod (20) = 15, that is, the data is mapped to the virtual node P15. According to the first correspondence, the physical node where the data requested to be read (key=abc, value=lll) is still a physical node. 1. At the same time, since P15 is obtained by splitting P5, the client device can correctly read the data.
504: 虚拟节点迁移, 客户端设备更新物理节点和虚拟节点的对应关系, 得到第二对应 关系; 其中, 虚拟节点迁移过程参见步骤 204, 这里不再赘述。 例如, 将虚拟节点 P15由物理 节点 1迁移到物理节点 6。 504: The virtual node is migrated, and the client device updates the correspondence between the physical node and the virtual node to obtain a second correspondence. For the virtual node migration process, refer to step 204, and details are not described herein. For example, the virtual node P15 is migrated from the physical node 1 to the physical node 6.
505: 客户端设备根据欲读取的数据映射的虚拟节点、 以及第二对应关系, 确定该数据 所在的物理节点, 向该物理节点请求读取数据。  505: The client device determines, according to the virtual node mapped by the data to be read, and the second corresponding relationship, the physical node where the data is located, and requests the physical node to read the data.
请求读取的数据(key=abc, value=l l l)映射到虚拟节点 P15, 根据第二对应关系可知, 虚拟节点 P15在物理节点 6上, 向物理节点 6请求读取数据。 The data requested to be read (key= a bc, value=l11) is mapped to the virtual node P15. According to the second correspondence, the virtual node P15 requests the physical node 6 to read data from the physical node 6.
进一步的, 如果欲读取的数据还没有从之前的物理节点上迁移到该物理节点, 则该物 理节点将读取数据的请求重定向 (或者说转发) 到之前的物理节点, 并从之前的物理节点 获取该数据, 保证客户端设备仍然能够读取到数据。  Further, if the data to be read has not been migrated from the previous physical node to the physical node, the physical node redirects (or forwards) the request for reading the data to the previous physical node, and from the previous The physical node obtains the data to ensure that the client device can still read the data.
例如, 欲读取的数据还没有从物理节点 1迁移到物理节点 6, 则物理节点 6接收到客户 端设备读取数据的请求后, 将该请求重定向到物理节点 1。  For example, if the data to be read has not been migrated from the physical node 1 to the physical node 6, the physical node 6 redirects the request to the physical node 1 after receiving the request from the client device to read the data.
本实施例基于本发明实施例提供的扩容方法, 提供一种相应的访问数据方案。 并且, 在虚拟节点分裂过程中, 虚拟节点与其分裂得到的新的虚拟节点对应同一物理节点, 因此, 在扩容过程中, 客户端设备仍然能正确访问到数据, 上层业务不会受到影响。 实施例 4  This embodiment provides a corresponding access data scheme based on the capacity expansion method provided by the embodiment of the present invention. In addition, in the virtual node splitting process, the virtual node and the new virtual node that is split are the same physical node. Therefore, during the capacity expansion process, the client device can still access the data correctly, and the upper layer service is not affected. Example 4
参见图 8, 本实施例提供了一种存储设备, 存储设备包括:  Referring to FIG. 8, the embodiment provides a storage device, where the storage device includes:
确定模块 601, 用于根据扩容前的物理节点的数量, 确定虚拟节点的数量, 使每个扩容 前的物理节点对应 n个虚拟节点, n小于预设值;  The determining module 601 is configured to determine the number of virtual nodes according to the number of physical nodes before the expansion, so that the physical nodes before each expansion correspond to n virtual nodes, where n is smaller than a preset value;
分裂模块 602, 用于当系统扩容时新增加了物理节点、且部分物理节点出现对应一个虚 拟节点的情况时, 扩容前的虚拟节点分裂, 产生新的虚拟节点, 使客户端根据分裂后的虚 拟节点的数量, 重新确定数据映射的虚拟节点;  The splitting module 602 is configured to: when a physical node is newly added when the system is expanded, and a part of the physical node corresponds to a virtual node, the virtual node before the expansion is split, and a new virtual node is generated, so that the client is virtualized according to the split. The number of nodes, re-determining the virtual nodes of the data mapping;
迁移模块 603, 用于迁移部分虚拟节点到新增加的物理节点;  a migration module 603, configured to migrate a part of the virtual node to the newly added physical node;
删除模块 604, 用于确定虚拟节点和其分裂得到的新的虚拟节点分别管理的数据, 分别 删除虚拟节点和其分裂得到的新的虚拟节点上不属于自身管理的数据。  The deleting module 604 is configured to determine data respectively managed by the virtual node and the new virtual node that is split by the virtual node, and delete data that is not managed by the virtual node and the new virtual node that is split by the virtual node.
其中, 迁移模块 603, 具体用于  Wherein, the migration module 603 is specifically used for
计算每个物理节点对应虚拟节点数量的平均值, 将物理节点对应虚拟节点数量超过平 均值的虚拟节点迁移到物理节点对应虚拟节点数量小于平均值的物理节点上。  The average value of the number of virtual nodes corresponding to each physical node is calculated, and the virtual node whose physical node corresponds to the average number of virtual nodes is migrated to the physical node whose physical node corresponds to the number of virtual nodes less than the average.
本实施例在每个物理节点对应较少虚拟节点的情况下, 扩容时通过虚拟节点分裂的方 式, 动态的迁移虚拟节点, 避免了预先设定很大数量的虚拟节点引起的可靠性问题, 提高 了存储设备的可靠性, 同时避免了没有虚拟节点可以分配引起的无法扩容问题, 提高了存 储设备的可扩展性。 实施例 5 In this embodiment, when each physical node corresponds to a small number of virtual nodes, the virtual node is dynamically migrated by the virtual node splitting during the capacity expansion, thereby avoiding the reliability problem caused by presetting a large number of virtual nodes, and improving The reliability of the storage device, while avoiding the problem of non-expansion caused by the fact that no virtual node can be allocated, improving the storage The scalability of the storage device. Example 5
参见图 9, 本实施例提供了一种客户端设备, 客户端设备包括:  Referring to FIG. 9, this embodiment provides a client device, where the client device includes:
更新模块 701, 用于当虚拟节点分裂时, 更新物理节点和虚拟节点的对应关系, 得到第 一对应关系;  The updating module 701 is configured to update a correspondence between the physical node and the virtual node when the virtual node is split, to obtain a first correspondence relationship;
访问模块 702, 用于确定欲访问的数据映射的虚拟节点, 根据第一对应关系, 确定数据 所在的第一物理节点, 向第一物理节点请求访问数据;  The access module 702 is configured to determine a virtual node of the data mapping to be accessed, determine, according to the first correspondence, the first physical node where the data is located, and request the access data from the first physical node;
更新模块 701, 还用于当虚拟节点迁移时, 更新物理节点和虚拟节点的对应关系, 得到 第二对应关系;  The update module 701 is further configured to: when the virtual node migrates, update the correspondence between the physical node and the virtual node to obtain a second correspondence;
访问模块 702, 还用于根据第二对应关系, 确定数据所在的第二物理节点, 向第二物理 节点请求访问数据。  The access module 702 is further configured to determine, according to the second correspondence, the second physical node where the data is located, and request the access data from the second physical node.
其中, 访问模块 702包括: 确定单元, 用于根据公式 Hash (key) mod (2N), 确定欲访问 的数据映射的虚拟节点,  The access module 702 includes: a determining unit, configured to determine a virtual node of the data mapping to be accessed according to the formula Hash (key) mod (2N),
其中, Hash表示哈希算法, key表示欲访问的数据的标识, mod表示取模运算, N为分 裂前虚拟节点的数量, 2N表示分裂后虚拟节点的数量。  Hash represents a hash algorithm, key represents the identifier of the data to be accessed, mod represents the modulo operation, N is the number of virtual nodes before the split, and 2N represents the number of virtual nodes after the split.
进一步, 访问模块 702, 还用于在当虚拟节点分裂时, 更新物理节点和虚拟节点的对应 关系之前, 确定欲访问的数据映射的虚拟节点, 根据第三对应关系, 确定数据所在的第一 物理节点, 向第一物理节点请求访问数据, 第三对应关系是虚拟节点分裂之前, 物理节点 和虚拟节点的对应关系。  Further, the accessing module 702 is further configured to: before updating the correspondence between the physical node and the virtual node when the virtual node is split, determine the virtual node of the data mapping to be accessed, and determine the first physical location where the data is located according to the third correspondence relationship. The node requests access data from the first physical node, and the third correspondence relationship is a correspondence between the physical node and the virtual node before the virtual node is split.
本实施例提供一种相应的访问数据方案。 并且, 在虚拟节点分裂过程中, 虚拟节点与 其分裂得到的新的虚拟节点对应同一物理节点, 因此, 在扩容过程中, 客户端设备仍然能 正确访问到数据, 上层业务不会受到影响。 本发明提供的存储设备和客户端设备, 与方法实施例属于同一构思, 本领域技术人员 可以清楚地了解到, 为描述的方便和简洁, 上述描述的装置和单元的具体工作过程, 可以 参考前述方法实施例中的对应过程, 在此不再赘述。 本领域技术人员可以清楚地了解到, 以上所描述的装置实施例仅仅是示意性的, 所述 单元 /模块的划分, 仅仅为一种逻辑功能划分, 实际实现时可以有另外的划分方式。 例如, 各功能单元 /模块可以集成在一个处理单元 /模块中, 也可以是各个单元 /模块单独物理存 在, 也可以两个或两个以上单元 /模块集成在一个单元 /模块。 上述集成的单元 /模块既可以 采用硬件的形式实现, 也可以采用软件功能单元 /模块的形式实现。 本领域普通技术人员可以理解实现上述实施例的全部或部分步骤可以通过硬件来完 成, 也可以通过程序来指令相关的硬件完成, 所述的程序可以存储于一种计算机可读存储 介质中, 上述提到的存储介质可以是只读存储器, 磁盘或光盘等。 以上所述仅为本发明的较佳实施例, 并不用以限制本发明, 凡在本发明的精神和原则 之内, 所作的任何修改、 等同替换、 改进等, 均应包含在本发明的保护范围之内。 This embodiment provides a corresponding access data scheme. In addition, in the virtual node splitting process, the virtual node and the new virtual node that is split are the same physical node. Therefore, during the capacity expansion process, the client device can still access the data correctly, and the upper layer service is not affected. The storage device and the client device provided by the present invention are the same as the method embodiment, and those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the device and the unit described above may refer to the foregoing. Corresponding processes in the method embodiments are not described herein again. It will be apparent to those skilled in the art that the device embodiments described above are merely illustrative, and the division of the units/modules is only a logical function division, and may be further divided in actual implementation. For example, each functional unit/module can be integrated in one processing unit/module, or each unit/module can be physically stored separately. It is also possible to integrate two or more units/modules in one unit/module. The above integrated unit/module can be implemented in the form of hardware or in the form of a software functional unit/module. A person skilled in the art may understand that all or part of the steps of implementing the above embodiments may be completed by hardware, or may be instructed by a program to execute related hardware, and the program may be stored in a computer readable storage medium. The storage medium mentioned may be a read only memory, a magnetic disk or an optical disk or the like. The above is only the preferred embodiment of the present invention, and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., which are within the spirit and scope of the present invention, should be included in the protection of the present invention. Within the scope.

Claims

权 利 要 求 书 Claim
1、 一种扩容的方法, 其特征在于, 所述方法包括: A method for expanding a capacity, the method comprising:
根据扩容前的物理节点的数量, 确定虚拟节点的数量, 使每个扩容前的物理节点对应 n 个虚拟节点, n小于预设值; Determine the number of virtual nodes according to the number of physical nodes before the expansion, so that the physical nodes before each expansion correspond to n virtual nodes, where n is smaller than the preset value;
当系统扩容时新增加了物理节点、 且部分物理节点出现对应一个虚拟节点的情况时, 扩 容前的虚拟节点分裂, 产生新的虚拟节点, 使客户端根据分裂后的虚拟节点的数量, 重新确 定数据映射的虚拟节点;  When a physical node is added to the system and a physical node corresponds to a virtual node, the virtual node before the expansion is split, and a new virtual node is generated, so that the client re-determines according to the number of the virtual nodes after the split. Virtual node of data mapping;
迁移部分虚拟节点到新增加的物理节点;  Migrate some virtual nodes to newly added physical nodes;
确定虚拟节点和其分裂得到的新的虚拟节点分别管理的数据, 分别删除虚拟节点和其分 裂得到的新的虚拟节点上不属于自身管理的数据。  The data managed by the virtual node and the new virtual node obtained by the splitting is determined, and the data that is not managed by the virtual node and the new virtual node that is split by the virtual node is deleted.
2、根据权利要求 1所述的方法, 其特征在于, 所述迁移部分虚拟节点到新增加的物理节 点, 具体包括:  The method according to claim 1, wherein the migrating part of the virtual node to the newly added physical node includes:
计算每个物理节点对应虚拟节点数量的平均值, 将物理节点对应虚拟节点数量超过所述 平均值的虚拟节点迁移到物理节点对应虚拟节点数量小于所述平均值的物理节点上。  The average value of the number of virtual nodes corresponding to each physical node is calculated, and the virtual node corresponding to the physical node corresponding to the virtual node is migrated to the physical node whose physical node corresponds to the virtual node whose number is smaller than the average value.
3、一种基于权利要求 1所述的扩容方法的访问数据的方法,其特征在于,所述方法包括: 当虚拟节点分裂时, 更新物理节点和虚拟节点的对应关系, 得到第一对应关系; 确定欲访问的数据映射的虚拟节点, 根据所述第一对应关系, 确定所述数据所在的第一 物理节点, 向所述第一物理节点请求访问数据; A method for accessing data according to the expansion method of claim 1, wherein the method comprises: updating a correspondence between a physical node and a virtual node when the virtual node is split, to obtain a first correspondence; Determining a virtual node of the data mapping to be accessed, determining, according to the first correspondence, a first physical node where the data is located, and requesting access data from the first physical node;
当虚拟节点迁移时, 更新物理节点和虚拟节点的对应关系, 得到第二对应关系; 根据所述第二对应关系, 确定所述数据所在的第二物理节点, 向所述第二物理节点请求 访问数据。  Updating the correspondence between the physical node and the virtual node to obtain a second correspondence relationship when the virtual node is migrated; determining, according to the second correspondence, the second physical node where the data is located, and requesting access to the second physical node data.
4、 根据权利要求 3所述的方法, 其特征在于, 所述确定欲访问的数据映射的虚拟节点, 具体包括:  The method according to claim 3, wherein the determining the virtual node of the data mapping to be accessed comprises:
根据公式 Hash (key) mod (2N), 确定欲访问的数据映射的虚拟节点,  Determine the virtual node of the data map to be accessed according to the formula Hash (key) mod (2N),
其中, Hash表示哈希算法, key表示欲访问的数据的标识, mod表示取模运算, N为分裂 前虚拟节点的数量, 2N表示分裂后虚拟节点的数量。  Hash represents a hash algorithm, key represents the identifier of the data to be accessed, mod represents the modulo operation, N is the number of virtual nodes before the split, and 2N represents the number of virtual nodes after the split.
5、 根据权利要求 3所述的方法, 其特征在于, 所述当虚拟节点分裂时, 更新物理节点和 虚拟节点的对应关系之前, 所述方法还包括: 确定欲访问的数据映射的虚拟节点, 根据第三对应关系, 确定所述数据所在的第一物理 节点, 向所述第一物理节点请求访问数据, 所述第三对应关系是虚拟节点分裂之前, 物理节 点和虚拟节点的对应关系。 The method according to claim 3, wherein the method further includes: before updating the correspondence between the physical node and the virtual node when the virtual node is split, the method further includes: Determining a virtual node of the data mapping to be accessed, determining, according to the third correspondence, the first physical node where the data is located, requesting access data from the first physical node, where the third correspondence is before the virtual node is split. The correspondence between physical nodes and virtual nodes.
6、根据权利要求 3所述的方法, 其特征在于, 所述向所述第二物理节点请求访问数据之 后, 所述方法还包括:  The method according to claim 3, wherein after the requesting the second physical node to access the data, the method further includes:
如果所述数据还没有从第一物理节点上迁移到第二物理节点, 所述第二物理节点从所述 第一物理节点获取所述数据。  If the data has not been migrated from the first physical node to the second physical node, the second physical node acquires the data from the first physical node.
7、 一种存储设备, 其特征在于, 所述存储设备包括: A storage device, wherein the storage device comprises:
确定模块, 用于根据扩容前的物理节点的数量, 确定虚拟节点的数量, 使每个扩容前的 物理节点对应 n个虚拟节点, n小于预设值;  a determining module, configured to determine the number of virtual nodes according to the number of physical nodes before the expansion, so that the physical nodes before each expansion correspond to n virtual nodes, where n is smaller than a preset value;
分裂模块, 用于当系统扩容时新增加了物理节点、 且部分物理节点出现对应一个虚拟节 点的情况时, 扩容前的虚拟节点分裂, 产生新的虚拟节点, 使客户端根据分裂后的虚拟节点 的数量, 重新确定数据映射的虚拟节点;  The splitting module is used to expand the physical node when the system is expanded, and some physical nodes appear corresponding to one virtual node. The virtual node before the expansion is split, and a new virtual node is generated, so that the client is based on the split virtual node. Quantity, redefining the virtual node of the data map;
迁移模块, 用于迁移部分虚拟节点到新增加的物理节点;  a migration module, configured to migrate part of the virtual node to the newly added physical node;
删除模块, 用于确定虚拟节点和其分裂得到的新的虚拟节点分别管理的数据, 分别删除 虚拟节点和其分裂得到的新的虚拟节点上不属于自身管理的数据。  The deletion module is configured to determine data managed by the virtual node and the new virtual node that is split by the virtual node, and delete the data that is not managed by the virtual node and the new virtual node that is split by the virtual node.
8、 根据权利要求 7所述的存储设备, 其特征在于, 所述迁移模块, 具体用于 计算每个物理节点对应虚拟节点数量的平均值, 将物理节点对应虚拟节点数量超过所述 平均值的虚拟节点迁移到物理节点对应虚拟节点数量小于所述平均值的物理节点上。  The storage device according to claim 7, wherein the migration module is specifically configured to calculate an average value of the number of virtual nodes corresponding to each physical node, and the number of virtual nodes corresponding to the physical node exceeds the average value. The virtual node is migrated to a physical node whose physical node corresponds to a virtual node whose number is smaller than the average value.
9、 一种客户端设备, 其特征在于, 所述客户端设备包括: A client device, the client device comprising:
更新模块, 用于当虚拟节点分裂时, 更新物理节点和虚拟节点的对应关系, 得到第一对 应关系;  An update module, configured to update a correspondence between a physical node and a virtual node when the virtual node is split, to obtain a first corresponding relationship;
访问模块, 用于确定欲访问的数据映射的虚拟节点, 根据所述第一对应关系, 确定所述 数据所在的第一物理节点, 向所述第一物理节点请求访问数据;  An access module, configured to determine a virtual node of the data mapping to be accessed, determining, according to the first correspondence, a first physical node where the data is located, and requesting access data from the first physical node;
所述更新模块, 还用于当虚拟节点迁移时, 更新物理节点和虚拟节点的对应关系, 得到 第二对应关系;  The update module is further configured to update a correspondence between the physical node and the virtual node when the virtual node migrates, to obtain a second correspondence relationship;
所述访问模块, 还用于根据所述第二对应关系, 确定所述数据所在的第二物理节点, 向 所述第二物理节点请求访问数据。 The access module is further configured to determine, according to the second correspondence, a second physical node where the data is located, and request access data from the second physical node.
10、 根据权利要求 9所述的客户端设备, 其特征在于, 所述访问模块包括: 确定单元, 用于 The client device according to claim 9, wherein the access module comprises: a determining unit, configured to:
根据公式 Hash (key) mod (2N), 确定欲访问的数据映射的虚拟节点,  Determine the virtual node of the data map to be accessed according to the formula Hash (key) mod (2N),
其中, Hash表示哈希算法, key表示欲访问的数据的标识, mod表示取模运算, N为分裂 前虚拟节点的数量, 2N表示分裂后虚拟节点的数量。  Hash represents a hash algorithm, key represents the identifier of the data to be accessed, mod represents the modulo operation, N is the number of virtual nodes before the split, and 2N represents the number of virtual nodes after the split.
11、 根据权利要求 9所述的客户端设备, 其特征在于, 所述访问模块, 还用于在当虚拟 节点分裂时, 更新物理节点和虚拟节点的对应关系之前,  The client device according to claim 9, wherein the access module is further configured to: before updating the correspondence between the physical node and the virtual node when the virtual node is split,
确定欲访问的数据映射的虚拟节点, 根据第三对应关系, 确定所述数据所在的第一物理 节点, 向所述第一物理节点请求访问数据, 所述第三对应关系是虚拟节点分裂之前, 物理节 点和虚拟节点的对应关系。  Determining a virtual node of the data mapping to be accessed, determining, according to the third correspondence, the first physical node where the data is located, requesting access data from the first physical node, where the third correspondence is before the virtual node is split. The correspondence between physical nodes and virtual nodes.
PCT/CN2012/070464 2012-01-17 2012-01-17 Capacity expansion method and device and data access method and device WO2013106993A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201280000231.1A CN103348653B (en) 2012-01-17 2012-01-17 The method and apparatus of dilatation and the method and apparatus of visit data
PCT/CN2012/070464 WO2013106993A1 (en) 2012-01-17 2012-01-17 Capacity expansion method and device and data access method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2012/070464 WO2013106993A1 (en) 2012-01-17 2012-01-17 Capacity expansion method and device and data access method and device

Publications (1)

Publication Number Publication Date
WO2013106993A1 true WO2013106993A1 (en) 2013-07-25

Family

ID=48798484

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2012/070464 WO2013106993A1 (en) 2012-01-17 2012-01-17 Capacity expansion method and device and data access method and device

Country Status (2)

Country Link
CN (1) CN103348653B (en)
WO (1) WO2013106993A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104219163A (en) * 2014-08-28 2014-12-17 杭州天宽科技有限公司 Load balancing method for node dynamic forward based on dynamic replication method and virtual node method
CN106843745A (en) * 2015-12-03 2017-06-13 南京中兴新软件有限责任公司 Capacity expansion method and device
CN107943412A (en) * 2016-10-12 2018-04-20 阿里巴巴集团控股有限公司 A kind of subregion division, the method, apparatus and system for deleting data file in subregion
CN109428913A (en) * 2017-08-25 2019-03-05 浙江宇视科技有限公司 A kind of storage expansion method and device

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106874218B (en) * 2017-01-12 2019-12-03 深圳市汉云科技有限公司 A kind of distribution method and its device of mapped address space

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6667957B1 (en) * 1998-03-14 2003-12-23 University Of Maryland Adaptive routing method for a dynamic network
CN1988518A (en) * 2006-12-13 2007-06-27 北京交通大学 Counterpart network searching method
CN101098271A (en) * 2006-06-29 2008-01-02 腾讯科技(深圳)有限公司 Expandable P2P flow media system
WO2009027612A2 (en) * 2007-08-24 2009-03-05 Thomson Licensing Method for constructing a virtual environment in a p2p network of the delaunay type by dynamic grouping of nodes, assistance device and related equipment
US7577134B2 (en) * 2005-08-19 2009-08-18 Brocade Communications Systems, Inc. Port expander for fibre channel fabrics in storage area networks

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102063473B (en) * 2010-12-21 2012-10-10 南京壹进制信息技术有限公司 Method for expanding storage space on line

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6667957B1 (en) * 1998-03-14 2003-12-23 University Of Maryland Adaptive routing method for a dynamic network
US7577134B2 (en) * 2005-08-19 2009-08-18 Brocade Communications Systems, Inc. Port expander for fibre channel fabrics in storage area networks
CN101098271A (en) * 2006-06-29 2008-01-02 腾讯科技(深圳)有限公司 Expandable P2P flow media system
CN1988518A (en) * 2006-12-13 2007-06-27 北京交通大学 Counterpart network searching method
WO2009027612A2 (en) * 2007-08-24 2009-03-05 Thomson Licensing Method for constructing a virtual environment in a p2p network of the delaunay type by dynamic grouping of nodes, assistance device and related equipment

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104219163A (en) * 2014-08-28 2014-12-17 杭州天宽科技有限公司 Load balancing method for node dynamic forward based on dynamic replication method and virtual node method
CN104219163B (en) * 2014-08-28 2016-08-17 杭州天宽科技有限公司 The load-balancing method that a kind of node based on dynamic copies method and dummy node method dynamically moves forward
CN106843745A (en) * 2015-12-03 2017-06-13 南京中兴新软件有限责任公司 Capacity expansion method and device
CN107943412A (en) * 2016-10-12 2018-04-20 阿里巴巴集团控股有限公司 A kind of subregion division, the method, apparatus and system for deleting data file in subregion
CN107943412B (en) * 2016-10-12 2021-10-22 阿里巴巴集团控股有限公司 Method, device and system for splitting partition and deleting data file in partition
CN109428913A (en) * 2017-08-25 2019-03-05 浙江宇视科技有限公司 A kind of storage expansion method and device
CN109428913B (en) * 2017-08-25 2021-06-01 浙江宇视科技有限公司 Storage capacity expansion method and device

Also Published As

Publication number Publication date
CN103348653B (en) 2016-01-20
CN103348653A (en) 2013-10-09

Similar Documents

Publication Publication Date Title
US10896102B2 (en) Implementing secure communication in a distributed computing system
US11288286B2 (en) Storage system with data consistency checking in synchronous replication using active snapshot set
US11055014B2 (en) Storage system providing automatic configuration updates for remote storage objects in a replication process
US10642694B2 (en) Monitoring containers in a distributed computing system
US10824512B2 (en) Managing journaling resources with copies stored in multiple locations
US20140351636A1 (en) Method, device, and system for data reconstruction
EP3502877B1 (en) Data loading method and apparatus for virtual machines
WO2020204882A1 (en) Snapshot-enabled storage system implementing algorithm for efficient reading of data from stored snapshots
CN110795206B (en) System and method for facilitating cluster-level caching and memory space
US20130332418A1 (en) Method of managing data in asymmetric cluster file system
WO2021057956A1 (en) Method, apparatus and system for managing mirror image file, and computer device and storage medium
CN112799841B (en) Method and device for data object storage management
WO2019148841A1 (en) Distributed storage system, data processing method and storage node
US20170351601A1 (en) Computer system, computer, and method
WO2013106993A1 (en) Capacity expansion method and device and data access method and device
US10877684B2 (en) Changing a distributed storage volume from non-replicated to replicated
US20160328153A1 (en) Memory data versioning
US20230273899A1 (en) Data migration in a distributive file system
CN111444157A (en) Distributed file system and data access method
JP6233403B2 (en) Storage system, storage device, storage device control method and control program, management device, management device control method and control program
CN117032586A (en) Flash register with write equalization
US11144232B2 (en) Storage system with efficient snapshot pair creation during synchronous replication of logical storage volumes
US11216204B2 (en) Degraded redundant metadata, DRuM, technique
US11256434B2 (en) Data de-duplication
CN111400098A (en) Copy management method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12866088

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12866088

Country of ref document: EP

Kind code of ref document: A1